PyTorch cannot find GPU, 2021 version

zzzhhh: <p>Environment:</p>
<ul>
<li>Remote Linux with core version 5.8.0. I am not a super user.</li>
<li>Python 3.8.6</li>
<li>CUDA Version: 11.1</li>
<li>GPU is RTX 3090 with driver version 455.23.05</li>
<li>CPU: Intel Core i9-10900K</li>
<li>PyTorch version: 1.8.0+cu111</li>
<li>System imposed RAM quota: 4GB</li>
<li>System imposed number of threads: 512198</li>
<li>System imposed RLIMIT_NPROC value: 300</li>
</ul>
<p>After I run the following code (immediately after I entered python3 command line, so nothing else run before):</p>
<pre><code class="lang-auto">import os
os.environ['OPENBLAS_NUM_THREADS'] = '2'
import torch
torch.cuda.is_available()
torch.cuda.device_count()
</code></pre>
<p><code>torch.cuda.device_count()</code> returns 0 and <code>torch.cuda.is_available()</code> returns <code>False</code> with some additional error messages</p>
<blockquote>
<p>/usr/local/lib/python3.8/dist-packages/torch/cuda/<strong>init</strong>.py:52: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 2: out of memory (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:109.)<br>
return torch._C._cuda_getDeviceCount() &gt; 0</p>
</blockquote>
<p>But I can run <code>nvidia-smi</code> and <code>nvcc</code> successfully</p>
<blockquote>
<p>nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2020 NVIDIA Corporation<br>
Built on Mon_Oct_12_20:09:46_PDT_2020<br>
Cuda compilation tools, release 11.1, V11.1.105<br>
Build cuda_11.1.TC455_06.29190527_0</p>
</blockquote>
<p>which means the GPU hardware and CUDA are installed. Why can PyTorch not see the GPU? Is it possible that PyTorch is divided into GPU version and non-GPU version and the sys admin happened to install a non-GPU version? If so, how can I tell if the PyTorch installed is GPU or non-GPU version? Is there any requirements on CUDA version so that CUDA 11.1 installed does not get along well with PyTorch 1.8.0? If so, what version of PyTorch is CUDA 11.1 happy to work with? Thank you for help.</p>

ptrblck: <aside class="quote no-group" data-username="zzzhhh" data-post="1" data-topic="135701">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/65b543/48.png" class="avatar"> zzzhhh:</div>
<blockquote>
<p>Is it possible that PyTorch is divided into GPU version and non-GPU version and the sys admin happened to install a non-GPU version?</p>
</blockquote>
</aside>
<p>Depending how you’ve installed PyTorch, you can pick between the CPU and CUDA runtimes as described in the <a href="https://pytorch.org/get-started/locally/">install instructions</a>.</p>
<aside class="quote no-group" data-username="zzzhhh" data-post="1" data-topic="135701">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/65b543/48.png" class="avatar"> zzzhhh:</div>
<blockquote>
<p>If so, how can I tell if the PyTorch installed is GPU or non-GPU version?</p>
</blockquote>
</aside>
<p>Check <code>print(torch.version.cuda)</code> as well as <code>python -m torch.utils.collect_env</code> and make sure a CUDA runtime is found.<br>
If that’s the case, your local setup is most likely broken (e.g. via updating an NVIDIA driver without a restart etc.).</p>
<aside class="quote no-group" data-username="zzzhhh" data-post="1" data-topic="135701">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/65b543/48.png" class="avatar"> zzzhhh:</div>
<blockquote>
<p>Is there any requirements on CUDA version so that CUDA 11.1 installed does not get along well with PyTorch 1.8.0?</p>
</blockquote>
</aside>
<p>Again, it depends on your setup. If you’ve installed the pip wheels or conda binaries with a CUDA runtime, the local CUDA toolkit won’t be used unless you are building a custom CUDA extension, as the binaries already ship with their own CUDA runtime.</p>

zzzhhh: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="135701">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Depending how you’ve installed PyTorch, you can pick between the CPU and CUDA runtimes as described in the <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">install instructions</a>.</p>
</blockquote>
</aside>
<p>The sys admin installed PyTorch. I don’t know what parameters he used for installation. Does PyTorch keep a copy of command line used for installation?</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="135701">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Check <code>print(torch.version.cuda)</code> as well as <code>python -m torch.utils.collect_env</code> and make sure a CUDA runtime is found.</p>
</blockquote>
</aside>
<p><code>print(torch.version.cuda)</code> returns</p>
<blockquote>
<p>11.1</p>
</blockquote>
<p>When I ran <code>torch.utils.collect_env</code> inside python3 command line (to incorporate code that sets <code>OPENBLAS_NUM_THREADS</code> variable), I was told</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “&lt;stdin&gt;”, line 1, in &lt;module&gt;<br>
AttributeError: module ‘torch.utils’ has no attribute ‘collect_env’</p>
</blockquote>
<p>But I downloaded the code of collect_env.py <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/collect_env.py" rel="noopener nofollow ugc">here</a> and run it (with a line to set <code>OPENBLAS_NUM_THREADS</code> variable added). The output is as follows:</p>
<blockquote>
<p>Collecting environment information…<br>
/usr/local/lib/python3.8/dist-packages/torch/cuda/<strong>init</strong>.py:52: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 2: out of memory (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:109.)<br>
return torch._C._cuda_getDeviceCount() &gt; 0<br>
PyTorch version: 1.8.0+cu111<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.1<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Pop!_OS 20.10 (x86_64)<br>
GCC version: (Ubuntu 10.2.0-13ubuntu1) 10.2.0<br>
Clang version: Could not collect<br>
CMake version: version 3.16.3<br>
Libc version: glibc-2.32</p>
<p>Python version: 3.8.6 (default, Jan 27 2021, 15:42:20)  [GCC 10.2.0] (64-bit runtime)<br>
Python platform: Linux-5.8.0-7642-generic-x86_64-with-glibc2.32<br>
Is CUDA available: False<br>
CUDA runtime version: 11.1.105<br>
GPU models and configuration: GPU 0: GeForce RTX 3090<br>
Nvidia driver version: 455.23.05<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.19.5<br>
[pip3] torch==1.8.0+cu111<br>
[pip3] torchvision==0.9.0+cu111<br>
[conda] Could not collect</p>
</blockquote>
<p>It gives the same error as running <code>torch.cuda.is_available()</code> and <code>torch.cuda.device_count()</code>. Is the CUDA runtime found?</p>
<p>PS, Pop!_OS is a variant of ubuntu. conda is actually installed in the system which I detected using command <code> dpkg -l|grep conda</code>:</p>
<blockquote>
<p>ii  conda-package-handling                           1.7.0-1<br>
amd64        create and extract conda packages of various formats</p>
</blockquote>
<p>Please let me know what I can do on my side for further troubleshooting.</p>

ptrblck: <p>The binaries are using the CUDA11.1 runtime, so your device should be recognized.<br>
Since you are seeing an error in the <code>__init__</code> method:</p>
<pre><code class="lang-python">UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 2: out of memory (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:109.)
</code></pre>
<p>this points to a setup issue so your sys admin might need to debug the NVIDIA driver and CUDA toolkit installation.</p>
<p>Since it seems you have limited right on this machine, you could check if any CUDA samples can be compiled and executed and might be able to use a docker container as a workaround (however, which could also fail given that the driver might not be installed correctly).</p>

zzzhhh: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="135701">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Since it seems you have limited right on this machine, you could check if any CUDA samples can be compiled and executed and might be able to use a docker container as a workaround (however, which could also fail given that the driver might not be installed correctly).</p>
</blockquote>
</aside>
<p>Most of CUDA samples can be compiled and executed, like <code>0_Simple/vectorAdd</code>, <code>6_Advanced/eigenvalues</code> and <code>7_CUDALibraries/simpleCUFFT</code>. Some can be compiled but not executed like <code>7_CUDALibraries/cuSolverSp_LinearSolver</code></p>
<blockquote>
<p>CUDA error at cuSolverSp_LinearSolver.cpp:278 code=1(CUSPARSE_STATUS_NOT_INITIALIZED) “cusparseCreate(&amp;cusparseHandle)”</p>
</blockquote>
<p>, and some can’t be compiled like <code>7_CUDALibraries/conjugateGradient</code>. Does it mean CUDA is actually set up? Since PyTorch is nothing different from these CUDA samples in terms of calling CUDA interface, if all CUDA APIs PyTorch is calling are those in samples that I can compile and execute, PyTorch should be good to run. But why can’t PyTorch find any GPU? Is it possible I install a PyTorch in my user space (or build up one from source) and this PyTorch version can see the GPU?</p>

ptrblck: <aside class="quote no-group" data-username="zzzhhh" data-post="6" data-topic="135701">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/65b543/48.png" class="avatar"> zzzhhh:</div>
<blockquote>
<p>Since PyTorch is nothing different from these CUDA samples in terms of calling CUDA interface, if all CUDA APIs PyTorch is calling are those in samples that I can compile and execute, PyTorch should be good to run.</p>
</blockquote>
</aside>
<p>I wouldn’t claim that’s the case, as e.g. an NVIDIA driver update without a restart would leave the machine in an “undefined behavior” status to my experience. I.e. some applications might work, other might crash. However, based on the samples it seems that at least the toolkit is partially working.</p>
<aside class="quote no-group" data-username="zzzhhh" data-post="6" data-topic="135701">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/65b543/48.png" class="avatar"> zzzhhh:</div>
<blockquote>
<p>Is it possible I install a PyTorch in my user space (or build up one from source) and this PyTorch version can see the GPU?</p>
</blockquote>
</aside>
<p>Yes, you could build PyTorch from source, but I would still claim that your setup might be in a flaky state.</p>

zzzhhh: <p>I think the GPU driver and CUDA were pre-installed when the server was purchased. Anyway, I have tried my best to solve the problem in a technical way. Now that this way does not work, I’m forced to solve it in a political way.</p>

Zongze_Yang: <p>Did you set the CUDA environment variable <code>CUDA_VISIBLE_DEVICES</code>?<br>
Ref: <a href="https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars" class="inline-onebox" rel="noopener nofollow ugc">Programming Guide :: CUDA Toolkit Documentation</a></p>

my3bikaht: <p>CUDA_VISIBLE_DEVICES used to limit visibility</p>

Zongze_Yang: <p>I encounter the same  “CUDA initialization warning” on a ubuntu server. After I set the variable CUDA_VISIBLE_DEVICES, the warning is gone.</p>
<p>In addition, if I do not set the variable, my CUDA program does not run on GPU as well.</p>

zzzhhh: <p>Hi, Zongze, Thank you for the hint. I set this environment variable and tried just now, but the problem persists. I now have a small cluster of powerful workstations under my control and everything is running properly, so I am not concerned with the remote server managed by a terrible sys admin any more. I’m now using TensorFlow2 because it supports distributed training that can make full use of the cluster. I can’t measure the power of RTX 3090 in training and inference, but I think my cluster should beat it. It’s a pity that the two beast-like RTX 3090’s are just kept idle and getting old simply because the sys admin can’t properly install the driver.</p>

Andrea_Rosasco: <p>Couldn’t you have installed miniconda in your home directory and then installed pytorch and cudatoolkit inside a conda environment?<br>
You shouldn’t need any root permission to do that.</p>