Torch.cuda.is_available() returns False on ssh server with NVIDIA GPU

username: ansh7xpex
cooked: <p>I am using my Institute GPU through ssh server (Pardon the terms, I am a newbie). I have been trying for so long but PyTorch torch.cuda.is_available() returns False.</p>
<p>Here is the output of nvidia-smi:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e90006b08d8987e0ece906dc6d5314d3cd4a3c0b.png" data-download-href="https://discuss.pytorch.org/uploads/default/e90006b08d8987e0ece906dc6d5314d3cd4a3c0b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e90006b08d8987e0ece906dc6d5314d3cd4a3c0b_2_522x500.png" alt="image" data-base62-sha1="xfdj0zCtZWXdYLAkP2sXieL2MBR" width="522" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e90006b08d8987e0ece906dc6d5314d3cd4a3c0b_2_522x500.png, https://discuss.pytorch.org/uploads/default/original/3X/e/9/e90006b08d8987e0ece906dc6d5314d3cd4a3c0b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/9/e90006b08d8987e0ece906dc6d5314d3cd4a3c0b.png 2x" data-dominant-color="3A182F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">732×701 78.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have also run some commands: <code>torch.cuda.is_available()</code> returns <code>False</code> , while <code>torch.backend.cudnn.enabled</code> is <code>True</code> . I have tried with <code>cudatoolkit 10.2</code> and <code>cudatoolkit 11.1</code> , as well as <code>cudatoolkit 11.3</code> also and I am still not being able to make a conda environment. After research I have come to know that the issue is due to driver incompatibility, but I still cannot find the solution.</p>
<p>Any help is appreciated. I am tagging <a class="mention" href="/u/ptrblck">@ptrblck</a> as I am in a desperate need of help. The results here are for <code>PyTorch 1.10.0</code> and <code>cudatoolkit 10.1.243</code>.</p>
<p><a href="https://i.stack.imgur.com/JTdEh.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/JTdEh.png" alt="enter image description here" width="643" height="211"></a></p>
<p>Here is conda list torch.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2c04cf84695e9e2692b2dce813a906b633e10ec0.png" data-download-href="https://discuss.pytorch.org/uploads/default/2c04cf84695e9e2692b2dce813a906b633e10ec0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c04cf84695e9e2692b2dce813a906b633e10ec0_2_690x104.png" alt="image" data-base62-sha1="6hpjyn0Prr85wepFpAof8xu1XUc" width="690" height="104" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c04cf84695e9e2692b2dce813a906b633e10ec0_2_690x104.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c04cf84695e9e2692b2dce813a906b633e10ec0_2_1035x156.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c04cf84695e9e2692b2dce813a906b633e10ec0_2_1380x208.png 2x" data-dominant-color="341129"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1856×282 50.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Your A100s need to use CUDA&gt;=11.0, so the binaries with <code>cudatoolkit==10.1.243</code> won’t work.<br>
I also see that you’ve MIG enabled on the A100s. Is this on purpose? If so, mask the desired MIG instance via <code>CUDA_VISIBLE_DEVICES</code> and is it in your script as multiple MIG instances cannot be used together.</p>

username: ansh7xpex
cooked: <p>Thanks for the reply <a class="mention" href="/u/ptrblck">@ptrblck</a>. I created a new environment and installed PyTorch using</p>
<p><code>conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=11.1 -c pytorch -c conda-forge</code></p>
<p>But it still shows torch.cuda.is_available() as false and gives a warning.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d63382a44a4cc14ef508d237f75c0497eb196a3.png" data-download-href="https://discuss.pytorch.org/uploads/default/8d63382a44a4cc14ef508d237f75c0497eb196a3" title="Screenshot from 2021-11-03 15-57-53"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d63382a44a4cc14ef508d237f75c0497eb196a3_2_690x60.png" alt="Screenshot from 2021-11-03 15-57-53" data-base62-sha1="kaLTNeWGGX2hFJVardquifVmwUj" width="690" height="60" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d63382a44a4cc14ef508d237f75c0497eb196a3_2_690x60.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d63382a44a4cc14ef508d237f75c0497eb196a3_2_1035x90.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d63382a44a4cc14ef508d237f75c0497eb196a3_2_1380x120.png 2x" data-dominant-color="35122A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-11-03 15-57-53</span><span class="informations">1855×163 29 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ansh7xpex
cooked: <p><a class="mention" href="/u/smth">@smth</a> Can you please help?</p>

username: ptrblck
cooked: <p>Is this a new setup, i.e. did you just install or update new drivers?<br>
The last error message claims the driver fails to initialize, which could come from a broken installation or an update without a restart etc.</p>

username: ansh7xpex
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I just got an ID and password from my institute to login. All I’ve done is to try and install Pytorch using conda.</p>

username: Mohammed
cooked: <p>Hey <a class="mention" href="/u/ansh7xpex">@ansh7xpex</a> I ran into a similar problem. You are probably installing the cpu version of pytorch. You need to make sure that you are installing the pytorch+cuda version. It seems that when you check for the pytorch version, you should expect somthing like ‘1.8.0+cu111’ indicating pytorch + cuda.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/0/9027fed8c414b8ac861f487219b8cfb2c4e361ee.png" alt="cuda" data-base62-sha1="kzgrDgeMg5Ar807Onba7Ea6Fspo" width="353" height="50"></p>
<p>I could not get conda command to work because of some connection error when downloading some files. So I ended up downloading the pytorch+cuda whl version on my computer and then using the pip command. You can download the whl from here <a href="https://download.pytorch.org/whl/cu111/torch-1.9.0%2Bcu111-cp39-cp39-win_amd64.whl" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu111/torch-1.9.0%2Bcu111-cp39-cp39-win_amd64.whl</a></p>
<p>Here is the pip command: pip install torch-1.8.0+cu111-cp39-cp39-win_amd64.whl torchvision==0.9.0+cu111 torchaudio==0.8.0 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a></p>
<p>You also need to create a new conda environment to avoid some installation errors.</p>
<p>I hope that helps.</p>

username: ansh7xpex
cooked: <p><a class="mention" href="/u/mohammed">@Mohammed</a> thanks for the reply. I used this command:</p>
<p><code>pip install torch-1.8.0+cu111-cp39-cp39-win_amd64.whl torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html</code></p>
<p>I do get the output for <code>torch.__version__ </code> you have posted but, the problem persists.</p>

username: Mohammed
cooked: <p><a href="https://discuss.pytorch.org/u/ansh7xpex">@ansh7xpex</a> did you install it in a new conda environment with python 9 and activate it? I also ran into some issues after running the pip command, but I figured out I needed to create another environment.</p>
<p>As you said earlier, you may have some compatibility issues with the driver. So you may need to upgrade or downgrade into another driver version. I have the latest driver version (496.13) with RTX 3080. I can see that you have 470.57.02. Give it a try and install another driver version.</p>
<p>You also can try ‘nvcc -V’ to check the cuda version that you have. I have V11.5.50.</p>

username: ansh7xpex
cooked: <p><a class="mention" href="/u/mohammed">@Mohammed</a>, I created a new conda environment with python 3.8, activated it and ran the command.<br>
I don’t think I can mess with the drivers because I have only a login and password and the GPU is maintained by the IT department at my institute.</p>

username: Mohammed
cooked: <p><a class="mention" href="/u/ansh7xpex">@ansh7xpex</a> I see what you mean.</p>
<p>You can check the following link to know more about the driver requirements.<br>
<a href="https://docs.nvidia.com/deploy/cuda-compatibility" class="onebox" target="_blank" rel="noopener nofollow ugc">https://docs.nvidia.com/deploy/cuda-compatibility</a></p>
<p>For cuda 11, the driver should be &gt;= 450.80.02 for linux or &gt;=456.38 for windows.</p>
<p>As <a class="mention" href="/u/ptrblck">@ptrblck</a> said, “Your A100s need to use CUDA&gt;=11.0,so the binaries with <code>cudatoolkit==10.1.243</code> won’t work.” You probably need to reach out to the IT so that they can upgrade the driver for you.</p>

username: ansh7xpex
cooked: <p>Thanks <a class="mention" href="/u/mohammed">@Mohammed</a>, I will reach out to them.</p>

username: ansh7xpex
cooked: <p>I finally solved the problem. The problem was not due to drivers or anything. My college senior gave me a piece of code to write at the beginning of the file that I wanted to run. Here is the code:</p>
<pre><code class="lang-auto">import os
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = "0"

import tensorflow as tf
from tensorflow.python.client import device_lib
print(tf.config.list_physical_devices("GPU"))

print("GPU Available: "+str(torch.cuda.is_available())) #To finally check if the GPUs are being detected.
</code></pre>
<p>Adding this piece of code helped me to run my model on GPU.</p>

username: ptrblck
cooked: <p>Could you please check if removing the <code>os.environ</code> calls would break your setup again?<br>
If so, I would guess you might have exported <code>CUDA_VISIBLE_DEVICES</code> in your default environment and could check it with <code>echo $CUDA_VISIBLE_DEVICES</code> in the terminal.</p>

username: ansh7xpex
cooked: <p>Yes, <a class="mention" href="/u/ptrblck">@ptrblck</a>, removing the <code>os.environ</code> calls breaks my setup again.</p>

username: ptrblck
cooked: <p>That’s interesting. Did you check your environment via <code>echo $CUDA_VISIBLE_DEVICES</code> or <code>export</code>? And if so, do you see any CUDA related env variables set to wrong values?<br>
You should not need to set the <code>os.environ</code> inside a script, as I would consider it quite flaky if you are setting these env vars too late (and the system env vars would be used instead).</p>

username: ansh7xpex
cooked: <p>I typed <code>export</code> in the terminal and found two variables that were related to cuda (I’m a newbie, pardon if I am wrong)</p>
<p><code>declare -x LD_LIBRARY_PATH="/usr/local/cuda-11.4/lib64:/usr/local/cuda/lib</code></p>
<p><code>declare -x PATH="/home/ansh.arora/tmp/ENTER/bin:/home/ansh.arora/tmp/ENTER/condabin:/usr/local/cuda-11.4/bin:/usr/lib64/qt-3.3/bin:/usr/local/cuda/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/ansh.arora/.local/bin:/home/ansh.arora/bin</code></p>