Pytorch cuda problem on gpu

dat1: <p>Hii pytorch developer, i am trying to use pytorch on gpu, however it shows ValueError: invalid literal for int() with base 10: “NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.”</p>
<p>additionally, it shows<br>
/home/seis/ret/prog/anaconda3/envs/cpi/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py:52: UserWarning: CUDA initialization: The NVIDIA driver on your system is too old (found version 9010). Please update your GPU driver by downloading and installing a new version from the URL: <a href="http://www.nvidia.com/Download/index.aspx" class="inline-onebox" rel="noopener nofollow ugc">Download Drivers | NVIDIA</a> Alternatively, go to: <a href="https://pytorch.org" rel="noopener nofollow ugc">https://pytorch.org</a> to install a PyTorch version that has been compiled with your version of the CUDA driver. (Triggered internally at  /opt/conda/conda-bld/pytorch_1603729066392/work/c10/cuda/CUDAFunctions.cpp:100.)<br>
return torch._C._cuda_getDeviceCount() &gt; 0<br>
False</p>
<p>Please suggest which pytorch version will support The NVIDIA driver on your system is too old (found version 9010)</p>

ptrblck: <p>You can take a look at <a href="https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html">this doc</a> in particular <code>Table 3</code> shows the driver requirement for each CUDA release.</p>

dat1: <p>Which pytorch version will be compatible with NVIDIA driver version 9010.Can you please suggest, because what the gpu i am using it is older and the engineer doesn’t want to upgrade it to the latest version of NVIDIA driver.</p>

Manuel_Alejandro_Dia: <p>You could check on the <a href="https://pytorch.org/get-started/previous-versions/" rel="noopener nofollow ugc">previous Pytorch versions</a> webpage to see which Pytorch version will work with the CUDA driver you have.</p>

dat1: <p><a class="mention" href="/u/manuel_alejandro_dia">@Manuel_Alejandro_Dia</a>  ```<br>
conda install pytorch==1.7.0 torchvision==0.8.0 torchaudio==0.7.0 cudatoolkit=9.2 -c pytorch</p>
<pre><code class="lang-auto"></code></pre>

dat1: <p>I think cudatoolkit==9.2 will not work so i need cudatoolkit==9.1 how to get it ???</p>

Manuel_Alejandro_Dia: <p>Maybe you could try with:</p>
<pre><code class="lang-auto">conda install pytorch==1.1.0 torchvision==0.3.0 cudatoolkit=9.0 -c pytorch
</code></pre>

dat1: <p>getting error again</p>
<p>In [1]: import torch</p>
<p>In [2]: torch.version.cuda<br>
Out[2]: ‘9.0.176’</p>
<p>In [3]: torch.cuda.is_available()<br>
Out[3]: False</p>

Manuel_Alejandro_Dia: <p>Can you post the output of <code>nvidia-smi</code> if you are in Linux?</p>
<p>Also, maybe check <a href="https://stackoverflow.com/questions/60987997/why-torch-cuda-is-available-returns-false-even-after-installing-pytorch-with" rel="noopener nofollow ugc">this stackoverflow</a> thread.</p>

Subhajit_Chatterjee: <p>Hello,<br>
I have windows operating system, where i am using WSL for running linux based codes. I have NCCl installed and its workinh fine for distributed pytorch. I have checked it using python script.</p>
<p>Now I have installed torch</p>
<p>when I am importing torch and checking torch.cuda.is_available()</p>
<p>Getting error,<br>
/home/joy/miniconda3/envs/VideoMae/lib/python3.8/site-packages/torch/cuda/<strong>init</strong>.py:138: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 2: out of memory (Triggered internally at /opt/conda/conda-bld/pytorch_1702400431970/work/c10/cuda/CUDAFunctions.cpp:108.)<br>
return torch._C._cuda_getDeviceCount() &gt; 0<br>
False</p>
<p>what would be the sollution of this problem.</p>
<p>I am sharing the output of my nvidia-smi</p>
<p>Thu Jan 11 10:51:38 2024<br>
±--------------------------------------------------------------------------------------+<br>
| NVIDIA-SMI 545.23.08              Driver Version: 546.33       CUDA Version: 12.3     |<br>
|-----------------------------------------±---------------------±---------------------+<br>
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |<br>
|                                         |                      |               MIG M. |<br>
|=========================================+======================+======================|<br>
|   0  NVIDIA GeForce RTX 4080        On  | 00000000:18:00.0 Off |                  N/A |<br>
|  0%   46C    P8              14W / 320W |     17MiB / 16376MiB |      0%      Default |<br>
|                                         |                      |                  N/A |<br>
±----------------------------------------±---------------------±---------------------+<br>
|   1  NVIDIA GeForce RTX 4080        On  | 00000000:5E:00.0 Off |                  N/A |<br>
|  0%   37C    P8              12W / 320W |      0MiB / 16376MiB |      0%      Default |<br>
|                                         |                      |                  N/A |<br>
±----------------------------------------±---------------------±---------------------+<br>
|   2  NVIDIA GeForce RTX 4080        On  | 00000000:B0:00.0 Off |                  N/A |<br>
|  0%   41C    P8               8W / 320W |      0MiB / 16376MiB |      0%      Default |<br>
|                                         |                      |                  N/A |<br>
±----------------------------------------±---------------------±---------------------+<br>
|   3  NVIDIA GeForce RTX 4080        On  | 00000000:D8:00.0 Off |                  N/A |<br>
|  0%   43C    P8              14W / 320W |     59MiB / 16376MiB |      0%      Default |<br>
|                                         |                      |                  N/A |<br>
±----------------------------------------±---------------------±---------------------+</p>
<p>±--------------------------------------------------------------------------------------+<br>
| Processes:                                                                            |<br>
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |<br>
|        ID   ID                                                             Usage      |<br>
|=======================================================================================|<br>
|    0   N/A  N/A       270      G   /Xwayland                                 N/A      |<br>
|    1   N/A  N/A       270      G   /Xwayland                                 N/A      |<br>
|    2   N/A  N/A       270      G   /Xwayland                                 N/A      |<br>
|    3   N/A  N/A       270      G   /Xwayland                                 N/A      |<br>
±--------------------------------------------------------------------------------------+</p>

ptrblck: <aside class="quote no-group" data-username="Subhajit_Chatterjee" data-post="10" data-topic="128011">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/subhajit_chatterjee/48/63036_2.png" class="avatar"> Subhajit_Chatterjee:</div>
<blockquote>
<p>I have NCCl installed and its workinh fine for distributed pytorch. I have checked it using python script.</p>
<p>Now I have installed torch</p>
<p>when I am importing torch and checking torch.cuda.is_available()</p>
<p>Getting error,<br>
/home/joy/miniconda3/envs/VideoMae/lib/python3.8/site-packages/torch/cuda/<strong>init</strong>.py:138: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 2: out of memory (Triggered internally at /opt/conda/conda-bld/pytorch_1702400431970/work/c10/cuda/CUDAFunctions.cpp:108.)<br>
return torch._C._cuda_getDeviceCount() &gt; 0<br>
False</p>
</blockquote>
</aside>
<p>Could you describe in which setup the PyTorch Distributed use case works fine and why a simple CUDA check fails?</p>

Subhajit_Chatterjee: <p>Sorry for unclear information. 3 days ago i have installed NCCL into the system for working with two GPUS. i have done all the installing and environment settings. i have 2 GPUs and i have installed environment on WSL.</p>
<p>conda create --name video python=3.8</p>
<p>After that i have installed</p>
<h1><a name="cuda-117-1" class="anchor" href="#cuda-117-1"></a>CUDA 11.7</h1>
<p>conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia</p>
<p>It was working fine, torch distributed running was activated while running the code.</p>
<p>yesterday i have added 2 more  same GPUs into the system and restarted the system. while checking NCCL is was responding that four GPUs are active. but when i activate the previous environment and try to check whether torch is accessing the GPUs or not. i got error.</p>