Issue with pytorch, CUDA and nvidia-drivers

username: ishackigozi
cooked: <p>Hello,<br>
We are having issues with using torch in our python environments. We have two A100’s GPU  with Nvidia Driver Version: 470.129.06. CUDA Libraries installed  11.3<br>
root@R940-01:~# nvcc --version<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2021 NVIDIA Corporation<br>
Built on Sun_Mar_21_19:15:46_PDT_2021<br>
Cuda compilation tools, release 11.3, V11.3.58<br>
Build cuda_11.3.r11.3/compiler.29745058_0</p>
<p>I have installed pytorch<br>
- cudatoolkit=11.3<br>
- pytorch<br>
- torchaudio<br>
- torchvision</p>
<p>But GPU driver and CUDA are not accessible by Pytorch. Please advise</p>
<p>(test) i.kigozi@R940-01:~$ python<br>
Python 3.9.0 (default, Nov 15 2020, 14:28:56)<br>
[GCC 7.3.0] :: Anaconda, Inc. on linux<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
torch.cuda.is_available()<br>
False</p>
</blockquote>
</blockquote>
</blockquote>

username: ptrblck
cooked: <p>Did you (re-)install the NVIDIA drivers recently (or the CUDA toolkit) without a restart?<br>
Based on the error I would guess PyTorch has trouble communicating with the driver, so you could check if any other CUDA sample works (or if the binary would work in a docker container). If this doesn’t help, you might need to reinstall the drivers.</p>

username: ishackigozi
cooked: <p>Hello,<br>
Thanks for the response, I just restarted the server and am having different issues.</p>
<p>Python 3.9.0 (default, Nov 15 2020, 14:28:56)<br>
[GCC 7.3.0] :: Anaconda, Inc. on linux<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
/home/i.kigozi/.conda/envs/test/lib/python3.9/site-packages/torch/cuda/<strong>init</strong>.py:83: UserWarning: CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (Triggered internally at  …/c10/cuda/CUDAFunctions.cpp:109.)<br>
return torch._C._cuda_getDeviceCount() &gt; 0<br>
exit()<br>
(test) i.kigozi@R940-01:~$ conda list | grep torch<br>
ffmpeg                    4.3                  hf484d3e_0    pytorch<br>
magma-cuda110             2.5.2                         1    pytorch<br>
pytorch-mutex             1.0                        cuda    pytorch<br>
torch                     1.12.0+cu116             pypi_0    pypi<br>
torchaudio                0.12.0+cu116             pypi_0    pypi<br>
torchvision               0.13.0+cu116             pypi_0    pypi<br>
(test) i.kigozi@R940-01:~$ nvidia-smi<br>
Tue Jul  5 13:17:18 2022<br>
±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 510.73.05    Driver Version: 510.73.05    CUDA Version: 11.6     |</p>
</blockquote>
</blockquote>
</blockquote>
<p>cuda version is 11.6</p>

username: ishackigozi
cooked: <p>On further checking on this, A root user is able to get the expect result<br>
root@R940-01:~# python<br>
Python 3.7.2 (default, Dec 29 2018, 06:19:36)<br>
[GCC 7.3.0] :: Anaconda, Inc. on linux<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
torch.cuda.is_available()<br>
True<br>
exit()</p>
</blockquote>
</blockquote>
</blockquote>
<p>But for a non-root user, they are not able to do the same.</p>
<p>(test) i.kigozi@R940-01:~$ python<br>
Python 3.9.12 (main, Jun  1 2022, 11:38:51)<br>
[GCC 7.5.0] :: Anaconda, Inc. on linux<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
torch.cuda.is_available()<br>
False</p>
</blockquote>
</blockquote>
</blockquote>
<p>Is this a permission issue?</p>

username: ptrblck
cooked: <p>This might be a permission issue, but also note that different Python environments are used.<br>
The root user uses Python 3.7.2 while the non-root user uses Python 3.9.12.<br>
This could also mean that someone installed a CPU-only PyTorch version into the Python 3.9.12 environment.</p>

username: ishackigozi
cooked: <p>Thank you for the response, please kindly advise on how I can investigate this issues.</p>
<p>For the non root user, I follow the instructions in this link</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/get-started/locally/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/get-started/locally/" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://pytorch.org/get-started/locally/" target="_blank" rel="noopener nofollow ugc">PyTorch</a></h3>

  <p>An open source machine learning framework that accelerates the path from research prototyping to production deployment.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>