Nvidia driver & cudatoolkit installed properly but check_driver fails

username: Ranahanocka
cooked: <p>I have successfully installed NVIDIA driver &amp; cudatoolkit via conda. However, I am not able to use cuda in pytorch (even though it installed successfully).</p>
<p>Previously, I was using Pytorch with CUDA 8.0, and wanted to upgrade. I removed / purge all CUDA through:</p>
<pre><code class="lang-auto">sudo apt-get --purge remove cuda
sudo apt-get autoremove
dpkg --list |grep "^rc" | cut -d " " -f 3 | xargs sudo dpkg --purge
</code></pre>
<p>Then I updated my Nvidia drivers to 4.10 via PPA (Ubuntu 16.04):</p>
<pre><code class="lang-auto">sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt-get update
sudo apt-get install nvidia-410
</code></pre>
<p>Everything worked smoothly. The output of <code>nvidia-smi</code>:</p>
<pre><code class="lang-auto">Fri Aug 23 22:29:48 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.78       Driver Version: 410.78       CUDA Version: N/A      |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:01:00.0  On |                  N/A |
| 25%   35C    P8    13W / 250W |    531MiB / 11177MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1445      G   /usr/lib/xorg/Xorg                           317MiB |
|    0      2035      G   compiz                                       101MiB |
|    0      3572      G   ...uest-channel-token=13099850080781834209   110MiB |
+-----------------------------------------------------------------------------+
</code></pre>
<p>The output of <code>cat /proc/driver/nvidia/version</code></p>
<pre><code class="lang-auto">NVRM version: NVIDIA UNIX x86_64 Kernel Module  410.78  Sat Nov 10 22:09:04 CST 2018
GCC version:  gcc version 4.9.4 (Ubuntu 4.9.4-2ubuntu1~16.04)
</code></pre>
<p>Since I wanted conda to manage my CUDA version, I installed the cudatoolkit through conda env (python 3.6):</p>
<p><code>conda install pytorch torchvision cudatoolkit=10.0 -c pytorch</code></p>
<p>again, everything installs perfectly. When I run:</p>
<pre><code class="lang-auto">print(torch.cuda.device_count()) # --&gt; 0
print(torch.version.cuda) # --&gt; 10.0.130
</code></pre>
<p>but using cuda fails. I get the following error message</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/home/rana/anaconda3/envs/py36torch12cu10/lib/python3.6/site-packages/torch/cuda/__init__.py", line 178, in _lazy_init
    _check_driver()
  File "/home/rana/anaconda3/envs/py36torch12cu10/lib/python3.6/site-packages/torch/cuda/__init__.py", line 99, in _check_driver
    http://www.nvidia.com/Download/index.aspx""")
AssertionError: 
Found no NVIDIA driver on your system. Please check that you
have an NVIDIA GPU and installed a driver from
http://www.nvidia.com/Download/index.aspx
</code></pre>
<p>I restarted, removed all irrelevant environment variables which may have caused issues (LD_LIBRARY_PATH), removed conda, reinstalled, tried cuda 9.2, but nothing works. I am not sure what the issue could be. Any ideas?</p>
<p>I searched a bit, and found <a href="https://discuss.pytorch.org/t/found-no-nvidia-driver-on-your-system-but-its-there/35063">this</a> pytorch thread. Since I completely removed CUDA from my system this shouldn’t be the problem, but I think somehow it may be related.</p>
<p><strong>EDIT:</strong><br>
It isn’t surprising given my error, but following <a href="https://github.com/pytorch/pytorch/issues/4546" rel="nofollow noopener">this issue</a>, I checked:<br>
<code>torch._C._cuda_getDriverVersion() # -&gt; 0</code></p>

username: ptrblck
cooked: <p>That’s some good debugging.<br>
Could you post the output of <code>dpkg -l | grep -i nvidia</code>?<br>
Probably unrelated to this issue, but are you using a secure boot option?</p>

username: Ranahanocka
cooked: <p>Good call… the output of <code>dpkg -l | grep -i nvidia</code> is</p>
<pre><code class="lang-auto">ii  bbswitch-dkms                              0.8-3ubuntu1                                 amd64        Interface for toggling the power on NVIDIA Optimus video cards
rc  nvidia-384                                 384.90-0ubuntu0.16.04.1                      amd64        NVIDIA binary driver - version 384.90
hi  nvidia-410                                 410.78-0ubuntu0~gpu16.04.1                   amd64        NVIDIA binary driver - version 410.78
rc  nvidia-opencl-icd-384                      384.90-0ubuntu0.16.04.1                      amd64        NVIDIA OpenCL ICD
ii  nvidia-prime                               0.8.2                                        amd64        Tools to enable NVIDIA's Prime
ii  nvidia-settings                            384.81-0ubuntu1                              amd64        Tool for configuring the NVIDIA graphics driver
</code></pre>
<p>very odd. so it seems the old driver (384) is still around. What do you think the best way to fix this is?<br>
About secure boot: I don’t think I changed the bootloader, and I am not running dualboot…</p>

username: ptrblck
cooked: <p>Thanks for the information.<br>
Based on the codes, it looks like <code>384.81</code> is still installed (at least <code>nvidia-settings</code>) and still contains config files. I would recommend to purge all drivers, and reinstall the latest (or desired) one.</p>

username: Ranahanocka
cooked: <p>Yay, it works! Posting my solution:</p>
<p>Just purged nvidia by running:</p>
<pre><code class="lang-auto">sudo apt-get remove --purge '^nvidia-.*'
</code></pre>
<p>after reinstalling 410 via ppa, the output of <code>dpkg -l | grep -i nvidia</code> is:</p>
<pre><code class="lang-auto">ii  bbswitch-dkms                              0.8-3ubuntu1                                 amd64        Interface for toggling the power on NVIDIA Optimus video cards
ii  libcuda1-410                               410.78-0ubuntu0~gpu16.04.1                   amd64        NVIDIA CUDA runtime library
hi  nvidia-410                                 410.78-0ubuntu0~gpu16.04.1                   amd64        NVIDIA binary driver - version 410.78
ii  nvidia-opencl-icd-410                      410.78-0ubuntu0~gpu16.04.1                   amd64        NVIDIA OpenCL ICD
ii  nvidia-prime                               0.8.2                                        amd64        Tools to enable NVIDIA's Prime
ii  nvidia-settings                            418.56-0ubuntu0~gpu16.04.1                   amd64        Tool for configuring the NVIDIA graphics driver
</code></pre>
<p>odd that nvidia settings is 418, but anyway it works. Also I used</p>
<pre><code class="lang-auto">sudo apt-mark hold nvidia-410
</code></pre>
<p>to make sure the driver won’t update with <code>sudo apt-get update</code>.</p>

username: ptrblck
cooked: <p>Good to hear it’s working! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Yangmei_Shen
cooked: <p>Thanks God! That helps me.</p>