Machine restarted and now PyTorch gives CUDA error

AndrewUlmer: <p>Hi, our machine restarted and now PyTorch is unable to work with CUDA, despite working before the restart. When I type <code>torch.cuda.device_count()</code> I get the following error:</p>
<p><code>/home/username/miniconda3/envs/tvae/lib/python3.8/site-packages/torch/cuda/__init__.py:80: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at  /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1635023442742/work/c10/cuda/CUDAFunctions.cpp:112.)   return torch._C._cuda_getDeviceCount() &gt; 0</code></p>
<p>Below is my PyTorch version, as well as the output of <code>nvidia-smi</code> and <code>nvcc -V</code>. Please let me know if you need any additional information.</p>
<p>Potentially relevant <code>nvidia-smi</code> output:<br>
`Fri Jun 10 12:27:40 2022<br>
±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 455.23.05    Driver Version: 455.23.05    CUDA Version: 11.1     |<br>
|-------------------------------±---------------------±---------------------+<br>
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>
|                               |                      |               MIG M. |<br>
|===============================+======================+======================|<br>
|   0  Quadro RTX 6000     On   | 00000000:17:00.0 Off |                  Off |<br>
| 34%   27C    P8    17W / 260W |      6MiB / 24220MiB |      0%      Default |<br>
|                               |                      |                  N/A |<br>
±------------------------------±---------------------±---------------------+<br>
|   1  Quadro RTX 6000     On   | 00000000:65:00.0  On |                  Off |<br>
| 34%   30C    P8     4W / 260W |    131MiB / 24212MiB |      0%      Default |<br>
|                               |                      |                  N/A |<br>
±------------------------------±---------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------+<br>
| Processes:                                                                  |<br>
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |<br>
|        ID   ID                                                   Usage      |<br>
|=============================================================================|<br>
|    0   N/A  N/A     10225      G   /usr/lib/xorg/Xorg                  4MiB |<br>
|    0   N/A  N/A     10579      G   /usr/bin/gnome-shell                0MiB |<br>
|    1   N/A  N/A     10225      G   /usr/lib/xorg/Xorg                101MiB |<br>
|    1   N/A  N/A     10579      G   /usr/bin/gnome-shell               28MiB |<br>
±----------------------------------------------------------------------------+`</p>
<p>Output of <code>nvcc -V</code>:<br>
<code>nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2019 NVIDIA Corporation Built on Sun_Jul_28_19:07:16_PDT_2019 Cuda compilation tools, release 10.1, V10.1.243</code></p>

AlphaBetaGamma96: <p>Do you use a conda environment? If so you might just need to re-activate it?</p>
<p>Also, do you know what cuda version you used before restarting? <code>nvcc</code> states 10.1 but if you were using say cuda <code>11.X</code> then you might need to respecify your <code>PATH</code> and <code>LD_LIBRARY_PATH</code> variables so PyTorch sees the right cuda install.</p>

AndrewUlmer: <p>I actually restarted the machine and the issue resolved. Thanks for the help, though!</p>

turbotimon: <p>For reference, there is a longer discussion about this problem <a href="https://discuss.pytorch.org/t/userwarning-cuda-initialization-cuda-unknown-error-this-may-be-due-to-an-incorrectly-set-up-environment-e-g-changing-env-variable-cuda-visible-devices-after-program-start-setting-the-available-devices-to-be-zero/129335">here</a>. This <a href="https://askubuntu.com/questions/1228423/how-do-i-fix-cuda-breaking-after-suspend/1309807#1309807" rel="noopener nofollow ugc">link</a> may also help</p>