Torch.cuda.is_available() kept switching to False

Bowen_D: <p>Hi team!</p>
<p>torch.cuda.is_available() kept switching to False on me. Can someone please shed some lights on how to debug this, my setup:</p>
<p>Nvidia Driver version:  510.108.03<br>
CUDA: 11.6<br>
Pytorch: 1.12.1+cu116</p>
<p>I installed pytorch through pip<br>
pip install torch==1.12.1+cu116 --extra-index-url <a href="https://download.pytorch.org/whl/cu116" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu116</a></p>
<p>I also tried <code> python -m torch.utils.collect_env</code>, the output I got when cuda.is_available() = True</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.12.1+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.17

Python version: 3.7.13 (default, Mar 29 2022, 02:18:16)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.13.0-1031-aws-x86_64-with-debian-bullseye-sid
Is CUDA available: True
CUDA runtime version: 11.6.124
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.12.1+cu116
[conda] numpy                     1.21.6                   pypi_0    pypi
[conda] torch                     1.12.1+cu116             pypi_0    pypi
</code></pre>
<p>The output I got when cuda.is_available returned False</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.12.1+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A
OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.17

Python version: 3.7.13 (default, Mar 29 2022, 02:18:16)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.13.0-1031-aws-x86_64-with-debian-bullseye-sid
Is CUDA available: False
CUDA runtime version: 11.6.124
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.12.1+cu116
[conda] numpy                     1.21.6                   pypi_0    pypi
[conda] torch                     1.12.1+cu116             pypi_0    pypi
</code></pre>
<p>Behavior observed:<br>
torch.cuda.is_available() return True for a while and switching to False. I am running this in a Kubernete cluster, I had to bounce the pod to get torch.cuda.is_available() returns True. I am using AWS g4 instance with NVIDIA T4 GPUs.</p>
<p>Thanks in advance!</p>

eqy: <p>Are you observing different behavior on different nodes with the same machine image? In that case I would check if there could be something unusual on the machine e.g., via <code>nvidia-smi</code>.</p>

Bowen_D: <p>I am not observing different behavior on different nodes. <code>torch.cuda.is_available()</code> switch to False on the same node. I was able to reproduce this on different nodes with same machine image.</p>
<p>We don’t have nvidia-smi installed, would that matter for pytorch to detect GPU? I see the following with <code>nvcc --version</code></p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_Mar__8_18:18:20_PST_2022
Cuda compilation tools, release 11.6, V11.6.124
Build cuda_11.6.r11.6/compiler.31057947_0
</code></pre>

eqy: <p>It is not a requirement to use PyTorch, but it may be useful to diagnose e.g., driver or hardware issues if those are the root cause.</p>