Pytorch installation issue, NVIDIA Driver 470, CUDA 11.3

mih30: <p>I have been working on a server with a single A100 GPU. I installed NVIDIA driver 470.141.03, and nvidia-smi runs fine, showing the expected output, including CUDA Version: 11.4. I then proceeded to create a python venv and install pytorch using the following command:</p>
<p><code>pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113</code>.</p>
<p>When I launch python and run the <code>torch.cuda.is_available</code> command I get the following output:</p>
<pre><code class="lang-auto">/home/azureuser/michael/ml-toolkit-env/lib/python3.8/site-packages/torch/cuda/__init__.py:80: UserWarning: CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (Triggered internally at  ../c10/cuda/CUDAFunctions.cpp:112.)
  return torch._C._cuda_getDeviceCount() &gt; 0
False
</code></pre>
<p>When I run <code>torch.version.cuda</code> I get 11.3, as expected.</p>
<p>Has anyone run across a similar issue in the past? Any suggestions on how to resolve this?</p>
<p>More detail: this is happening on a cloud VM in Azure.</p>

ptrblck: <p>Your setup seems to have trouble communicating with the NVIDIA driver so make sure you can compile and execute e.g. the CUDA samples first.</p>

mih30: <p>Hi,</p>
<p>It seems you’re correct, the CUDA samples won’t work either. For example, running <a href="https://github.com/zchee/cuda-sample/blob/master/1_Utilities/deviceQuery/deviceQuery.cpp#L68" rel="noopener nofollow ugc">deviceQuery</a> fails with error code 3. According to <a href="https://docs.nvidia.com/cuda/archive/10.1/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1g3f51e3575c2178246db0a94a430e0038" rel="noopener nofollow ugc">this document</a> it would appear that the CUDA driver and runtime could not be initialized. Considering the fact that we have a compatible driver installed as confirmed by nvidia-smi, could this indicate a hardware failure? Or could it also be something else?</p>
<p>Thanks for the help!</p>
<p>Best Regards,<br>
Michael</p>

ptrblck: <p>I doubt it’s a hardware defect and would probably just reinstall the driver.</p>

mih30: <p>Hi,</p>
<p>We have tried to reinstall the driver multiple times, and tried several versions (515, 520, 450). All seem to result in failure. We are using <a href="https://github.com/Azure/azhpc-images/tree/master/ubuntu/ubuntu-20.x/ubuntu-20.04-hpc" rel="noopener nofollow ugc">this</a> as our base image. It’s an official Azure Ubuntu 20.04 image that comes pre-packages with many of the libraries required for HPC, and comes pre-installed with what ostensibly is a correct driver and CUDA combination. We have tried several driver versions, restoring it to the original image after each unsuccessful attempt. The error code is always identical.</p>
<p>Is there anything besides a driver reinstall we could try?</p>
<p>Kind Regards,<br>
Michael</p>

ptrblck: <p>As It seems to be quite system-specific and unrelated to PyTorch, I would probably contact your system admin or vendor who maintains these nodes.</p>

mih30: <p>Understood, thanks! I will post here what I discover after I work through this error.</p>

mih30: <p>Hi,</p>
<p>We have resolved the issue. It appears that having MIG enabled was causing the issue. We have disabled it using the following commands and it worked:</p>
<pre><code class="lang-auto">sudo nvidia-smi -i 0 -mig 0
sudo nvidia-smi -i 0 --query-gpu=pci.bus_id,mig.mode.current --format=csv
sudo reboot
</code></pre>
<p>If anyone comes across a similar issue, this MAY help.</p>
<p>Kind Regards,<br>
Michael</p>

spro: <p>Disabling MIG worked for me! I was using a cloud VM from Lambda Labs</p>