CUDA initialization: Unexpected error from cudaGetDeviceCount(),Torch.cuda.is_available() returns False with cuda11.6 or (cuda10.2),How do I solve it?

LUO77123: <p>I created multiple environments based on Cuda10.2 and Cuda11.6 and reported the same error on the A100 card。 CUDA initialization: Unexpected error from cudaGetDeviceCount(),How do I solve it?<br>
nvidia-smi =11.6, A100<br>
1.export Cuda10.2-&gt;nvcc -V=10.2<br>
conda create -n XXX python=3.9<br>
conda activate XXX<br>
conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=10.2 -c pytorch<br>
python<br>
import torch<br>
torch.cuda.is_available()<br>
return False<br>
2.export Cuda11.6-&gt;nvcc -V=11.6<br>
conda create -n XXX python=3.9<br>
conda activate XXX<br>
conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge (or) pip3 install torch torchvision torchaudio --extra-index-url <a href="https://download.pytorch.org/whl/cu116" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu116</a><br>
python<br>
import torch<br>
torch.cuda.is_available()<br>
return False</p>
<p>Such errors are reported（UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 802: system not yet initialized (Triggered internally at /opt/conda/conda-bld/pytorch_1639180549130/work/c10/cuda/CUDAFunctions.cpp:112.)）</p>
<p>Version:<br>
eg:nvcc=11.6,nvidia-smi=11.6,pytorch=1.12<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f34844bf944bc5ef3608520eb9f5e32b9bd6c80f.png" data-download-href="https://discuss.pytorch.org/uploads/default/f34844bf944bc5ef3608520eb9f5e32b9bd6c80f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f34844bf944bc5ef3608520eb9f5e32b9bd6c80f.png" alt="image" data-base62-sha1="yIaRjIR99nPJQgJVVmNbFYApCbl" width="488" height="500" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">843×863 23.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>It seems your driver has trouble to initialize the device:</p>
<pre><code class="lang-auto">UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 802: system not yet initialized
</code></pre>
<p>Was this setup working before at all? If so, what changed which might cause the issue (e.g. did you update the drivers etc.)?</p>

LUO77123: <p>This is a newly launched server. I tried to call GPU, and the above problem occurred.</p>

ptrblck: <p>OK, I see. In this case you could check if any other CUDA sample would run (I would expect to see the same or similar errors) and might need to reinstall the drivers in this case.</p>

bask0: <p>Any news here? We are having exactly the same problem, A40s work well but not A100s. The configuration of A40 and A100 nodes are identical. I tested with different PyTorch versions (1.12, 1.11, 1.10).</p>

Mollylulu: <p>seem error, have you solved this problem?</p>

ptrblck: <p>No updates, as <a class="mention" href="/u/luo77123">@LUO77123</a> didn’t follow up and the error is usually raised by a config error.<br>
Check if any CUDA applications can run and make sure the right PyTorch binary with the desired CUDA runtime is installed.</p>

bask0: <p>This did solve the issue (missing nvidia-fabricmanager): <a href="https://forums.fast.ai/t/notes-on-using-nvidia-a100-40gb/89894" class="inline-onebox" rel="noopener nofollow ugc">Notes on using NVIDIA A100 (40GB) - Deep Learning - fast.ai Course Forums</a></p>

qgallouedec: <p>Reboot worked for me.</p>

zhanwenchen: <p>I had a problem with a 4xA100 box where both host CUDA samples and docker PyTorch self-contained binaries returned cudaGetDeviceCount errors. I just figured out the root cause: A100 has MIG mode enabled by default, thus making getDeviceCount() error out.</p>
<p>I solved it by using <code>sudo nvidia-smi -mig 0</code> to disable MIG mode, reboot since it was pending (perhaps due to loginctl linger settings), and the do <code>sudo nvidia-smi -mig 0</code> again.</p>

Dj_Jag: <p>I had this problem 3 month, no solution (ERROR 802 in Cuda Samples) Cuda 11.8, installed Nvidia drivers and toolkit<br>
Fabric manager  - unuseless</p>