torch.cuda.DeferredCudaCallError: CUDA call failed lazily at initialization with error: device >= 0 && device < num_gpus

username: Edenzzzz
cooked: <p>This is thrown whenever I run anything without specifying CUDA_VISIBLE_DEVICES.<br>
Full stack trace:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.cuda.device()
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
TypeError: __init__() missing 1 required positional argument: 'device'
&gt;&gt;&gt; torch.cuda.current_device()
Traceback (most recent call last):
  File "/nobackup/wenxuan/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/torch/cuda/__init__.py", line 260, in _lazy_init
    queued_call()
  File "/nobackup/wenxuan/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/torch/cuda/__init__.py", line 145, in _check_capability
    capability = get_device_capability(d)
  File "/nobackup/wenxuan/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/torch/cuda/__init__.py", line 381, in get_device_capability
    prop = get_device_properties(device)
  File "/nobackup/wenxuan/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/torch/cuda/__init__.py", line 399, in get_device_properties
    return _get_device_properties(device)  # type: ignore[name-defined]
RuntimeError: device &gt;= 0 &amp;&amp; device &lt; num_gpus INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1682343964576/work/aten/src/ATen/cuda/CUDAContext.cpp":50, please report a bug to PyTorch. 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/nobackup/wenxuan/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/torch/cuda/__init__.py", line 674, in current_device
    _lazy_init()
  File "/nobackup/wenxuan/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/torch/cuda/__init__.py", line 264, in _lazy_init
    raise DeferredCudaCallError(msg) from e
torch.cuda.DeferredCudaCallError: CUDA call failed lazily at initialization with error: device &gt;= 0 &amp;&amp; device &lt; num_gpus INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1682343964576/work/aten/src/ATen/cuda/CUDAContext.cpp":50, please report a bug to PyTorch. 

CUDA call was originally invoked at:

['  File "&lt;stdin&gt;", line 1, in &lt;module&gt;\n', '  File "&lt;frozen importlib._bootstrap&gt;", line 1007, in _find_and_load\n', '  File "&lt;frozen importlib._bootstrap&gt;", line 986, in _find_and_load_unlocked\n', '  File "&lt;frozen importlib._bootstrap&gt;", line 680, in _load_unlocked\n', '  File "&lt;frozen importlib._bootstrap_external&gt;", line 850, in exec_module\n', '  File "&lt;frozen importlib._bootstrap&gt;", line 228, in _call_with_frames_removed\n', '  File "/nobackup/wenxuan/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/torch/__init__.py", line 1146, in &lt;module&gt;\n    _C._initExtension(manager_path())\n', '  File "&lt;frozen importlib._bootstrap&gt;", line 1007, in _find_and_load\n', '  File "&lt;frozen importlib._bootstrap&gt;", line 986, in _find_and_load_unlocked\n', '  File "&lt;frozen importlib._bootstrap&gt;", line 680, in _load_unlocked\n', '  File "&lt;frozen importlib._bootstrap_external&gt;", line 850, in exec_module\n', '  File "&lt;frozen importlib._bootstrap&gt;", line 228, in _call_with_frames_removed\n', '  File "/nobackup/wenxuan/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/torch/cuda/__init__.py", line 197, in &lt;module&gt;\n    _lazy_call(_check_capability)\n', '  File "/nobackup/wenxuan/miniconda3/envs/pytorch3d/lib/python3.9/site-packages/torch/cuda/__init__.py", line 195, in _lazy_call\n    _queued_calls.append((callable, traceback.format_stack()))\n']
</code></pre>
<p>Environment:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8e67a258dea537ba4f03bd6f767cfca181694f7a.png" data-download-href="https://discuss.pytorch.org/uploads/default/8e67a258dea537ba4f03bd6f767cfca181694f7a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8e67a258dea537ba4f03bd6f767cfca181694f7a.png" alt="image" data-base62-sha1="kjLPPwPTOq1qoi3fjpZVh4IH4Iq" width="424" height="500" data-dominant-color="292929"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">668×787 30.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Your GPU5 is in an error state so try to fix it before digging into PyTorch.</p>

username: ite-jin
cooked: <p>Is your problem solved? I ran into the same problem and can’t solve it now</p>

username: yt_Liu
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="186769">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>GPU5 is in an error state</p>
</blockquote>
</aside>
<p>I also encountered this problem, how do you solve it?</p>

username: ptrblck
cooked: <p>I don’t know what’s causing the issue but I would check <code>dmesg</code> for any <code>Xid</code>s, reboot the workstation, and reinstall the driver.</p>