RuntimeError: device >= 0 && device < num_gpus INTERNAL ASSERT FAILED

devsy: <p>Hi,<br>
I got the following error in my docker container.<br>
It says CUDA is available but couldn’t get the current_device.<br>
I was able to run nvidia-smi in my container, but even I pass the GPUID to CUDA_VISIBLE_DEVICES, the application cannot catch the device.<br>
Could you give any guidance to solve this problem?<br>
Thanks!</p>
<p>root@XXXX:/project# python<br>
Python 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10)<br>
[GCC 10.3.0] on linux<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
torch.cuda.is_available()<br>
True<br>
torch.cuda.current_device()<br>
Traceback (most recent call last):<br>
File “/opt/conda/lib/python3.8/site-packages/torch/cuda/<strong>init</strong>.py”, line 242, in _lazy_init<br>
queued_call()<br>
File “/opt/conda/lib/python3.8/site-packages/torch/cuda/<strong>init</strong>.py”, line 125, in _check_capability<br>
capability = get_device_capability(d)<br>
File “/opt/conda/lib/python3.8/site-packages/torch/cuda/<strong>init</strong>.py”, line 357, in get_device_capability<br>
prop = get_device_properties(device)<br>
File “/opt/conda/lib/python3.8/site-packages/torch/cuda/<strong>init</strong>.py”, line 375, in get_device_properties<br>
return _get_device_properties(device)  # type: ignore[name-defined]<br>
RuntimeError: device &gt;= 0 &amp;&amp; device &lt; num_gpus INTERNAL ASSERT FAILED at “…/aten/src/ATen/cuda/CUDAContext.cpp”:50, please report a bug to PyTorch.</p>
</blockquote>
</blockquote>
</blockquote>
<p>The above exception was the direct cause of the following exception:</p>
<p>Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/opt/conda/lib/python3.8/site-packages/torch/cuda/<strong>init</strong>.py”, line 552, in current_device<br>
_lazy_init()<br>
File “/opt/conda/lib/python3.8/site-packages/torch/cuda/<strong>init</strong>.py”, line 246, in _lazy_init<br>
raise DeferredCudaCallError(msg) from e<br>
torch.cuda.DeferredCudaCallError: CUDA call failed lazily at initialization with error: device &gt;= 0 &amp;&amp; device &lt; num_gpus INTERNAL ASSERT FAILED at “…/aten/src/ATen/cuda/CUDAContext.cpp”:50, please report a bug to PyTorch.</p>
<p>CUDA call was originally invoked at:</p>
<p>[’  File “”, line 1, in \n’, ’  File “”, line 991, in _find_and_load\n’, ’  File “”, line 975, in _find_and_load_unlocked\n’, ’  File “”, line 671, in _load_unlocked\n’, ’  File “”, line 843, in exec_module\n’, ’  File “”, line 219, in _call_with_frames_removed\n’, ’  File “/opt/conda/lib/python3.8/site-packages/torch/<strong>init</strong>.py”, line 798, in \n    _C._initExtension(manager_path())\n’, ’  File “”, line 991, in _find_and_load\n’, ’  File “”, line 975, in _find_and_load_unlocked\n’, ’  File “”, line 671, in _load_unlocked\n’, ’  File “”, line 843, in exec_module\n’, ’  File “”, line 219, in _call_with_frames_removed\n’, ’  File “/opt/conda/lib/python3.8/site-packages/torch/cuda/<strong>init</strong>.py”, line 179, in \n    _lazy_call(_check_capability)\n’, ’  File “/opt/conda/lib/python3.8/site-packages/torch/cuda/<strong>init</strong>.py”, line 177, in _lazy_call\n    _queued_calls.append((callable, traceback.format_stack()))\n’]</p>

ptrblck: <p>Are you able to run any other CUDA application in your docker container?<br>
Did you make sure to specify the used GPUs via e.g. <code>docker run --gpus all</code>?</p>

cmosguy: <p>I am on a cluster with A100 GPUs, and I am facing the same issue.  Do you have any solutions that can resolve this?</p>

jbm: <p>I’m hitting the same error on an A100 machine. Did you find a way around it?</p>

jbm: <p>What’s confusing about the error is that there doesn’t even seem to be such a property as <code>num_gpus</code>—it’s <code>device_count</code>… weird… clearly seems to suggest a software error of some kind (maybe some deprecated version of something somewhere)</p>

pretzel: <p>Might be different, but thought I’d add in case it helps. I’m not using conda and hit the error somewhat differently.</p>
<p>I got this error on a AWS P4 with 8 A100s this morning in a jupyter notebook. When I ran env variables before I imported torch, it worked:</p>
<pre><code class="lang-auto">import os
os.environ["CUDA_VISIBLE_DEVICES"] = "4" 
os.environ["WORLD_SIZE"] = "1"
import torch
</code></pre>
<p>Seperately, when I ran the exact thing you ran in a interactive session, i got no error:</p>
<pre><code class="lang-auto">Python 3.8.12 (default, Nov  2 2021, 13:56:07) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.0.1+cu118'
&gt;&gt;&gt; torch.cuda.is_available()
True
&gt;&gt;&gt; torch.cuda.current_device()
0
&gt;&gt;&gt; exit()
</code></pre>

devsy: <p>Sorry for late reply,<br>
I was not able to run other CUDA applications so i reinstalled all drivers and container libraries.<br>
I think there was some mismatch between container suites and drivers.</p>

jackie97: <p>It works for me! That’s genius! Thank you soooo much!</p>

xingyu_hu: <p>excellent! this works for me! Thank you for this ,'cause this helps me solve a issue troubling me for several days!</p>

bhatch: <p>I hate myself. It worked for me, too. Mucho thanks <a class="mention" href="/u/pretzel">@pretzel</a>!</p>

gaoliyao: <p>This is genius… It is working amazingly…</p>

gps6101982: <aside class="quote no-group" data-username="pretzel" data-post="6" data-topic="178118">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/5f8ce5/48.png" class="avatar"> pretzel:</div>
<blockquote>
<p><code>os.environ["CUDA_VISIBLE_DEVICES"] = "4" </code></p>
</blockquote>
</aside>
<p>Sets the devices only to GPU number 5 right? what about multiple GPU’s [“0,1,2,3” in the above case] it gives me the same error once I enable more than one GPU. How do I overcome this?</p>

Travis_Pence: <p>This worked for me. I had to set the environment variable before I imported torch</p>

AKT_TARAFDER: <p><a class="mention" href="/u/gps6101982">@gps6101982</a> hello,<br>
I really need to know how did you solve when you assigned options</p>
<pre><code class="lang-auto">import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1,2,34" 
os.environ["WORLD_SIZE"] = "4"
import torch
</code></pre>
<p>I also want to know how did you launch it?<br>
it would really help me out. I am stuck on this for quite some time.</p>

gps6101982: <p><a class="mention" href="/u/akt_tarafder">@AKT_TARAFDER</a> Here is what I did to overcome the issue;hope this helps</p>
<p>I suppose you are working with either DataParallel or DistributedDataParallel libraries and running on a server.<br>
The server can provide GPUs across nodes or on one single node. In case the GPUs are across nodes, Nvidia MPI integrates them. You can use ‘nvidia-smi’ to verify if the GPUs are an MPI cluster. PyTorch seems to have some issues here, and I am unsure how to handle this. Further inputs will be helpful.</p>
<p>The second case is one single node with multiple GPUs.In an SBATCH,  a single node option can be requested using the -N option.</p>
<pre><code class="lang-auto">#SBATCH -N 1  #request for one node.
</code></pre>
<p>here, I <strong>didn’t need the environ variable</strong><br>
with DataParallel, the usual python command ran it well</p>
<pre><code class="lang-auto">python train.py
</code></pre>
<p>With DistributedDataParallel, torchrun worked.</p>
<pre><code class="lang-auto">torchrun --nproc_per_node=#GPUs --nnodes=1 train.py
</code></pre>
<p>Further inputs on MPI will be really helpful.</p>

AKT_TARAFDER: <p><a class="mention" href="/u/gps6101982">@gps6101982</a> Thank you for the reply. But I am doing some work, where I required to make sure than each script has the access to all the GPUs in the multi-node system, But specifically work on. a particular tensor slice according to rank/ place a tensor on a GPU(identified by the rank)</p>
<p>I could not use Data Parallel, because I required manual computation handling in each GPU.<br>
but for now I really need something that can be launched using mpi4y and I can manually handle all thread having individual control of each GPU, and having access to all of them.</p>
<p>torchrun does not allow that much control as mpi4py does, The problem I am facing is that I am trying to access specific ranks here:</p>
<pre><code class="lang-auto">from mpi4py import MPI
import os


world_comm = MPI.COMM_WORLD
my_rank = world_comm.Get_rank()

os.environ["CUDA_VISIBLE_DEVICES"] ='0,1' 
os.environ["WORLD_SIZE"] = "2"

import torch

#import os
#os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
#os.environ["CUDA_VISIBLE_DEVICES"]="0,1"

if __name__ == "__main__":
    world_comm = MPI.COMM_WORLD
    world_size = world_comm.Get_size()
    my_rank = world_comm.Get_rank()

    # Get the number of GPUs and assign one based on rank
    num_gpus = torch.cuda.device_count()
    print(f'num_gpus:{num_gpus}')
    device_id = my_rank % num_gpus  # Cycle through available GPUs
    print(f"num_gpus: {num_gpus} , device_id:{device_id}, my_rank:{my_rank}")
    
    #current_device=torch.cuda.current_device()
    #print(f'current_device:{current_device}')

    # Set the current device for this process
    torch.cuda.set_device(device_id)

    print(f"World Size: {world_size}   Rank: {my_rank}   Assigned GPU: {device_id}")

    # Create a tensor and move it to the assigned GPU
    tensor = torch.randn((2, 2), device=f'cuda:{my_rank}')
    print(f"Rank {my_rank} is running on GPU {device_id} with tensor:\n{tensor.device}")
</code></pre>
<p>the command:</p>
<pre><code class="lang-auto">mpiexec -n 2 python mpi3.py 
</code></pre>
<p>Here each process should access different separate GPUS.</p>
<p>If I only assign 0 in the CUDA_VISIBLE_DEVICES if works fine, otherwise it runs into that runtime error discussed here in this thread.</p>

fsm_1506: <p>Try setting the env variable like so</p>
<pre><code class="lang-auto">os.environ["CUDA_VISIBLE_DEVICES"] = "0,1,2,3" 
</code></pre>

gps6101982: <p>This didn’t help with MPI enabled</p>