Invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:59

username: Yang-HangWA
cooked: <p>I have studyed similar topic method to solve the problem, but nothing change. I use the code in <a href="https://discuss.pytorch.org/t/i-have-3-gpu-why-torch-cuda-device-count-only-return-1/7245" class="inline-onebox">I have 3 gpu, why torch.cuda.device_count() only return '1'</a>.<br>
Run the script below:</p>
<pre><code class="lang-auto">import torch
import sys
print('__Python VERSION:', sys.version)
print('__pyTorch VERSION:', torch.__version__)
print('__CUDA VERSION')
from subprocess import call
# call(["nvcc", "--version"]) does not work
#! nvcc --version
print('__CUDNN VERSION:', torch.backends.cudnn.version())
print('__Number CUDA Devices:', torch.cuda.device_count())
print('__Devices')
call(["nvidia-smi", "--format=csv", "--query-gpu=index,name,driver_version,memory.total,memory.used,memory.free"])
print('Active CUDA Device: GPU', torch.cuda.current_device())

print ('Available devices ', torch.cuda.device_count())
print ('Current cuda device ', torch.cuda.current_device())
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">__Python VERSION: 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31) 
[GCC 7.3.0]
__pyTorch VERSION: 1.4.0
__CUDA VERSION
__CUDNN VERSION: 7603
__Number CUDA Devices: 1
__Devices
index, name, driver_version, memory.total [MiB], memory.used [MiB], memory.free [MiB]
0, Tesla V100-PCIE-32GB, 418.56, 32480 MiB, 11 MiB, 32469 MiB
1, Tesla V100-PCIE-32GB, 418.56, 32480 MiB, 0 MiB, 32480 MiB
2, Tesla V100-PCIE-32GB, 418.56, 32480 MiB, 0 MiB, 32480 MiB
3, Tesla V100-PCIE-32GB, 418.56, 32480 MiB, 0 MiB, 32480 MiB
4, Tesla V100-PCIE-32GB, 418.56, 32480 MiB, 0 MiB, 32480 MiB
5, Tesla V100-PCIE-32GB, 418.56, 32480 MiB, 0 MiB, 32480 MiB
6, Tesla V100-PCIE-32GB, 418.56, 32480 MiB, 0 MiB, 32480 MiB
7, Tesla V100-PCIE-32GB, 418.56, 32480 MiB, 0 MiB, 32480 MiB
Active CUDA Device: GPU 0
Available devices  1
Current cuda device  0
</code></pre>
<p>Run:</p>
<pre><code class="lang-auto">import pycuda
from pycuda import compiler
import pycuda.driver as drv

drv.init()
print("%d device(s) found." % drv.Device.count())
           
for ordinal in range(drv.Device.count()):
    dev = drv.Device(ordinal)
    print (ordinal, dev.name())
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">1 device(s) found.
0 Tesla V100-PCIE-32GB
</code></pre>
<p>When I use --gpu=1 or anyother gpu index except 0.<br>
It will  break with message:</p>
<pre><code class="lang-auto">  File "/opt/conda/envs/pytorch-ci/lib/python3.6/site-packages/torch/cuda/__init__.py", line 292, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (101) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:59
</code></pre>
<p>Btw, all the script run in a docker container.</p>
<p>So somebody can help?</p>

username: ptrblck
cooked: <p>Could you check, if you’ve set the environment variable <code>CUDA_VISIBLE_DEVICES</code> to a particular device and if so, remove it?</p>

username: Yang-HangWA
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for your reply. I solve it by unset  CUDA_VISIBLE_DEVICES.</p>

username: ilianherzi
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>! I was wondering if it’s possible to train in tandem by setting CUDA_VISIBLE_DEVICES twice for two separate training scripts? I have a single machine with two GPUs and I’d like to each of them for a separate training task.<br>
For instance what I have now is</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=0,1 python train0.py
</code></pre>
<p>and</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=2,3 python train1.py
</code></pre>

username: ptrblck
cooked: <p>It would be generally possible to execute a script on a specific GPU.<br>
However, since you have two GPUs in your machine, you would have to use:</p>
<pre><code class="lang-python">CUDA_VISIBLE_DEVICES=0 python train0.py
# and
CUDA_VISIBLE_DEVICES=1 python train1.py
</code></pre>
<p>In your current code snippet you would try to use 4 GPUs (2 for each script).</p>

username: ilianherzi
cooked: <p>Hey ptrblck, excuse me, I meant 4 GPUs. I’m currently using pytorch lightning with ddp and I keep getting the aforementioned error.</p>

username: ptrblck
cooked: <p>If you are using <code>CUDA_VISIBLE_DEVICES</code>, the passed GPU indices will be mapped to indices starting at 0 in your script.<br>
E.g. if you are using:</p>
<pre><code class="lang-python">CUDA_VISIBLE_DEVICES=2,3 python train1.py
</code></pre>
<p>you would have to use <code>'cuda:0'</code> and <code>'cuda:1'</code> inside <code>train1.py</code> instead of <code>'cuda:2'</code> and <code>'cuda:3'</code>.<br>
Could this be the issue in your script?</p>

username: ilianherzi
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Unfortunately no,  I don’t think so b/c in train1.py the devices are determined programmatically in the pytorch lightning pl.LightningModule by calling self.device.</p>
<p>Any other ideas why this error might be occuring?<br>
Thanks!</p>

username: ptrblck
cooked: <p>I’m not familiar with Lightning so pinging <a class="mention" href="/u/williamfalcon">@williamFalcon</a> in case he’s seen something similar.</p>
<p>Are you seeing the same issue using “plain PyTorch” and if so could you post a code snippet we could debug?</p>