Setting visible devices with Distributed Data Parallel

Diego: <p>Is it possible make CUDA_VISIBLE_DEVICES and DDP work together?</p>
<p>I am trying to run a script on an 8 GPU server like so:</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=0,2,3,4,5,6,7 python -m torch.distributed.launch --nproc_per_node=7 --use_env main.py
</code></pre>
<p>but I always run into:</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: invalid device ordinal
</code></pre>
<p>Here is the output of <code>nvidiia-smi</code>:</p>
<pre><code class="lang-auto">ue Aug 18 15:21:16 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  On   | 00000000:04:00.0 Off |                  N/A |
| 20%   13C    P8     7W / 235W |      0MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 108...  On   | 00000000:05:00.0 Off |                  N/A |
| 23%   18C    P8     8W / 235W |      0MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 108...  On   | 00000000:08:00.0 Off |                  N/A |
| 23%   20C    P8     8W / 235W |      0MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX 108...  On   | 00000000:09:00.0 Off |                  N/A |
| 23%   23C    P8     8W / 235W |      0MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  GeForce GTX 108...  On   | 00000000:84:00.0 Off |                  N/A |
| 23%   18C    P8    11W / 235W |      0MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  GeForce GTX 108...  On   | 00000000:85:00.0 Off |                  N/A |
| 20%   16C    P8     7W / 235W |      0MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   6  GeForce GTX 108...  On   | 00000000:88:00.0 Off |                  N/A |
| 20%   15C    P8     7W / 235W |      0MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   7  GeForce GTX 108...  On   | 00000000:89:00.0 Off |                  N/A |
| 23%   25C    P8     7W / 235W |      0MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

</code></pre>
<p>What am I missing?</p>

mrshenli: <p>Hey <a class="mention" href="/u/diego">@Diego</a>, the launching script will launch multiple sub-processes, which might be inherit the  <code>CUDA_VISIBLE_DEVICES</code> value you passed to the command line. A work around would be setting <code>CUDA_VISIBLE_DEVICES</code> in <code>main.py</code> before loading any cuda-related packages. Note that the recommended way to use DDP is one-process-per-device, i.e., each process should exclusively run on one GPU. If you want this, you need to set <code>CUDA_VISIBLE_DEVICES</code> to a different value for each subprocess.</p>
<p>BTW, what’s the default <code>CUDA_VISIBLE_DEVICES</code> value in your machine? I would assume the script should be able to see all devices by default if <code>CUDA_VISIBLE_DEVICES</code> wasn’t set. And when the program throws <code>RuntimeError: CUDA error: invalid device ordinal</code>, do you know which device it tries to access?</p>

Diego: <p>sorry this was a mistake by me. I had set the <code>device_ids</code> variable in the DDP constructor besides using the CUDA_VISIBLE_DEVICES variable, once I removed the former the script runs as expected.</p>