Gpu devices: nvidia-smi and cuda.get_device_name() output appear inconsistent

prairie-guy: <p>I have three GPU’s and have been trying to set <code>CUDA_VISIBLE_DEVICES</code> in my environment, but am confused by the difference in the ordering of the gpu’s in <code>nvidia-smi</code> and <code>torch.cuda.get_device_name</code> Here is the output of both:</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| NVIDIA-SMI 387.34                 Driver Version: 387.34                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN Xp            Off  | 00000000:02:00.0  On |                  N/A |
| 23%   38C    P8    17W / 250W |    811MiB / 12188MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Graphics Device     Off  | 00000000:03:00.0 Off |                  N/A |
| 34%   49C    P8    26W / 250W |      0MiB / 12058MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Graphics Device     Off  | 00000000:04:00.0 Off |                  N/A |
| 28%   40C    P8    24W / 250W |      0MiB / 12058MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
</code></pre>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.cuda.get_device_name(0)
'Graphics Device'
&gt;&gt;&gt; torch.cuda.get_device_name(1)
'TITAN Xp'
&gt;&gt;&gt; torch.cuda.get_device_name(2)
'Graphics Device'
</code></pre>
<p>I would have expected the device numbers to be consistent across these applications. If not, then what should I expect if I am using another library like keras or tensor-flow?</p>
<p>Thanks in advance.</p>

albanD: <p>Hi,</p>
<p>The device numbering is consistent accros all applications, except <code>nvidia-smi</code> that ignores the <code>CUDA_DEVICE_ORDER</code> environment variable.<br>
The problem is that by default the device ordering is <code>FASTEST_FIRST</code> while <code>nvidia-smi</code> uses <code>PCI_BUS_ID</code>.<br>
To make your applications consistent with <code>nvidia_smi</code>, just add <code>export CUDA_DEVICE_ORDER=PCI_BUS_ID</code> to your bashrc (or equivalent) such that every application uses <code>nvidia-smi</code>'s ordering.</p>

prairie-guy: <p><a class="mention" href="/u/alband">@albanD</a> you are awesome! That’s exactly what I needed. Can’t tell you how much time I spent on this. I’m sure I missed it, in cuda and/or Pytorch related documentation. Thanks again.</p>

Muhammad_Sohail: <p>thanks, it helped me.</p>

raining_day513: <p>either echoing <code>$PCI_BUS_ID</code> or <code>$CUDA_DEVICE_ORDER</code> I got the same result: nothing? So I assume this won’t work in my case.</p>

ptrblck: <p>You are supposed to set the env variable <code>CUDA_DEVICE_ORDER</code> to <code>PCI_BUS_ID</code> as explained in the <a href="https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars">CUDA docs</a>.<br>
<code>echo $CUDA_DEVICE_ORDER</code> might not return anything if you haven’t exported it already.</p>