Why the two GPUs on my machine have the same ID, so that Pytorch can only choose one?

username: ShuokaiPan
cooked: <p>Hi there,</p>
<p>I know my question is not very related to Pytorch, but I was trying to use pytorch on these two GPUs, so I was wondering if anyone could help me out.</p>
<p>I have two GPUs on my machine, one is Quadro k620 and one is Quadro k2200. But for some strange reason, they have the same physical ID. See the outputs from <code>lshw</code> and <code>nvidia-smi</code>.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/cdcea4552e44a10d4cdf6c14b99dd60f9f9cd203.png" data-download-href="https://discuss.pytorch.org/uploads/default/cdcea4552e44a10d4cdf6c14b99dd60f9f9cd203" title="Screenshot from 2017-08-21 11-13-55"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/c/cdcea4552e44a10d4cdf6c14b99dd60f9f9cd203_2_690x322.png" alt="Screenshot from 2017-08-21 11-13-55" width="690" height="322" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/c/cdcea4552e44a10d4cdf6c14b99dd60f9f9cd203_2_690x322.png, https://discuss.pytorch.org/uploads/default/optimized/2X/c/cdcea4552e44a10d4cdf6c14b99dd60f9f9cd203_2_1035x483.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/c/cdcea4552e44a10d4cdf6c14b99dd60f9f9cd203.png 2x" data-dominant-color="3A162E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2017-08-21 11-13-55</span><span class="informations">1324×619 92.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/19ea55b4009182de6d8049b5a3c3d733827b223a.png" data-download-href="https://discuss.pytorch.org/uploads/default/19ea55b4009182de6d8049b5a3c3d733827b223a" title="Screenshot from 2017-08-21 11-14-22"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/1/19ea55b4009182de6d8049b5a3c3d733827b223a_2_690x403.png" alt="Screenshot from 2017-08-21 11-14-22" width="690" height="403" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/1/19ea55b4009182de6d8049b5a3c3d733827b223a_2_690x403.png, https://discuss.pytorch.org/uploads/default/original/2X/1/19ea55b4009182de6d8049b5a3c3d733827b223a.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/1/19ea55b4009182de6d8049b5a3c3d733827b223a.png 2x" data-dominant-color="411E35"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2017-08-21 11-14-22</span><span class="informations">882×516 71.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When I use <code>torch.cuda.device_count()</code>, it tells me that two device are available. But I can only use quadro k2200, no matter how I set <code>torch.cuda.device</code>.  I have also tested in tensorflow, and I can only selected gpu:0. If I tried to select gpu:1, it will report no device gpu:1 is available.</p>
<p>So does anyone know how can make the two GPUs have different IDs?</p>
<p>Thanks a lot for your help.<br>
Shuokai</p>

username: QuantScientist
cooked: <p>The output from your <code>lshw</code> command shows the SAME physical ID fo both of the GPU’s. Did you see that?</p>
<p>Run this code:</p>
<pre><code class="lang-auto">import torch
import sys
print('__Python VERSION:', sys.version)
print('__pyTorch VERSION:', torch.__version__)
print('__CUDA VERSION')
from subprocess import call
# call(["nvcc", "--version"]) does not work
! nvcc --version
print('__CUDNN VERSION:', torch.backends.cudnn.version())
print('__Number CUDA Devices:', torch.cuda.device_count())
print('__Devices')
call(["nvidia-smi", "--format=csv", "--query-gpu=index,name,driver_version,memory.total,memory.used,memory.free"])
print('Active CUDA Device: GPU', torch.cuda.current_device())

print ('Available devices ', torch.cuda.device_count())
print ('Current cuda device ', torch.cuda.current_device())
</code></pre>

username: ShuokaiPan
cooked: <p>HI Solomon K,</p>
<p>Thanks a lot for your reply. Yes I do see that my two GPUs have the same physical ID, and this is exactly where I am confused about. Why are they assigned the same ID?</p>
<p>The output from your code is following:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/5/539246bbb6c730b1cbcffe20d769dd33d20fae82.png" alt="Screenshot from 2017-08-21 17-44-24" width="556" height="248"></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/690c32b69c1be6d201117f2312face3bd980d1a1.png" data-download-href="https://discuss.pytorch.org/uploads/default/690c32b69c1be6d201117f2312face3bd980d1a1" title="Screenshot from 2017-08-21 17-44-48"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/6/690c32b69c1be6d201117f2312face3bd980d1a1_2_690x114.png" alt="Screenshot from 2017-08-21 17-44-48" width="690" height="114" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/6/690c32b69c1be6d201117f2312face3bd980d1a1_2_690x114.png, https://discuss.pytorch.org/uploads/default/optimized/2X/6/690c32b69c1be6d201117f2312face3bd980d1a1_2_1035x171.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/6/690c32b69c1be6d201117f2312face3bd980d1a1.png 2x" data-dominant-color="3E1C32"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2017-08-21 17-44-48</span><span class="informations">1334×222 49.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Sorry the part of your code that run commandline commands does not work and I have not figure our how to make it work, so I just run them separately. You can see, it seems nvidia labels them differently, but lshw labels them as having the same ID. What could be wrong in this case?</p>
<p>Thanks,<br>
Shuokai</p>

username: QuantScientist
cooked: <p>Strange …<br>
Run this and report the output:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/QuantScientist/Deep-Learning-Boot-Camp/blob/master/docker/deps_nvidia_docker.sh" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/QuantScientist/Deep-Learning-Boot-Camp/blob/master/docker/deps_nvidia_docker.sh" target="_blank" rel="nofollow noopener">QuantScientist/Deep-Learning-Boot-Camp/blob/master/docker/deps_nvidia_docker.sh</a></h4>
<pre><code class="lang-sh">#!/usr/bin/env bash

apt-get install nvidia-modprobe

# curl -O -s https://raw.githubusercontent.com/minimaxir/keras-cntk-docker/master/deps_nvidia_docker.sh
if lspci | grep -i 'nvidia'
then
  echo "\nNVIDIA GPU is likely present."
else
  echo "\nNo NVIDIA GPU was detected. Exiting ...\n"
  exit 1
fi

echo "\nChecking for NVIDIA drivers ..."      
# Check for CUDA and try to install.
if ! dpkg-query -W cuda; then
  # The 16.04 installer works with 16.10.
  curl -O http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_8.0.61-1_amd64.deb
  dpkg -i ./cuda-repo-ubuntu1604_8.0.61-1_amd64.deb
  apt-get update
</code></pre>

  This file has been truncated. <a href="https://github.com/QuantScientist/Deep-Learning-Boot-Camp/blob/master/docker/deps_nvidia_docker.sh" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: ShuokaiPan
cooked: <p>Hi,</p>
<p>The output from your script is the following,</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/301a86a1119ebcb2d591f37e3f8bf859e34fe8a5.PNG" data-download-href="https://discuss.pytorch.org/uploads/default/301a86a1119ebcb2d591f37e3f8bf859e34fe8a5" title="save"><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/301a86a1119ebcb2d591f37e3f8bf859e34fe8a5.PNG" alt="save" width="460" height="500" data-dominant-color="131313"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">save</span><span class="informations">865×940 34.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Seems to be similar to the outputs I got previously. Can you see if anything is wrong?</p>
<p>Cheers,<br>
Shuokai</p>

username: FuriouslyCurious
cooked: <p>Reinstall NVidia driver in the HOST OS.</p>
<p>When you swap graphics card or remove and change anything physically, the driver does NOT update device info automatically.  You have to rerun the driver installer.</p>
<p>Let me know if this solves your issue.</p>

username: QuantScientist
cooked: <p><a class="mention" href="/u/furiouslycurious">@FuriouslyCurious</a> may be right.</p>

username: ShuokaiPan
cooked: <p>HI guys,</p>
<p>Thanks a lot for your help. Yes, indeed I recently added another GPU.  I will reinstall Nvidia driver now. But could I ask how to do this in Ubuntu? The problem is I also have Cuda 8.0 installed, will reinstalling Nvidia driver makes me reinstall Cuda 8.0?  Please see the pic below to see Nvidia packages in my machine.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/95b3023a0b793f292c7b6e865b0a0002e45b5cc8.png" data-download-href="https://discuss.pytorch.org/uploads/default/95b3023a0b793f292c7b6e865b0a0002e45b5cc8" title="Screenshot from 2017-08-22 09-24-58"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/9/95b3023a0b793f292c7b6e865b0a0002e45b5cc8_2_690x246.png" alt="Screenshot from 2017-08-22 09-24-58" width="690" height="246" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/9/95b3023a0b793f292c7b6e865b0a0002e45b5cc8_2_690x246.png, https://discuss.pytorch.org/uploads/default/optimized/2X/9/95b3023a0b793f292c7b6e865b0a0002e45b5cc8_2_1035x369.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/9/95b3023a0b793f292c7b6e865b0a0002e45b5cc8_2_1380x492.png 2x" data-dominant-color="3B162E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2017-08-22 09-24-58</span><span class="informations">1483×530 115 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have searched around, but the answers I found vary. Some say I should use this command,</p>
<pre><code>`sudo apt-get remove --purge nvidia*`
</code></pre>
<p>Some says I should use,</p>
<pre><code>sudo nvidia-uninstall
</code></pre>
<p>To reinstall, I just use,</p>
<pre><code>sudo add-apt-repository ppa:graphics-drivers/ppa
</code></pre>
<p>Is this the correct way? I guess my Cuda 8.0 will not be removed during the process?</p>
<p>Sorry I am relatively new to ubuntu and want confirm before I do this.</p>
<p>Cheers</p>

username: ShuokaiPan
cooked: <p>HI guys,</p>
<p>I have reinstalled the nvidia driver following the my previous post. But the problem remains. I can still only use one GPU, and lshw gives the info exactly as before. Both GPU have the same physical ID, although nvidia-smi labels them differently.</p>
<p>I also reboot my machine after reinstall Nvidia driver. Strange…<br>
What could I do then?<br>
Cheers</p>

username: QuantScientist
cooked: <p>Can you change their location in the motherboard?</p>

username: ShuokaiPan
cooked: <p>THere are only two available slots for PCI 16, so I switch the position for them?</p>

username: ShuokaiPan
cooked: <p>Also when use the following command,</p>
<pre><code>ubuntu-drivers devices
</code></pre>
<p>Here is what I got,</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/62e6e9e939468a9fe149bf0af7eb0c5c2fc1cb3d.png" data-download-href="https://discuss.pytorch.org/uploads/default/62e6e9e939468a9fe149bf0af7eb0c5c2fc1cb3d" title="Screenshot from 2017-08-22 10-31-32"><img src="https://discuss.pytorch.org/uploads/default/original/2X/6/62e6e9e939468a9fe149bf0af7eb0c5c2fc1cb3d.png" alt="Screenshot from 2017-08-22 10-31-32" width="690" height="295" data-dominant-color="411F36"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2017-08-22 10-31-32</span><span class="informations">810×347 51.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It seems Ubuntu can only find one GPU which is K2200 in this case.</p>

username: FuriouslyCurious
cooked: <p><a class="mention" href="/u/shuokaipan">@ShuokaiPan</a>   Don’t bother with NVidia repositories and apt-get.  There is a better way.</p>
<p>Download the Ubuntu 16.04 drivers from NVidia website here:<br>
<a href="http://www.nvidia.com/object/unix.html" class="onebox" target="_blank" rel="nofollow noopener">http://www.nvidia.com/object/unix.html</a><br>
It is a “run” file, so you just run it with bash and it will do a full uninstall and reinstall of the driver.</p>
<p>Then download NVidia CUDA Docker image here and run “nvidia-smi” inside the Docker image:<br>
<a href="https://hub.docker.com/r/nvidia/cuda/tags/" class="onebox" target="_blank" rel="nofollow noopener">https://hub.docker.com/r/nvidia/cuda/tags/</a></p>
<p>Let us know what the output shows.</p>

username: ShuokaiPan
cooked: <p>Hi <a class="mention" href="/u/furiouslycurious">@FuriouslyCurious</a></p>
<p>Thanks for the help. I will try that when convenient these days. But what is the difference between installing CUDA on ubuntu natively and installing the CUDA docker image?</p>
<p>Cheers</p>

username: QuantScientist
cooked: <p>As I stated above, if you use nvidia-docker (as opposed to plain docker), you only need the drivers on the HOST machine.<br>
Refer to my scripts above again.</p>

username: Moon_Lockwood
cooked: <p>If anyone else runs across this, I think that they are on separate pci busses (pci@0000:03 and 04) and each is device 0 on it’s bus (03:00.0 and 04:00.0). I might be wrong!</p>