One GPU lost, how to train on another without reboot?

Mart.Slaaf: <p>Hello!</p>
<p>I have 4 GPU on my computer. And I have one lost:</p>
<pre><code class="lang-auto">$&gt;nvidia-smi
Unable to determine the device handle for GPU 0000:86:00.0: GPU is lost.
</code></pre>
<p>But another one is still working. e.g.</p>
<pre><code class="lang-auto">$&gt;nvidia-smi -i 0
</code></pre>
<p>works good.</p>
<p>For the some reasons, I don’t want to reboot (some other computations).<br>
Is there any way to specify which GPU to use for the PyTorch?</p>
<p>Right now even <code>torch.cuda.is_available()</code> returns <code>False</code>, and any attempt to <code>x.to(device)</code> returns initialization errors.<br>
Restart of python kernel doesn’t help the situation.</p>

spanev: <p>What’s the output of:</p>
<pre><code class="lang-auto">dmesg | grep GPU
</code></pre>
<p>?</p>

Mart.Slaaf: <pre><code class="lang-auto">[   17.159322] [drm] [nvidia-drm] [GPU ID 0x00001800] Loading driver
[   17.160936] [drm] [nvidia-drm] [GPU ID 0x00003b00] Loading driver
[   17.162486] [drm] [nvidia-drm] [GPU ID 0x00008600] Loading driver
[   17.164018] [drm] [nvidia-drm] [GPU ID 0x0000af00] Loading driver
[838654.790542] NVRM: GPU at PCI:0000:86:00: GPU-811a52e6-d27e-3f2d-b972-503473cd15d9
[838654.790551] NVRM: GPU Board Serial Number:
[838654.790556] NVRM: Xid (PCI:0000:86:00): 79, GPU has fallen off the bus.
[838654.790631] NVRM: GPU at 00000000:86:00.0 has fallen off the bus.
[838654.790633] NVRM: GPU is on Board .
[838654.790657] NVRM: A GPU crash dump has been created. If possible, please run
</code></pre>

spanev: <p>Well I don’t think you can do something easily without rebooting for this time.<br>
But to avoid this problem in the future, you should set persistence mode:</p>
<pre><code class="lang-auto">/usr/bin/nvidia-smi -pm 1
</code></pre>
<p>Find complete info about it here <a href="https://www.cyberciti.biz/faq/debian-ubuntu-rhel-fedora-linux-nvidia-nvrm-gpu-fallen-off-bus/" rel="nofollow noopener">https://www.cyberciti.biz/faq/debian-ubuntu-rhel-fedora-linux-nvidia-nvrm-gpu-fallen-off-bus/</a></p>

Mart.Slaaf: <p>I have them in persistence mode already.<br>
Also, if you know the answer, what’s the underlying mechanism, why should it help?</p>

kl_divergence: <p>You can specify which GPU to use by specifing <code>CUDA_VISIBLE_DEVICES</code> environment variable. This will tell pytorch which GPU to use.</p>

Mart.Slaaf: <p>Many thanks!<br>
I’ve found this solution, and thought that it doesn’t work, but more accurate check shows, that it works, but with some additional porps.<br>
It doesn’t see the card, which is in order after the dropped one (3 is invisible, 2 is dropped, 0 and 1 are ok).</p>
<p>test_gpu.py:</p>
<pre><code class="lang-auto">import torch
print(torch.cuda.is_available())
print(torch.cuda.device_count())
print(torch.zeros(10, 10).to(torch.device('cuda:2')))
</code></pre>
<pre><code class="lang-auto">$&gt;export CUDA_VISIBLE_DEVICES=0,1,3
$&gt;python test_gpu.py
True
2
Traceback (most recent call last):
  File "test_gpu.py", line 4, in &lt;module&gt;
    print(torch.zeros(10, 10).to(torch.device('cuda:2')))
RuntimeError: CUDA error: invalid device ordinal
</code></pre>
<p>Do you have any ideas, how to fix it?</p>

kl_divergence: <p>This is happening because you’re not passing correct gpu id. If you’re not using docker, do <code>nvidia-smi</code> to see GPU ids and then specify which GPU you want. Like this<br>
<code>CUDA_VISIBLE_DEVICES=0 python3 main.py</code><br>
FYI, if you are specifying multiple gpu ids, you need to ensure that your model has multi gpu support with nn.dataparallel, if not consider using only one.</p>

Mart.Slaaf: <p>But I’m moving tensor to the specific device.<br>
Moreover, if I set the only one visible GPU, all is ok with 0 or 1, but doesn’t work for 3. (2 is the dropped one).</p>
<p>And yes, I’ve checked the device id’s, and moreover I’ve used code like this before GPU failed. So everything is ok with id, I think.</p>

IgorKasianenko: <p>Hi, I am facing same issue, did you find the root of the error?<br>
I have only 2nd GPU lost and I want to continue training.<br>
Yesterday I did reboot, but today the error is back, so any inputs about what was your cause would be great. Thanks!</p>
<p>Was it hardware problem like here?</p><aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/45891934/gpu-is-lost-during-execution-of-either-tensorflow-or-theano-code/54771507#54771507" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/7368263/mega" target="_blank" rel="noopener nofollow ugc">
    <img alt="Mega" src="https://www.gravatar.com/avatar/ae06d9f55c506d9be1c52ba378139be7?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/45891934/gpu-is-lost-during-execution-of-either-tensorflow-or-theano-code/54771507#54771507" target="_blank" rel="noopener nofollow ugc">GPU is lost during execution of either Tensorflow or Theano code</a>
</h4>

<div class="tags">
  <strong>gpu, nvidia, tensorflow-gpu, cudnn</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stackoverflow.com/users/7368263/mega" target="_blank" rel="noopener nofollow ugc">
    Mega
  </a>
  on <a href="https://stackoverflow.com/questions/45891934/gpu-is-lost-during-execution-of-either-tensorflow-or-theano-code/54771507#54771507" target="_blank" rel="noopener nofollow ugc">05:06PM - 19 Feb 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>