A computer with 4 GPUs has different error

username: yelanyelan
cooked: <p>My computer has 4 GPUs, they are all NVIDIA GeForce 2080TI, my environment is:<br>
NVIDIA-SMI 470.42.01    Driver Version: 470.42.01    CUDA Version: 11.4.20210623  cuDNN  Version: 8.2.4.15-1+cuda11.4   pytorch Version: 1.7.0<br>
The same code runs on the 1st and 3rd GPU and everything works fine with pytorch, but there are different errors on GPU 0 and 2.</p>
<p>When using <code>CUDA_VISIBLE_DEVICES=0</code>：I usually connect the screen to GPU 0 for display, and when it is running, I always closed the visual interface. It reported when running:</p>
<blockquote>
<p>/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [43,0,0], thread: [32,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [43,0,0], thread: [33,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [43,0,0], thread: [34,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [43,0,0], thread: [35,0,0] Assertion <code>idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"</code> failed.<br>
…<br>
Traceback (most recent call last):<br>
…<br>
File “/home/Project/test.py”, line 47, in get_rate_eachc<br>
_, inverse, counts = torch.unique(CH_area[i], return_inverse=True, return_counts=True)<br>
File “/home/wenqu/workspace/pytorch/lib/python3.6/site-packages/torch/_jit_internal.py”, line 265, in fn<br>
return if_true(*args, **kwargs)<br>
File “/home/wenqu/workspace/pytorch/lib/python3.6/site-packages/torch/_jit_internal.py”, line 265, in fn<br>
return if_true(*args, **kwargs)<br>
File “/home/wenqu/workspace/pytorch/lib/python3.6/site-packages/torch/functional.py”, line 682, in _unique_impl<br>
return_counts=return_counts,<br>
RuntimeError: transform: failed to synchronize: cudaErrorAssert: device-side assert triggered</p>
</blockquote>
<p>When using <code>CUDA_VISIBLE_DEVICES=0</code>：</p>
<blockquote>
<p>Traceback (most recent call last):<br>
…<br>
File “/home/workspace/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home/Project/test2.py”, line 44, in forward<br>
theta = self.tanh(self.conv0(x))<br>
File “/home/workspace/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home/workspace/pytorch/lib/python3.6/site-packages/torch/nn/modules/conv.py”, line 423, in forward<br>
return self._conv_forward(input, self.weight)<br>
File “/home/workspace/pytorch/lib/python3.6/site-packages/torch/nn/modules/conv.py”, line 420, in _conv_forward<br>
self.padding, self.dilation, self.groups)<br>
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR<br>
You can try to repro this exception using the following code snippet. If that doesn’t trigger the error, please include your original repro script when reporting this issue.</p>
<p>import torch<br>
torch.backends.cuda.matmul.allow_tf32 = True<br>
torch.backends.cudnn.benchmark = True<br>
torch.backends.cudnn.deterministic = False<br>
torch.backends.cudnn.allow_tf32 = True<br>
data = torch.randn([4, 3, 224, 224], dtype=torch.float, device=‘cuda’, requires_grad=True)<br>
net = torch.nn.Conv2d(3, 128, kernel_size=[3, 3], padding=[1, 1], stride=[1, 1], dilation=[1, 1], groups=1)<br>
net = net.cuda().float()<br>
out = net(data)<br>
out.backward(torch.randn_like(out))<br>
torch.cuda.synchronize()</p>
<p>ConvolutionParams<br>
data_type = CUDNN_DATA_FLOAT<br>
padding = [1, 1, 0]<br>
stride = [1, 1, 0]<br>
dilation = [1, 1, 0]<br>
groups = 1<br>
deterministic = false<br>
allow_tf32 = true<br>
input: TensorDescriptor 0x499d450<br>
type = CUDNN_DATA_FLOAT<br>
nbDims = 4<br>
dimA = 4, 3, 224, 224,<br>
strideA = 150528, 50176, 224, 1,<br>
output: TensorDescriptor 0x77c4b480<br>
type = CUDNN_DATA_FLOAT<br>
nbDims = 4<br>
dimA = 4, 128, 224, 224,<br>
strideA = 6422528, 50176, 224, 1,<br>
weight: FilterDescriptor 0x7c448fe0<br>
type = CUDNN_DATA_FLOAT<br>
tensor_format = CUDNN_TENSOR_NCHW<br>
nbDims = 4<br>
dimA = 128, 3, 3, 3,<br>
Pointer addresses:<br>
input: 0x7fc1ba000000<br>
output: 0x7fc1b2000000<br>
weight: 0x7fc2211b9000</p>
</blockquote>
<p>When using <code>CUDA_VISIBLE_DEVICES=0,1,2</code>： Reported the same error as GPU 0.<br>
These errors do not occur on 1,3 GPU. So I cant determine where the problem is, whether my code is wrong or the GPUs have some problems. Does anyone have the same problem？<br>
Thx u for any help.</p>

username: ptrblck
cooked: <p>I’m not sure which error is raised on which device, as you’ve mentioned device 0 in both error messages.<br>
Based on the first error, your are running into an indexing issue. Rerun the code with <code>CUDA_LAUNCH_BLOCKING=1 python script.pt args</code> and check which operation fails. Once you’ve isolated it, check the indices and make sure they are valid.</p>
<p>After this is fixed, rerun the code on the other devices and check, if the cuDNN error is still raised.<br>
<code>CUDNN_STATUS_INTERNAL_ERROR</code> is also raised, if cuDNN is running into a sticky error caused by a previously failed operation. Since you were seeing the invalid indexing assert, cuDNN might just reraise it.</p>

username: yelanyelan
cooked: <p>Anyway, thank you very much for your answer. Now my situation is a bit complicated.<br>
On device 0,first, I was trying to fix the indexing issue with <code>CUDA_LAUNCH_BLOCKING=1</code>. I found it might come from torch.gather, so I tried to cancel the use of it. It worked and device 0 can training with ‘CUDA_LAUNCH_BLOCKING=1’. But after training for a while, device 0 disappeared from nvidia-smi. I used <code>sudo nvidia-persistenced --persistence-mode</code> to fix this issue.</p>
<p>By the way, I have a custom loss, maybe its gradient will be relatively large，we temporarily call it 'A loss‘’. Continue training, I found that device 0 has the following situations：</p>
<ol>
<li>It can training, and anything a simple network: for example, a network with only one convolutional layer and using mseloss. (Whether using MSE loss or Aloss, whether using <code>CUDA_LAUNCH_BLOCKING=1</code> or not)</li>
<li>When using a Complex network(with resnet34 backbone and jumper connection, decoding network):</li>
</ol>
<ul>
<li>It will report <code> cuDNN error: CUDNN_STATUS_INTERNAL_ERROR</code> when running without using <code>CUDA_LAUNCH_BLOCKING=1</code>.</li>
<li>When running with <code>CUDA_LAUNCH_BLOCKING=1</code>, it can be trained not to ask any questions. But its loss is easy to become NAN when using Aloss.</li>
</ul>
<p>I make a few guesses：</p>
<ol>
<li>On device 0, it seems to be more sensitive to large values than others. So it is more likely to cause the value to overflow and cause loss to become NAN.</li>
<li>This problem may not come from pytorch but the GPU itself. Because my code does not have any problems mentioned in the above device 0 on device 1, 3.</li>
<li>But I cannot understand <code>CUDA_LAUNCH_BLOCKING=1</code> cannot solve the problem of <code>cuDNN error: CUDNN_STATUS_INTERNAL_ERROR</code></li>
</ol>
<p>As for device 2, when I try to use A to solve it, it will arise <code>Unable to find a valid cuDNN algorithm to run convolution</code>. I carefully referred to the questions you answered:<br>
<a href="https://discuss.pytorch.org/t/unable-to-find-a-valid-cudnn-algorithm-to-run-convolution/78724/2">Unable to find a valid cuDNN algorithm to run convolution</a><br>
And it doesnt worked.</p>
<p>In the end, I found that the above two devices 0 and 2 that have problems are both when trying to train:<br>
Their Fan and Pwr Usage will become ERR.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/1/912ab6562b76c0cd4f0a421016e39bf3ad6f7d2a.jpeg" alt="QQ截图20211105153021" data-base62-sha1="kIcK78RXGVEUqIfiIAFYNRRKm9A" width="473" height="245"></p>
<p>I am very grateful to your help. If you have any relevant information please help me.</p>

username: ptrblck
cooked: <p><code>CUDA_LAUNCH_BLOCKING=1</code> will synchronize each call and thus slow down your code.<br>
Based on your description and the output of <code>nvidia-smi</code> your devices might indeed be overheating.<br>
Could you open the case and put a large fan in front of the machine while it’s running?<br>
If that’s possible, check <code>nvidia-smi -l</code> and observe if the temperature would still be rising to critical levels.</p>

username: yelanyelan
cooked: <p>I think you are right, so I just went to the computer room to disassemble my computer. I tried to clean and reapply the silicone grease for my two problematic devices.<br>
Then I found that after removing the two GPUs that were out of order, the two GPUs that I used to work normally also had problems.</p>
<ul>
<li>Device 0（Before it was device 1）raise <code>RuntimeError: CUDA error: unspecified launch failure</code></li>
<li>Device 1（Before it was device 3）raise <code>cuDNN error: CUDNN_STATUS_INTERNAL_ERROR</code></li>
</ul>
<p>They seem to inherit the original bug.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cdf9b17c630e7ee6412648f634705d55cdd87a0a.png" data-download-href="https://discuss.pytorch.org/uploads/default/cdf9b17c630e7ee6412648f634705d55cdd87a0a" title="QQ图片20211105172016"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cdf9b17c630e7ee6412648f634705d55cdd87a0a.png" alt="QQ图片20211105172016" data-base62-sha1="to8Qvp15tG0gUU1ZJCCpasg7iTE" width="517" height="209" data-dominant-color="110F0F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">QQ图片20211105172016</span><span class="informations">714×289 13.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This device should be the original device 3. I make sure that I just started the computer, and their temperature is not high. I am a little confused now.</p>

username: ptrblck
cooked: <p>OK, that’s indeed weird and the “unspecified lunch failure” often points to an issue in the lower stack.<br>
I assume all devices are 2080Tis? Could you update PyTorch to the latest version and check, if it would still be failing? I guess you’ve built PyTorch from source?<br>
Would it also be possible to post a minimal code snippet to reproduce the issue (the gather operation might be sufficient)? If so, I would try to reproduce your setup and see if I could also hit the issue.</p>

username: yelanyelan
cooked: <p>All devices are 2080ti.<br>
I would try to update or built PyTorch. If nothing works, I should try to post my error code.</p>

username: ptrblck
cooked: <p>Thanks! Also, could you check <code>dmesg</code> and search for <code>Xid</code>s?<br>
<code>dmesg -T | grep -i xid</code> would show if errors are detected. If so, could you post the errors here, too?</p>