RuntimeError: Function 'TBackward' returned nan values in its 0th output

Lei_Yang: <h2>RuntimeError</h2>
<p>Sorry to bore, but may I know some hints to solve this problem?</p>
<ol>
<li>
<p>I use the mnist example provided by pytorch by running <strong>main.py</strong> with <strong>autograd anomaly detection</strong> set to <strong>True</strong>, and get this error.</p>
</li>
<li>
<p>I actually observed this error in another codes of mine, and thought it was triggered by taking sqrt of a very small small number. I tested my codes (and this example) with cpu and on other machine and they run just fine.</p>
</li>
<li>
<p>I think I might have something wrong with my cuda installation but I cannot find a solution. So, I am here to look for any hints that might be helpful. Do I need to install cuda and cudnn that matches the version of cudatoolkit shipped with pytorch conda install?</p>
</li>
</ol>
<h3>Pytorch version</h3>
<p>pytorch                   1.5.0           py3.8_cuda10.1.243_cudnn7.6.3_0    pytorch</p>
<h3>here are the error report</h3>
<p>Warning: Error detected in TBackward. Traceback of forward call that caused the error:<br>
File “main.py”, line 139, in <br>
main()<br>
File “main.py”, line 130, in main<br>
train(args, model, device, train_loader, optimizer, epoch)<br>
File “main.py”, line 42, in train<br>
output = model(data)<br>
File “/home/yanglei/anaconda3/envs/pt150cu101/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “main.py”, line 29, in forward<br>
x = self.fc1(x)<br>
File “/home/yanglei/anaconda3/envs/pt150cu101/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/yanglei/anaconda3/envs/pt150cu101/lib/python3.8/site-packages/torch/nn/modules/linear.py”, line 87, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
File “/home/yanglei/anaconda3/envs/pt150cu101/lib/python3.8/site-packages/torch/nn/functional.py”, line 1610, in linear<br>
ret = torch.addmm(bias, input, weight.t())<br>
(print_stack at /opt/conda/conda-bld/pytorch_1587428094786/work/torch/csrc/autograd/python_anomaly_mode.cpp:60)<br>
Traceback (most recent call last):<br>
File “main.py”, line 139, in <br>
main()<br>
File “main.py”, line 130, in main<br>
train(args, model, device, train_loader, optimizer, epoch)<br>
File “main.py”, line 44, in train<br>
loss.backward()<br>
File “/home/yanglei/anaconda3/envs/pt150cu101/lib/python3.8/site-packages/torch/tensor.py”, line 198, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/home/yanglei/anaconda3/envs/pt150cu101/lib/python3.8/site-packages/torch/autograd/<strong>init</strong>.py”, line 98, in backward<br>
Variable._execution_engine.run_backward(<br>
RuntimeError: Function ‘TBackward’ returned nan values in its 0th output.</p>

ptrblck: <aside class="quote no-group" data-username="Lei_Yang" data-post="1" data-topic="106211">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lei_yang/48/32358_2.png" class="avatar"> Lei_Yang:</div>
<blockquote>
<p>Do I need to install cuda and cudnn that matches the version of cudatoolkit shipped with pytorch conda install?</p>
</blockquote>
</aside>
<p>You don’t need a local CUDA installation, if you don’t plan to build PyTorch from source or any custom CUDA extension (or of course any other CUDA application).<br>
That being said, could you update to the latest stable release and check, if you are still facing this issue?</p>

Lei_Yang: <p>Hi ptrblck,</p>
<p>Thanks for your reply. After testing multiple settings of PyTorch and Cuda from conda install and docker (using the anibali/pytorch image) as well, I finally changed a GPU card and the errors disappeared. I mounted the GPU card to another machine, on which everything runs fine, and reproduced the error. So, I think it is the GPU card’s problem, but not due to PyTorch or other packages or contaminated environments.</p>
<p>BTW, I was suggested using this gpu-burn tool (<a href="https://github.com/wilicc/gpu-burn" rel="noopener nofollow ugc">https://github.com/wilicc/gpu-burn</a>) to test if the GPU card is problematic. I used it and it works well. So, I just put it here for those who encounter similar errors.</p>

Lei_Yang: <p>I just test it again on two pytorch settings (one being the latest)</p>
<ol>
<li>pytorch 1.7.1 py38_cuda11.0.221_cudnn8.0.5_0</li>
<li>pytorch 1.7.0 py38_cuda11.0.221_cudnn8.0.3_0<br>
and they also encounter the same error.</li>
</ol>
<p>One situation is a bit strange to me.<br>
When I set False to the anomaly detection mode, I am able to run with the main.py script in the mnist folder from pytorch example git repo.<br>
However, in the same setting, error occurs if I run the main.py script in the word_language_model folder.</p>
<p>The error is as follow (main.py refers to the one under word_language_model)</p>
<p>Traceback (most recent call last):<br>
File “main.py”, line 217, in <br>
train()<br>
File “main.py”, line 181, in train<br>
torch.nn.utils.clip_grad_norm_(model.parameters(), args.clip)<br>
File “/home/yangle/anaconda3/envs/pytorch_latest/lib/python3.8/site-packages/torch/nn/utils/clip_grad.py”, line 38, in clip_grad_norm_<br>
if clip_coef &lt; 1:<br>
RuntimeError: CUDA error: an illegal memory access was encountered</p>

ptrblck: <p>Are you seeing the illegal memory access using the “bad” GPU or another one?<br>
Which GPU are you using at the moment? I assume you haven’t changed anything in the tutorial and are just running the script as it is?</p>

Lei_Yang: <p>All errors occur ONLY on the “bad” GPU. Above I just want to clarify that even with the latest stable releases of pytorch, errors still occur on the “bad” GPU.</p>
<ol>
<li>
<p><strong>I just ran the scripts as it is.</strong> In this situation, even with the “bad” GPU, the main.py in pytorch/mnist runs well; no error occurs. But the main.py in pytorch/word_language_model (using the LSTM model) will have the error “RuntimeError: CUDA error: an illegal memory access was encountered”.</p>
</li>
<li>
<p>In the second situation where I made a minimal change to the scripts for the testing purpose: I added the line “torch.autograd.set_detect_anomaly(True):” prior to the iteration for loop. Then, in BOTH scripts of mnist and word_language_model, appears the error " RuntimeError: Function ‘TBackward’ returned nan values in its 0th output" (not necessarily the TBackward, but could be other functions).</p>
</li>
</ol>

ptrblck: <p>If these issues are specific to a particular GPU, I would assume it might be a hardware defect.<br>
Are you able to reproduce the illegal memory access in every run on this device and are other GPUs working fine?</p>

Lei_Yang: <p>Regarding this, I did two experiments.</p>
<ol>
<li>
<p>On my own desktop, using the “bad” GPU and another GPU.<br>
On the “bad” GPU, the two example scripts report errors (when autograd anomaly detection is activated). On the other GPU, no error is reported.</p>
</li>
<li>
<p>On another desktop, the same errors can be reproduced with the “bad” GPU. Using another GPU on this another desktop will not report errors.</p>
</li>
</ol>
<p>The “bad” GPU is RTX2080Ti, while the other GPU is 1080Ti (because no available 2080Ti at hand). I don’t think it’s PyTorch’s problem; it shall be a hardware issue.</p>
<p>I don’t know if this is a too-much request for PyTorch: Is it possible to have a small testing code in PyTorch (or there has been one but I am just not aware) to check if a GPU is faulty? In this way, it might be helpful to identify if this is a hardware or software problem. Anyway, it might be too much for PyTorch, then just forget it.</p>

ptrblck: <p>I don’t think it’s a good approach to add hardware checks into a framework, as these kind of issues are so variable that the majority of issues might point to the hardware even though we might be using a faulty kernel. I’m a bit afraid we could claim a hardware issue too early and ship bad code.</p>
<p>There is one thing you could check using the bad GPU:<br>
could you rerun the code, let it error out, and check <code>dmesg</code> for any <code>xid</code> entries and report the error code here, please?</p>

Lei_Yang: <p>Hi Ptrblck. Sorry for my late reply. Unfortunately, I have returned the ‘bad’ GPU card to the vendor for exchanging a new one, so cannot do what you’ve suggested.</p>

ptrblck: <p>I hope the new device is working properly! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>