RuntimeError: Function 'MulBackward0' returned nan values in its 0th output

yumu: <p>im working on an SNN and my loss.backwards gives me this weird error<br>
im using an adam optimiser a relu activation function and nll_loss with autograd</p>
<p>any idea of what could cause this ?</p>

albanD: <p>Hi,</p>
<p>Did you run with anomaly detection?<br>
If so, the only reason I can see for multiplication to return nan is because the forward gave you nan as well.</p>

yumu: <p>yes i did run with anomaly detection also I tried to check for nans in forward with a xi[xi != xi] = 0 but doing this gives me this error instead<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [100, 20]] is at version 1202; expected version 1200 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>

albanD: <p>doing <code>xi[xi != xi] = 0</code> will zero-out the nans after the op. But that does not change the fact that you have a nan in the op.<br>
The anomaly mode should point you to the forward function that is causing the nan. I guess one of the input is either nan or infinite? You want to prevent such input (not output) to be nan or inf.</p>

yumu: <p>i also did data[data != data] = 0 at the very beggining to prevent this nans in the database and the forward function only takes from that database</p>
<h2>this is the full tracker btw</h2>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in <br>
1 spiking_model = SpikingNet(device, n_time_steps=1200, begin_eval=0)<br>
----&gt; 2 train_many_epochs(spiking_model)</p>
<p> in train_many_epochs(model)<br>
27         epoch = i<br>
28         optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.5)<br>
—&gt; 29         train(model, device, train_set_loader, optimizer, epoch, logging_interval=1)<br>
30         test(model, device, test_set_loader)<br>
31</p>
<p> in train(model, device, train_set_loader, optimizer, epoch, logging_interval)<br>
10         target = target.long()<br>
11         loss = F.nll_loss(output, target)<br>
—&gt; 12         loss.backward()<br>
13<br>
14         torch.nn.utils.clip_grad_norm_(model.parameters(),0.5)</p>
<p>~.conda\envs\py36\lib\site-packages\torch\tensor.py in backward(self, gradient, retain_graph, create_graph)<br>
193                 products. Defaults to <code>False</code>.<br>
194         “”"<br>
–&gt; 195         torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
196<br>
197     def register_hook(self, hook):</p>
<p>~.conda\envs\py36\lib\site-packages\torch\autograd_<em>init</em>_.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)<br>
97     Variable._execution_engine.run_backward(<br>
98         tensors, grad_tensors, retain_graph, create_graph,<br>
—&gt; 99         allow_unreachable=True)  # allow_unreachable flag<br>
100<br>
101</p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [100, 20]] is at version 1202; expected version 1200 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
<p>i don’t know if the anomaly mode gives you anything else i didn’t find anything</p>

albanD: <p>This error is because you modify some things inplace that should not be modifie inplace. This has nothing to do with the anomaly mode.<br>
You can fix this by changing <code>data[data != data] = 0</code> to <code>data = data.clone(); data[data != data] = 0</code> so that you don’t modify the original data inplace.</p>
<p>The anomaly mode tells you about the nan. If you remove this and you have the nan error again, you should have an additional stack trace that tells you about the forward function (make sure to enable the anomaly mode before the you run the forward).</p>

yumu: <p>i did what you just said and i have this</p>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in <br>
1 spiking_model = SpikingNet(device, n_time_steps=1200, begin_eval=0)<br>
----&gt; 2 train_many_epochs(spiking_model)</p>
<p> in train_many_epochs(model)<br>
27         epoch = i<br>
28         optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.5)<br>
—&gt; 29         train(model, device, train_set_loader, optimizer, epoch, logging_interval=1)<br>
30         test(model, device, test_set_loader)<br>
31</p>
<p> in train(model, device, train_set_loader, optimizer, epoch, logging_interval)<br>
10         target = target.long()<br>
11         loss = F.nll_loss(output, target)<br>
—&gt; 12         loss.backward()<br>
13<br>
14         torch.nn.utils.clip_grad_norm_(model.parameters(),0.5)</p>
<p>~.conda\envs\py36\lib\site-packages\torch\tensor.py in backward(self, gradient, retain_graph, create_graph)<br>
193                 products. Defaults to <code>False</code>.<br>
194         “”"<br>
–&gt; 195         torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
196<br>
197     def register_hook(self, hook):</p>
<p>~.conda\envs\py36\lib\site-packages\torch\autograd_<em>init</em>_.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)<br>
97     Variable._execution_engine.run_backward(<br>
98         tensors, grad_tensors, retain_graph, create_graph,<br>
—&gt; 99         allow_unreachable=True)  # allow_unreachable flag<br>
100<br>
101</p>
<p>RuntimeError: Function ‘MulBackward0’ returned nan values in its 0th output.</p>
<p>still nothing leading to the forward function<br>
i have also checked the output of the forward function and couldn’t find nans (i might have missed it tho)</p>

albanD: <p><code>0 * inf</code> will also give you nan. So you might want to look for them.</p>
<p>Do you silence warnings in your code? The other stack trace should be returned as a python warning.</p>

yumu: <p>i don’t silence warning but im using jupyter and i don’t know how this IDE handles warnings as i don’t see any other stack traces<br>
edit: ive tried on spyder and same thing no warnings</p>

silvester: <p>Sorry for reviving this post.<br>
When I get this error the program is crashed. Is there a way to set it up in python such that when it come across this issue, it just simply <code>continue</code> the <code>for</code> and try the other parameters for training.</p>

albanD: <p>Hi,</p>
<p>This is just a regular python error. So you can catch it and silence it like any other python errors.</p>

tianq01: <p><a class="mention" href="/u/alband">@albanD</a> thanks for your tips. I got MulBackward0 nan error in anomaly detection mode.<br>
the forward function that is causing the nan is below loss:</p>
<blockquote>
<p><span class="mention">@weighted_loss</span><br>
def smooth_l1_loss(pred, target, beta=1.0):<br>
assert beta &gt; 0<br>
assert pred.size() == target.size() and target.numel() &gt; 0<br>
diff = torch.abs(pred - target)<br>
loss = torch.where(diff &lt; beta, 0.5 * diff * diff / beta, &lt;–this line<br>
diff - 0.5 * beta)<br>
return loss</p>
</blockquote>
<p>so the <strong>diff</strong> tensor is nan here?<br>
is it the <strong>first place</strong> that nan occurs?</p>

albanD: <p>This is the first place in the backward pass!<br>
So you might want to check if there were nan in the forward. Otherwise, the most likely reason could be that <code>beta</code> is very close to <code>0</code>?</p>

tianq01: <p>thanks for the quick response <a class="mention" href="/u/alband">@albanD</a></p>
<p>some background:<br>
1)we are using pytorch based mmdetection framework, faster-rcnn with FPN and res50 backbone.<br>
2)the problem is when training with many more epochs, nan may occur. we are sure the dataset is fine, and there is no nan issue using tensorflow based counterpart. It is not very easy to repo, although we alreay set determinism config. (faster rcnn training examples are still randomly selected in each iteration).</p>
<p>we print the key tensors in each iteraction, at some point, the value became bigger and bigger and finally nan:(sorry, only 1 file is allowed to upload)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae97161f07de6fe1f7803eb76dbef6f4cf49cc43.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ae97161f07de6fe1f7803eb76dbef6f4cf49cc43" title="4"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/ae97161f07de6fe1f7803eb76dbef6f4cf49cc43_2_690x465.jpeg" alt="4" data-base62-sha1="oUuMqam6gSZBMBPZujhQxprlPOP" width="690" height="465" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/ae97161f07de6fe1f7803eb76dbef6f4cf49cc43_2_690x465.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae97161f07de6fe1f7803eb76dbef6f4cf49cc43.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae97161f07de6fe1f7803eb76dbef6f4cf49cc43.jpeg 2x" data-dominant-color="141313"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">4</span><span class="informations">785×530 139 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>the tensors in the red box are the res50 output feature maps.  however at this point detect_anomaly did not report any error, and the forward pass is continuing…and the exception is not triggered until backward phase.</p>
<p>my questions:<br>
1)why forward computation can continue when nan already occurs?<br>
2)any suggestion to debug this issue?<br>
thanks.</p>

tianq01: <p>the full stacktrace under detect_anomaly mode(not the totally same with my original post):<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/2/628c156be398365a8fe1400c53f06e84c9904557.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/628c156be398365a8fe1400c53f06e84c9904557" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/2/628c156be398365a8fe1400c53f06e84c9904557_2_690x304.jpeg" alt="1" data-base62-sha1="e3MV2tuHFheo3ciuem18kYWJPwj" width="690" height="304" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/2/628c156be398365a8fe1400c53f06e84c9904557_2_690x304.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/2/628c156be398365a8fe1400c53f06e84c9904557_2_1035x456.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/2/628c156be398365a8fe1400c53f06e84c9904557.jpeg 2x" data-dominant-color="0D0D0D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">1229×542 156 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

tianq01: <p>the value became bigger and bigger:<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3c4415bdb24c856c8ce58ddfb646dfd29b9079bb.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3c4415bdb24c856c8ce58ddfb646dfd29b9079bb" title="3"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3c4415bdb24c856c8ce58ddfb646dfd29b9079bb_2_690x433.jpeg" alt="3" data-base62-sha1="8B8wcZef0O8izzo44632xh7PWbN" width="690" height="433" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3c4415bdb24c856c8ce58ddfb646dfd29b9079bb_2_690x433.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/3/c/3c4415bdb24c856c8ce58ddfb646dfd29b9079bb.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/c/3c4415bdb24c856c8ce58ddfb646dfd29b9079bb.jpeg 2x" data-dominant-color="121312"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">3</span><span class="informations">847×532 148 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

albanD: <p>Hi,</p>
<p>If the values become bigger and bigger, this is the issue. The fact that it ends up so big that you get nan is not the root problem.<br>
You just want to make sure your network stops diverging.<br>
Do you properly zero out the gradients between iterations? Do you properly set the learning rate and other optimizer settings?</p>

tianq01: <p>hi.<br>
yes. the zero out is done in mmcv package after_train_iter call:<br>
see <a href="https://github.com/open-mmlab/mmcv/blob/master/mmcv/runner/hooks/optimizer.py" rel="nofollow noopener">https://github.com/open-mmlab/mmcv/blob/master/mmcv/runner/hooks/optimizer.py</a></p>
<p>learning rate and optimizer setting are also ok. As mentioned, the training with the same config using tensorflow implementation does not NaN.</p>
<p>ps we tried clip gradients. not working.<br>
anyway the problem is not so easy to repo, also the behavior is not totally the same for each repro. e.g. in another repo, the value does become bigger and bigger, but not so big – compared to  1.8035e+25 in above screenshot – then nan occurs.</p>
<p>any suggestion?<br>
print tensor gradients in backward phase using hook? what to do next if we find gradients anomalous somewhere…I suppose faster-rcnn_fpn implementation should be already stable in mmdetection…</p>

albanD: <aside class="quote no-group" data-username="tianq01" data-post="18" data-topic="70346">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/dc4da7/48.png" class="avatar"> tianq01:</div>
<blockquote>
<p>learning rate and optimizer setting are also ok. As mentioned, the training with the same config using tensorflow implementation does not NaN.</p>
</blockquote>
</aside>
<p>You should check the optimizers doc in details as there might be differences! Momentum in particular IIRC</p>

tianq01: <p>Could you point me the doc link?<br>
besides debugging the nan directly, we also tried multiple comparative experiments. it proves changing interpolation method in FPN upsample from nearest neighbor to bilinear, which is the method used in TF implementation,  solve the nan issue… <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>