LogBackward returned nan values in its 0th output

Ahmed_Abdelaziz: <p>Hi,</p>
<p>While training my model I got NaNs as the result of loss function after many iterations. I used <code>anaomly_detection</code> to see what is causing that issue and I got that error.</p>
<pre><code class="lang-auto">Function 'LogBackward' returned nan values in its 0th output
</code></pre>
<p>This is the only log funtion I used in my forward path</p>
<pre><code class="lang-auto"> return o.log()
</code></pre>
<p>And that is a sample of the values it receives</p>
<pre><code class="lang-auto">         [0.0004, 0.0006, 0.0010,  ..., 0.0143, 0.0143, 0.0143],
         ...,
         [0.0007, 0.0005, 0.0007,  ..., 0.0146, 0.0146, 0.0146],
         [0.0007, 0.0005, 0.0007,  ..., 0.0147, 0.0147, 0.0147],
         [0.0007, 0.0005, 0.0007,  ..., 0.0147, 0.0147, 0.0147]],
</code></pre>
<p>I cannot catch any zeros or negative numbers passed to it. Can you maybe tell me if anyline I should use to catch the number that is causing that issue or anyways to recover from this?</p>

albanD: <p>Hi,</p>
<p>Maybe some values in there are small enough that they produce <code>-inf</code>?<br>
If you add a <code>o = o.clamp(min=1e-4)</code>, does it prevent issues?</p>

Ahmed_Abdelaziz: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="92820">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>min=1e</p>
</blockquote>
</aside>
<p>I got same error</p>
<pre><code class="lang-auto">    o = o.clamp(min=1e-4)
    return o.log(),planlogits
</code></pre>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "./train.py", line 351, in &lt;module&gt;
    main(args)
  File "./train.py", line 137, in main
    train_joint
  File "./train.py", line 221, in train
    gr_loss.backward()
  File "~/anaconda3/envs/torch-sci/lib/python3.6/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "~/anaconda3/envs/torch-sci/lib/python3.6/site-packages/torch/autograd/__init__.py", line 100, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Function 'LogBackward' returned nan values in its 0th output.
</code></pre>

albanD: <p>And don’t you have a warning showing a second stack trace just above by any chance? That should point to the faulty log function.</p>

Ahmed_Abdelaziz: <p>I see that the warning is pointing to the exact function I pointed</p>
<pre><code class="lang-auto">Warning: Error detected in LogBackward. Traceback of forward call that caused the error:
  File "train.py", line 339, in &lt;module&gt;
    main(args)
  File "train.py", line 133, in main
    train_joint
  File "train.py", line 202, in train
    p, planlogits = graph_model(batch)
  File "~/anaconda3/envs/torch-sci/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "~/torch-sci/GraphWriter/models/newmodel.py", line 167, in forward
    return o.log(),planlogits
</code></pre>
<p>However, I tried to remove the faulty log function but I got this instead. Do you think it’s a cascaded error coming somehow from a different module?</p>
<pre><code class="lang-auto">RuntimeError: Function 'MulBackward0' returned nan values in its 0th output.
</code></pre>
<p>As a note, I printed the min o value before clamping and I see it even received the NaN value even before passing to the log function.</p>

albanD: <p>Thanks for the trace. Just wanted to confirm that we’re not looking at the wrong place in the code!</p>
<p>Ho so the forward value is Nan already?</p>

Ahmed_Abdelaziz: <p>Thanks for your quick response. Yes And I don’t know how this is possible. I assumed that anomaly detection find the first occurrence of NaN and reports it, doesn’t it?</p>
<p>Any better way to find the first nan in network? Here is the warning from the second error</p>
<pre><code class="lang-auto">Warning: Error detected in MulBackward0. Traceback of forward call that caused the error:
  File "train.py", line 339, in &lt;module&gt;
    main(args)
  File "train.py", line 133, in main
    train_joint
  File "train.py", line 202, in train
    p, planlogits = graph_model(batch)
  File "~/anaconda3/envs/torch-sci/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "~/torch-sci/GraphWriter/models/newmodel.py", line 158, in forward
    z = (1-s)*z
</code></pre>
<p>I can keep tracking it back that way, It’s kinda time consuming because I have to run my model for couple of iterations until I get the next error</p>

albanD: <aside class="quote no-group" data-username="Ahmed_Abdelaziz" data-post="7" data-topic="92820">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/958977/48.png" class="avatar"> Ahmed_Abdelaziz:</div>
<blockquote>
<p>I assumed that anomaly detection find the first occurrence of NaN and reports it</p>
</blockquote>
</aside>
<p>It does, but it can <em>only</em> look at the backward pass. So if nans happen in the forward, they won’t be reported.<br>
You will need to add prints directly in your code to check where the nan appears.<br>
In particular, you can check <code>your_tensor.isnan().any()</code> ? To check that in the forward.</p>

Ahmed_Abdelaziz: <p>I will try to debug my forward path then, thanks for pointing this out!</p>

Ahmed_Abdelaziz: <p>Hi <a class="mention" href="/u/alband">@albanD</a>, I figured the nan source in the forward pass, It’s a masked softmax that uses <code>-inf</code> to mask the <code>False</code> values, but I guess I have many -infs that’s why it can return nan.</p>
<pre><code class="lang-auto">    unnorm.masked_fill_(emask,-float('inf'))
    attn = F.softmax(unnorm,dim=2)
    out = torch.bmm(attn,emb)
</code></pre>
<p>I tried the below line as alternative, but the values that should be masked have values near the others that shouldn’t. Any ideas here?</p>
<pre><code class="lang-auto">    unnorm.masked_fill_(emask,-1 * 1e-15)

</code></pre>

albanD: <p><code>1e-15</code> is practically the same as <code>inf</code> for float32.<br>
You can try <code>1e-6</code> to avoid nan (this is the best precision you can have for a float).</p>

ali_safaya: <p>Hi,</p>
<p>I have a similar problem, and since I am too late, I am only writing this for future reference. The proper value is not <code>-1e-15</code> or <code>-1e-6</code>, I think you meant to use <code>-1e6</code> since this is approximately <code>-Inf</code>.<br>
but <code>-1e-15</code> is approximately zero.</p>

Hajar_Merbouh: <p>Hi <a class="mention" href="/u/ahmed_abdelaziz">@Ahmed_Abdelaziz</a> , I am having the same issue while using a masked softmax that uses -inf to mask the false values. Did you find a solution to the given error ?</p>