Clip_grad_norm_() returns nan

Maks_Botlhale: <p>Hi everyone</p>
<p>I’m training a model using torch and the clip_grad_norm_ function is returning a tensor with nan:<br>
tensor(nan, device=‘cuda:0’)<br>
Is there any specific reason why this would happen? Thanks for the help.</p>

albanD: <p>Hi,</p>
<p>This will might happen if the norm of your Tensors is 0? Or if it has a single element?</p>

Maks_Botlhale: <p>Please excuse my late response. So the the tensor more than one element but I did notice that the elements in the tensor are very close to zero. Could this also cause the norm to be nan? And how would I get around this? Thanks</p>

BramVanroy: <p><a class="mention" href="/u/alband">@albanD</a> can correct me if I’m wrong but <code>clip_grad_norm_</code> is an in-place operation and doesn’t return anything (None) which might be implicitly cast to <code>nan</code>. So use it like this (and do not assign it to anything):</p>
<pre><code class="lang-python">clip_grad_norm_(model.parameters(), 1.0)
</code></pre>

albanD: <p>I’m not sure, from the <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.clip_grad_norm_.html?highlight=clip_grad_norm#torch.nn.utils.clip_grad_norm_">doc</a> it does modify the weights inplace but also returns the total norm.</p>
<p><a class="mention" href="/u/maks_botlhale">@Maks_Botlhale</a>  Which norm are you using? This is most likely due to the content of your weights yes <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

Maks_Botlhale: <p>Hi</p>
<p>I’m using norm_type=2. Yes, the clip_grad_norm_(model.parameters(), 1.0) function does return the total_norm and it’s this total norm that’s nan.</p>

albanD: <p>Is any element in any parameter nan (or inf) by any chance? You can use <code>p.isinf().any()</code> to check.</p>

Maks_Botlhale: <p>I just checked for that, none of the elements in parameters are infinite. See the screenshot below. I tried decreasing the learning rate and that didn’t help; some people suggested changing the dropout rate, that also didn’t help. I also noticed that the validation loss is also nan. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/5/957775e57cd749d41f31bb4099ad79c0d2409e41.png" data-download-href="https://discuss.pytorch.org/uploads/default/957775e57cd749d41f31bb4099ad79c0d2409e41" title="check_inf"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/5/957775e57cd749d41f31bb4099ad79c0d2409e41.png" alt="check_inf" data-base62-sha1="lkf5l02961MDsggg68ll9ZHmAiR" width="690" height="111" data-dominant-color="363636"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">check_inf</span><span class="informations">1608×259 29.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

albanD: <p>This is surprising…<br>
The clip_grad_norm_ function is pretty simple and is there: <a href="https://github.com/pytorch/pytorch/blob/1c6ace87d127f45502e491b6a15886ab66975a92/torch/nn/utils/clip_grad.py#L25-L41">https://github.com/pytorch/pytorch/blob/1c6ace87d127f45502e491b6a15886ab66975a92/torch/nn/utils/clip_grad.py#L25-L41</a><br>
Can you try to copy paste that in your code and check it gives nan as well? Then you can add some prints there to see when the nan appears <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Maks_Botlhale: <p>I copied and pasted that as suggested, and I am still getting nan values when it’s calculating the total norm. Line 36 of the code I copied calculates the total norm as:<br>
<code>total_norm = torch.norm(torch.stack([torch.norm(p.grad.detach(), norm_type).to(device) for p in parameters]), norm_type)</code></p>
<p>I did the <code>p.grad.detach()</code> function on a separate line and I noticed that’s where the nan values start popping up.</p>

albanD: <p>Ho right (sorry I missed that…). It computes the grad norm, not the Tensors norm!<br>
You need to check if the gradients of the parameters contain nans: <code>p.grad.isinf().any()</code></p>

Maks_Botlhale: <p>Yes, that function also returns False. See the screenshot below.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d275a10fed0299460bbeb639dd3ae9d4193e8310.png" data-download-href="https://discuss.pytorch.org/uploads/default/d275a10fed0299460bbeb639dd3ae9d4193e8310" title="check_grads"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d275a10fed0299460bbeb639dd3ae9d4193e8310.png" alt="check_grads" data-base62-sha1="u1OhRoye7WZQppowjzXMVU2eW9a" width="690" height="111" data-dominant-color="363636"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">check_grads</span><span class="informations">1606×259 29.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

albanD: <p>Well if you said in your comment above that <code>p.grad.detach()</code> has nan, then the grad must have nans already.</p>

Yikai_Kang: <p>You can try this quite simple example, maybe you can find a solution:</p>
<pre><code class="lang-auto">import torch
x = torch.tensor([1., 2.])
x.grad = torch.tensor([0.4, float("inf")])
torch.nn.utils.clip_grad_norm_(x, 5)
print(x.grad)
</code></pre>

External_happy: <p>Any solutions to this problem? I have nan values after a few iterations.</p>