Gradient clipping is not working properly

MrPositron: <p>Hello!</p>
<p>I am using gradient clipping during training as follows:</p>
<pre><code class="lang-auto">optimizer.zero_grad()
loss = criterion(output, target)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm = 1)
optimizer.step()
</code></pre>
<p>However, as you can see from the wandb plots<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/1/816f544013dfa96b8c296a6ad09abba6f76153d2.png" data-download-href="https://discuss.pytorch.org/uploads/default/816f544013dfa96b8c296a6ad09abba6f76153d2" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/816f544013dfa96b8c296a6ad09abba6f76153d2_2_690x297.png" alt="image" data-base62-sha1="it278N7JQpxYXWrvUwGaRDujsfo" width="690" height="297" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/816f544013dfa96b8c296a6ad09abba6f76153d2_2_690x297.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/816f544013dfa96b8c296a6ad09abba6f76153d2_2_1035x445.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/816f544013dfa96b8c296a6ad09abba6f76153d2_2_1380x594.png 2x" data-dominant-color="F7FAFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1484×639 13.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Gradients explode, ranging from -3e5 to 3e5.</p>
<p>This plot shows the disribution of weights across each mini-batch. Thus, it is not an accumulated sum of all gradients.</p>
<p>I don’t know what is happening <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Could you add a check before and after the clipping is applied, iterate all parameters, and print their max. abs. value to isolate if the clipping is indeed not working?</p>

MrPositron: <p>Thanks! Okay, I will try that</p>

MrPositron: <p>Gradients before norm clipping:<br>
<code>[4.451817512512207, 2.2666594982147217, 12.370549201965332, 2.5617222785949707, 3.411081314086914, 32.17192840576172, 2.2899510860443115, 48.48966979980469, 2.751540184020996]</code><br>
And here after it:<br>
<code>[4.451817512512207, 2.2666594982147217, 12.370549201965332, 2.5617222785949707, 3.411081314086914, 32.17192840576172, 2.2899510860443115, 48.48966979980469, 2.751540184020996]</code></p>
<p>The way I compute gradients is as following:</p>
<pre><code class="lang-auto">def get_gradients(model):
	grads = []
	for params in model.parameters():
		grads.append(torch.max(torch.abs(params)).item())
	return grads
</code></pre>
<p>Here is how I print them:</p>
<pre><code class="lang-auto">grads = get_gradients(model)
print(grads)
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm = 1)
grads = get_gradients(model)
print(grads)
</code></pre>

ptrblck: <p>Thanks for the update.<br>
Sorry, for not being clear in my previous post, but could you print the <code>params.grad</code> attributes?<br>
The parameters themselves won’t be changed, but their gradients should. Also, the <code>norm</code> before and after would be interesting to see:</p>
<pre><code class="lang-python">print(torch.norm(torch.cat([p.grad.view(-1) for p in model.parameters()])))
</code></pre>

MrPositron: <p>Oh. I made a mistake. Sorry <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>You were perfectly clear. I wanted to print gradients, i.e.  <code>grads.append(torch.max(torch.abs(params.grad)).item())</code>. But somwehow I forgot it. Such a silly mistake.</p>
<p>I will fix the mistake, and will also chechk the norm. Thanks for you reply! I will reply asap.</p>

MrPositron: <p>I checked gradients, and everythin is fine. I am sorry for taking your time. I think that W&amp;B just logs the gradients when they are not yet clipped.</p>