Gradient value clipping not working

username: shashank_gupta1
cooked: <p>Hello everyone! I am trying to train a simple 2 layer MLP on tabular input for a reinforcement learning task with ReLU activations in between and no activation in the end (network predicts log scores). Because of ReLU activations, the training becomes unstable quickly and loss shoots up very high during training. I have added gradient clipping to stabilize the training, but it’s not working always. I have added the clipping as follows:</p>
<blockquote>
<p><code>optimizer.zero_grad()</code><br>
<code> obj.backward()</code><br>
<code> torch.nn.utils.clip_grad_value_(doc_scorer.parameters(), 0.9)</code><br>
` optimizer.step()</p>
</blockquote>
<p>However, the gradient values are not getting clipped, as can be seen from the wandb log attached<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/0/50de5e761389e07cfa79c173069b935a5eb5eb03.png" data-download-href="https://discuss.pytorch.org/uploads/default/50de5e761389e07cfa79c173069b935a5eb5eb03" title="Screenshot 2023-06-13 at 09.35.28"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50de5e761389e07cfa79c173069b935a5eb5eb03_2_690x479.png" alt="Screenshot 2023-06-13 at 09.35.28" data-base62-sha1="bxoCv7dOiaG4a0hNfi3O78abpvl" width="690" height="479" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50de5e761389e07cfa79c173069b935a5eb5eb03_2_690x479.png, https://discuss.pytorch.org/uploads/default/original/3X/5/0/50de5e761389e07cfa79c173069b935a5eb5eb03.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/0/50de5e761389e07cfa79c173069b935a5eb5eb03.png 2x" data-dominant-color="F9FBFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-06-13 at 09.35.28</span><span class="informations">838×582 15 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/0/40d0a76f6ac5b5df8f1ff4e1da3624edbf0b0162.png" data-download-href="https://discuss.pytorch.org/uploads/default/40d0a76f6ac5b5df8f1ff4e1da3624edbf0b0162" title="Screenshot 2023-06-13 at 09.35.53"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/40d0a76f6ac5b5df8f1ff4e1da3624edbf0b0162_2_690x473.png" alt="Screenshot 2023-06-13 at 09.35.53" data-base62-sha1="9fnB29SBh6U1fPYPWftPZQLHyFk" width="690" height="473" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/40d0a76f6ac5b5df8f1ff4e1da3624edbf0b0162_2_690x473.png, https://discuss.pytorch.org/uploads/default/original/3X/4/0/40d0a76f6ac5b5df8f1ff4e1da3624edbf0b0162.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/0/40d0a76f6ac5b5df8f1ff4e1da3624edbf0b0162.png 2x" data-dominant-color="F9FAFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-06-13 at 09.35.53</span><span class="informations">878×602 15.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have read that wandb logs gradient before applying the clipping operations, but if gradient clipping were to function properly, the parameters of the network should not increase in magnitude over time<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/6/7612430b0c9ee466b6f57de894ef8b096394f177.png" data-download-href="https://discuss.pytorch.org/uploads/default/7612430b0c9ee466b6f57de894ef8b096394f177" title="Screenshot 2023-06-13 at 09.37.35"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/6/7612430b0c9ee466b6f57de894ef8b096394f177.png" alt="Screenshot 2023-06-13 at 09.37.35" data-base62-sha1="gQvsYMKo1VsHGXMAcW7UjGrF5iv" width="690" height="474" data-dominant-color="F9FBFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-06-13 at 09.37.35</span><span class="informations">860×592 22.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>.</p>
<p>Can anyone help me with any suggestions or pointers to solve this problem? Thanks.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="shashank_gupta1" data-post="1" data-topic="181988">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shashank_gupta1/48/19577_2.png" class="avatar"> shashank_gupta1:</div>
<blockquote>
<p>I have read that wandb logs gradient before applying the clipping operations, but if gradient clipping were to function properly, the parameters of the network should not increase in magnitude over time</p>
</blockquote>
</aside>
<p>Could you explain why this should be the case?<br>
Also, did you check the actual gradient values after applying the clipping operation?</p>

username: shashank_gupta1
cooked: <p>Thank you for your reply. I checked the gradient values from the gradient tensor, and it is clipping. I was suspicious because the parameter values increased over iterations. I realized it happened because all gradient values were clipped to max-min and changing the parameter over time.</p>