Gradient Clipping is not work as expected

XezXey: <p>Hi all, I have an exploding gradient problem when train the minibatch for 150-200 epochs with batch size = 256 and there’s about 30-60 minibatch (This depends on my specific config). But I have an exploding gradient issues even if I add the code below.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4664d03c7c5580f59a79a3ae1e6e895801b8523.png" data-download-href="https://discuss.pytorch.org/uploads/default/a4664d03c7c5580f59a79a3ae1e6e895801b8523" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4664d03c7c5580f59a79a3ae1e6e895801b8523.png" alt="image" data-base62-sha1="nsltZiVcxdqN1ZZjuOcU1PIGPoT" width="690" height="72" data-dominant-color="F4F0E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">842×88 13.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As you can see this below images, notice that in step about 40k there’s the swing of gradients between ± 20k, 40k and 60k respectively. I don’t know why this happens because i use the clip_grad_value_ above. Also Using the learning rate decay from 0.01 to about 0.008 at step 40k.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c061caa859f88a0977250e2a1f898294c94dc65.png" data-download-href="https://discuss.pytorch.org/uploads/default/0c061caa859f88a0977250e2a1f898294c94dc65" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c061caa859f88a0977250e2a1f898294c94dc65_2_690x147.png" alt="image" data-base62-sha1="1ImP7moslFreIlMXXnb0otThEXP" width="690" height="147" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c061caa859f88a0977250e2a1f898294c94dc65_2_690x147.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c061caa859f88a0977250e2a1f898294c94dc65_2_1035x220.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c061caa859f88a0977250e2a1f898294c94dc65_2_1380x294.png 2x" data-dominant-color="F8FAFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1763×378 26.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Or do I need to update the weight parameters by myself  something like this<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/e/7e19619de8406b6628615091e834cbdeef763535.png" alt="image" data-base62-sha1="hZwxvF1DvrqFq8W6cd3adL8ek61" width="542" height="82"><br>
But i think optimizer.step() should do the job and the clip_grad_value_ is an inplace operation so i don’t need to take the return value from function. Please correct if i did anything wrong. Thank you very much</p>
<p>Best regards,<br>
Mint</p>

TinfoilHat0: <p>Hi it seems correct to me however, perhaps your norm bound is too high ? What happens when you lower it from 100 to 10 or so ?</p>

XezXey: <p>Hi, I’ve decrease it to ±1 and it’s still have these problem. But there’s 2 different kind of computing the bound by norm or by a value. Maybe using value is a cause of the problem. I’m not sure about this</p>

XezXey: <p>any solution? I still got this problem<br>
Thank you</p>

TinfoilHat0: <p>It seems correct to me. Can you, for example, try printing the L2 norm of gradients over iterations to see if the code is really working as expected?</p>