Training loss decrease firstly but increase later

JethroJC: <p>I trained LSTM-MDN model using Adam, the training loss decreased firstly, but after serveral hundreds epoch, it increased and higher than Initial value. Then I retrained from the point where the loss was lowest and reduced learning rate 10 times(from 1e-3 to 1e-4). The training loss alse decreased firstly and increased later.I initially thought there was some bugs in the code, but I didn’t find any bug. Then I replaced adam with SGD(momentum=0),  training loss didn’t increased, but it converged to a relatively large value which was higher than the loss from adam, so I thougtht there was something wrong in adam.<br>
I never found the reason, I hope someone can help me find the reason.Thansk!<br>
loss (adam)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/3e6ca9d9380a2fe90fd35e4794c9c6d98bf2a862.png" data-download-href="https://discuss.pytorch.org/uploads/default/3e6ca9d9380a2fe90fd35e4794c9c6d98bf2a862" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3e6ca9d9380a2fe90fd35e4794c9c6d98bf2a862_2_690x289.png" alt="image" data-base62-sha1="8UepDNK6R1KDkR9T51as9JvjwZA" width="690" height="289" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3e6ca9d9380a2fe90fd35e4794c9c6d98bf2a862_2_690x289.png, https://discuss.pytorch.org/uploads/default/optimized/2X/3/3e6ca9d9380a2fe90fd35e4794c9c6d98bf2a862_2_1035x433.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/3/3e6ca9d9380a2fe90fd35e4794c9c6d98bf2a862_2_1380x578.png 2x" data-dominant-color="FDFDFE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2610×1096 106 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

JethroJC: <p>loss function<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/ded1e9bc2245a6c5b82f9d80a40a8ab6f0430753.png" data-download-href="https://discuss.pytorch.org/uploads/default/ded1e9bc2245a6c5b82f9d80a40a8ab6f0430753" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/d/ded1e9bc2245a6c5b82f9d80a40a8ab6f0430753_2_690x204.png" alt="image" data-base62-sha1="vN9Jzrdy9c65BUs47cV5Y5odpJh" width="690" height="204" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/d/ded1e9bc2245a6c5b82f9d80a40a8ab6f0430753_2_690x204.png, https://discuss.pytorch.org/uploads/default/original/2X/d/ded1e9bc2245a6c5b82f9d80a40a8ab6f0430753.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/d/ded1e9bc2245a6c5b82f9d80a40a8ab6f0430753.png 2x" data-dominant-color="E8E9F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1020×302 38.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Do you zero out the gradients after each optimizer step?<br>
I’ve seen similar behavior when the gradients were accidentally accumulated.</p>

JethroJC: <aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="24893">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>zero out</p>
</blockquote>
</aside>
<p>I really appreciate you replying to me.Here is my code.Is there a wrong order between computing the loss and calling zero_grad? Thank you very much~~<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/373a23c0b6138be9991fa4e89602509d2ba531d2.png" data-download-href="https://discuss.pytorch.org/uploads/default/373a23c0b6138be9991fa4e89602509d2ba531d2" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/373a23c0b6138be9991fa4e89602509d2ba531d2_2_690x176.png" alt="image" data-base62-sha1="7SyP7sgbCc7yeyBi5li4h1aygEO" width="690" height="176" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/373a23c0b6138be9991fa4e89602509d2ba531d2_2_690x176.png, https://discuss.pytorch.org/uploads/default/optimized/2X/3/373a23c0b6138be9991fa4e89602509d2ba531d2_2_1035x264.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/3/373a23c0b6138be9991fa4e89602509d2ba531d2_2_1380x352.png 2x" data-dominant-color="2F302A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1750×448 32.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

JethroJC: <p>When I replaced adam with SGD(momentum=0), training loss didn’t increase, but it converged to a relatively large value which was higher than the loss from adam.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/4935d2251ceeef078f7f30d5d9d8332846ead193.png" data-download-href="https://discuss.pytorch.org/uploads/default/4935d2251ceeef078f7f30d5d9d8332846ead193" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4935d2251ceeef078f7f30d5d9d8332846ead193_2_690x273.png" alt="image" data-base62-sha1="arEaigT0E2o0ACV2NwwENGDIZ6r" width="690" height="273" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4935d2251ceeef078f7f30d5d9d8332846ead193_2_690x273.png, https://discuss.pytorch.org/uploads/default/optimized/2X/4/4935d2251ceeef078f7f30d5d9d8332846ead193_2_1035x409.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/4/4935d2251ceeef078f7f30d5d9d8332846ead193_2_1380x546.png 2x" data-dominant-color="FDFEFE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2384×946 77 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The order of your calls looks alright.<br>
I’m unfortunately not really familiar with your use case and the loss function you are using.<br>
Also, what is <code>ensure_shared_grad</code> doing? Is it just copying the current gradients to another model?</p>

JethroJC: <p>Yes, it is. Thanks for your replying.I’m still looking for the reason.Maybe there is something wrong with adam?</p>

sh0416: <p>I also encounter this issue.<br>
Is there any suggestion when we use adam optimizer?</p>

Arthur_Garon: <p>I had a similar issue. The problem was that I was training with mixed precision bf16, which caused numerical instability with the Adam optimizer.<br>
I switched to fp32 for backward, and everything worked well.</p>