Loss increasing during epoch

mohitsharma29: <p>So I am training a set of videos over two models. And I am logging my loss after each epoch. Both of the models are performing poorly wrt. loss. During my first epoch, the loss first dips and then increase and in the next epoch, it takes a step up. What can be the possible reasons? I am using Adam Optimizer and its multi-label classification.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/7/975cb2be214c3beef7370b3a56a5e20892a5e2d6.png" data-download-href="https://discuss.pytorch.org/uploads/default/975cb2be214c3beef7370b3a56a5e20892a5e2d6" title="Screenshot from 2020-04-11 03-09-35"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/975cb2be214c3beef7370b3a56a5e20892a5e2d6_2_690x394.png" alt="Screenshot from 2020-04-11 03-09-35" data-base62-sha1="lB0HAvdjvwAYF6BlU8zRsacpIkm" width="690" height="394" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/975cb2be214c3beef7370b3a56a5e20892a5e2d6_2_690x394.png, https://discuss.pytorch.org/uploads/default/original/3X/9/7/975cb2be214c3beef7370b3a56a5e20892a5e2d6.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/7/975cb2be214c3beef7370b3a56a5e20892a5e2d6.png 2x" data-dominant-color="FCFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-04-11 03-09-35</span><span class="informations">913×522 21.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/2/02a1417aba1af3b042fa9901b1f40aa8498d2930.png" data-download-href="https://discuss.pytorch.org/uploads/default/02a1417aba1af3b042fa9901b1f40aa8498d2930" title="Screenshot from 2020-04-11 03-10-00"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/02a1417aba1af3b042fa9901b1f40aa8498d2930_2_690x394.png" alt="Screenshot from 2020-04-11 03-10-00" data-base62-sha1="ngrwaNyGI2LCE68bh1AXZxvj6U" width="690" height="394" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/02a1417aba1af3b042fa9901b1f40aa8498d2930_2_690x394.png, https://discuss.pytorch.org/uploads/default/original/3X/0/2/02a1417aba1af3b042fa9901b1f40aa8498d2930.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/2/02a1417aba1af3b042fa9901b1f40aa8498d2930.png 2x" data-dominant-color="FCFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-04-11 03-10-00</span><span class="informations">905×518 20.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

jubick: <p>Adjusting learning rate and/or gradient clipping might help</p>

mohitsharma29: <p>Hey, thank you. I believe that using Adam should handle the adjusting learning rate part right?<br>
I will apply gradient clipping and see what happens.</p>

AladdinPerzon: <p>As <a class="mention" href="/u/jubick">@jubick</a> said the first thing that comes to mind is lowering the learning rate. What is your current learning rate?  Adam doesn’t adjust the learning rate, for that you would need to use a learning rate scheduler. Example of how to set the learning rate for Adam:</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
</code></pre>
<p>Share a code snippet and it’ll help debugging what could be wrong in your code</p>

mohitsharma29: <p>Okay so I left it at the default LR that Adam provides. I was earlier using Adadelta and switched to Adam because it adds an additional momemtum term and is more stable.<br>
This is the optimizer snippet. The whole point is to skip using a scheduler.<br>
optimizer = Adam(parameters, weight_decay=opt.weight_decay)</p>

AladdinPerzon: <p>What do you mean with the point is to skip using a scheduler?</p>
<p>I would since it’s diverging in your case still try and lower the learning rate below the default value and see if that changes anything.</p>
<p>Edit: if that doesn’t help then I think some code would be helpful to further debug</p>

b02202050: <p>Feel the same of <a href="https://discuss.pytorch.org/u/AladdinPerzon">AladdinPerzon</a> said.<br>
I would lower the lr to approx. 1e-5 to 1e-4 to see what happened.<br>
And another thing I will do is to disable the weight decay since it may cause the training harder.</p>

mohitsharma29: <p>So after suggestions, all of you, I am thinking about adding a scheduler with starting LR as 0.0001. However, I am skeptical about weight decay. My dataset is highly skewed, so i actually want my training to be difficult so that I dont fit quickly.</p>