Validation Loss decreasing slightly

Atila1: <p>Dear All,</p>
<p>I am training my network, I can figure out that I am not overfitting as the validation loss can still decrease as well as the training loss.</p>
<p>However, I see that after training for a while, the validation loss decreases slightly maybe every 11 -12 epochs, at some point I say that it will become steady, however, I can see that it still decreases but then it need long time ( 12 epoch) to decrease.</p>
<p>I wonder why it does not keep decreasing steadily as in the beginning of the epochs or even start to increase which means I started to overfit because then I can not decide on the number of epochs I should train for. Is that a problem with the optimizer learning rate, as I am not using a decayer for the learning rate and just keeping it fixed?</p>
<p>The plot of the validation loss looks like that<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/e/be23a1575a139fb789428ed3efcdad80f1a4d560.png" data-download-href="https://discuss.pytorch.org/uploads/default/be23a1575a139fb789428ed3efcdad80f1a4d560" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/be23a1575a139fb789428ed3efcdad80f1a4d560_2_510x500.png" alt="image" data-base62-sha1="r833wpWgD37pNMf7pVS4Fip1zSU" width="510" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/be23a1575a139fb789428ed3efcdad80f1a4d560_2_510x500.png, https://discuss.pytorch.org/uploads/default/original/3X/b/e/be23a1575a139fb789428ed3efcdad80f1a4d560.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/e/be23a1575a139fb789428ed3efcdad80f1a4d560.png 2x" data-dominant-color="FCFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">525×514 23.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

googlebot: <p>this just reflects that gradients and optimizer steps from training become smaller. and the “noise” is obviously from suboptimal stochastic steps in parameter space w.r.t. validation set.</p>
<p>optimizer tuning would maybe achieve faster <em>training</em> loss convergence, and would affect a validation loss graph indirectly. OTOH, big steps may lead to solutions that generalize worse.</p>

Atila1: <p><a class="mention" href="/u/googlebot">@googlebot</a>  Thanks a lot for your response.</p>
<p>But, sorry I have not got it clearly. So, do you recommend decreasing the learning rate after a few epochs or increase it?</p>

googlebot: <p>LR increase by epoch is usually only done as “warmup” for optimizers that collect gradient stats, like adam.  And I’m not sure if LR decrease is warranted with that graph. So, in this specific case, <em>overall</em> LR increase can maybe reach same training loss in less epochs, but adding a LR decay scheduler may be advisable (depends on optimizer too).</p>
<p>It is hard to say how to schedule LR optimally - some people even do hyperoptimization runs to find that.</p>
<p>But my point was, your validation graph is fine if the slowdown just reflects slowdown on a training graph. As for reasoning about overfit, you kinda have to “smooth” plot points one way or the other, as the volatility is inherent.</p>