Does my loss curve show the model is overfitting?

Cyber_punk: <p>The following is my loss curve. It includes the training batch loss and validation loss.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/3/23eeb25292868baa1cb826bf6e746e17f1f07e26.png" data-download-href="https://discuss.pytorch.org/uploads/default/23eeb25292868baa1cb826bf6e746e17f1f07e26" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/23eeb25292868baa1cb826bf6e746e17f1f07e26_2_690x426.png" alt="image" data-base62-sha1="57S7dSstDrE1kyC4OaoYIaeyupg" width="690" height="426" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/23eeb25292868baa1cb826bf6e746e17f1f07e26_2_690x426.png, https://discuss.pytorch.org/uploads/default/original/3X/2/3/23eeb25292868baa1cb826bf6e746e17f1f07e26.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/3/23eeb25292868baa1cb826bf6e746e17f1f07e26.png 2x" data-dominant-color="FCFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1002×619 29.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I also checked the F1 Score and AUROC on the validation data. Both of them seems performing well:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4dfcb531353de10704a521c5403823765e409414.png" data-download-href="https://discuss.pytorch.org/uploads/default/4dfcb531353de10704a521c5403823765e409414" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4dfcb531353de10704a521c5403823765e409414_2_690x404.png" alt="image" data-base62-sha1="b7UbJ5Uw14YG6pvLGozYAcdtSV6" width="690" height="404" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4dfcb531353de10704a521c5403823765e409414_2_690x404.png, https://discuss.pytorch.org/uploads/default/original/3X/4/d/4dfcb531353de10704a521c5403823765e409414.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/d/4dfcb531353de10704a521c5403823765e409414.png 2x" data-dominant-color="FCFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">953×558 29.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>However I’m confused about two things:</p>
<p>(1) Both training loss and validation loss are decreasing, <strong>but validation loss is higher than training loss consistently, while validation loss never increases</strong>. It seems I can continue the training even at the end (100k step). <strong>Is this still considerred overfitting?</strong> If early stopping is implemented, stopping will not be triggered yet since ther validation loss is still decreasing.</p>
<p>(2) <strong>The training loss drops sharply at the beginning of each epoch</strong>. What is the reason behind this? Do I need to worry about it? I have shuffled the training data in DataLoader. In the model, the last layer is an vector inner product layer. When replacing this layer with a block of 3 linear layers with leakyReLU activation functions, and keeping everything else the same, the sharp drop in training loss disappears. In this way, I think the sharp drop in training loss is caused by the vector inner product layer. But why?</p>

vdw: <p>Overfitting is generally indicated by the training loss decreasing further, but with the validation loss starting to increase at some point (e.g., after some epochs). So no, I wouldn’t call this overfitting.</p>
<p>Why would a sharp drop in the training loss a thing of concern. If you have a large training dataset with a “simple” distribution – silly example: 1 Million images with 50% red circles and %50 blue squares – the the model will have learned everything there is too learn after 1 epoch, I’m sure :).</p>

dhruvbird: <aside class="quote no-group" data-username="Cyber_punk" data-post="1" data-topic="184039">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/258eb7/48.png" class="avatar"> Cyber_punk:</div>
<blockquote>
<p><strong>Is this still considerred overfitting?</strong></p>
</blockquote>
</aside>
<p>I would personally not consider this to be overfitting, but it is somewhat concerning that your train and val accuracy is so far apart. This could happen due to a variety of non-exhaustive reasons. A common one is probably some sort of bias when you chose your train and val split and you have some different distribution of data in both.</p>
<p>I’m not too worried about the train and val loss - these can be apart since you’re computing the train loss on a mini-batch with gradient updates whereas val loss is computed w/o gradient updates. I would run through the training set and val sets once w/o backward (with the model in eval mode) and check if the losses line up closer than what you’re seeing here.</p>

KFrank: <p>Hi Cyber!</p>
<aside class="quote no-group" data-username="Cyber_punk" data-post="1" data-topic="184039" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/258eb7/48.png" class="avatar"> Cyber_punk:</div>
<blockquote>
<p>(1) Both training loss and validation loss are decreasing, <strong>but validation loss is higher than training loss consistently, while validation loss never increases</strong>.</p>
</blockquote>
</aside>
<p>This is not overfitting (by my definition) and the general structure of<br>
your results does not surprise me.</p>
<p>Your validation loss continues to decrease (albeit slowly) and your<br>
validation performance metrics continue to increase.  So (as long as<br>
your are happy investing in further training time) keep training to get<br>
a better model.</p>
<p>The gap between your training loss and validation loss is quite typical<br>
of realistic models.</p>
<p>Here’s, crudely, what’s going on:</p>
<p>Your training set encodes two kinds of information: information about<br>
the real problem you’re trying to solve (say cats vs. dogs), e.g., these<br>
cats have intelligent expressions on their faces and these dogs are<br>
scratching fleas; and “random” information specific to your training set,<br>
e.g.,  this cat picture has a chessboard in the background and this dog<br>
picture has a cluster of dark pixels in the upper right corner.</p>
<p>When you train on your training set, your model learns whatever it<br>
can to get the right answer:  both “looks smart,” cat, “fleas,” dog; and<br>
“chessboard,” cat, “dark pixels,” dog help your model make good<br>
predictions on your training set.  But only the first two “features” help<br>
your model make good predictions on your validation set.  Since your<br>
model makes use of both kinds of information with the training set,<br>
it does a better job – lower loss – on the training set than on the<br>
validation where is can only make use of the first kind of information.</p>
<p>How does overfitting happen?  It’s when the model trains its parameters<br>
to focus on “chessboard” vs. “pixels” <em>at the expense of</em> “smart” vs. “fleas.”<br>
It gets better and better and better at the “random,” training-set-specific<br>
information – so the training loss goes down, but it makes increasingly<br>
little use of the “real” information, so the validation loss goes up.  The<br>
more you train, the <em>worse</em> your model becomes (expect for the artificial<br>
case of the training set).</p>
<p>So: validation loss keeps going down – keep training (if you care); validation<br>
loss starts going up (systematically – not just little fluctuations) – stop<br>
training because you’re making your model worse.</p>
<aside class="quote no-group">
<blockquote>
<p>(2) <strong>The training loss drops sharply at the beginning of each epoch</strong>. What is the reason behind this?</p>
</blockquote>
</aside>
<p>I don’t know – but it might be interesting and fun to try to figure it out.</p>
<aside class="quote no-group">
<blockquote>
<p>Do I need to worry about it?</p>
</blockquote>
</aside>
<p>The short answer is no – your validation metrics look fine, so your model<br>
is working.</p>
<p>But something is going on.  It might be innocuous, but it could be a real<br>
issue that is perhaps making your training less efficient.  If it were me, I<br>
would try to track down the cause of the training-loss drops, but I wouldn’t<br>
knock myself about it.</p>
<aside class="quote no-group">
<blockquote>
<p>I think the sharp drop in training loss is caused by the vector inner product layer. But why?</p>
</blockquote>
</aside>
<p>I don’t know how an inner-product layer works, so no idea why.  However,<br>
given that you can use your validation set to see how well your model is<br>
actually doing, I would use whatever version of your model ends up<br>
giving you better performance on your validation set after training.  If the<br>
training has these unexpected drops in it, so be it (but I’d still spend at<br>
least a little time trying to track down the cause).</p>
<p>Best.</p>
<p>K. Frank</p>