Models keep overfitting despite using regularization e.t.c

Fatih_Haslak: <p>I have tried data augmentation, regularization, penalty loss, normalization, dropout, learning rate schedulers, etc., but my models still tend to overfit. Sometimes I get good results in the very first epoch, but then the performance keeps dropping afterward. In longer trainings (e.g., 200 epochs), the best validation loss only appears in 2–3 epochs.</p>
<p>I encounter this problem not only with one specific setup but also across different datasets, different loss functions, and different model architectures. It feels like a persistent issue rather than a case-specific one.</p>
<p>Where might I be making a mistake?</p>

KFrank: <p>Hi Fatih!</p>
<aside class="quote no-group" data-username="Fatih_Haslak" data-post="1" data-topic="223329" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fatih_haslak/48/66598_2.png" class="avatar"> Fatih_Haslak:</div>
<blockquote>
<p>Where might I be making a mistake?</p>
</blockquote>
</aside>
<p>The most common cause of overfitting is training too long on not enough<br>
training data.  As you describe it, you see overfitting more or less right<br>
away which would suggest that you’re using only a small amount of training<br>
data.</p>
<p>Another possibility that could look like overfitting (that I wouldn’t actually<br>
call overfitting) would be if your training data and validation / test data<br>
differed significantly in character.  For example, if you trained a cat-dog<br>
classifier on images of cats and dogs, but then used images of catfish and<br>
dogfish for your validation data you would not expect to get good results<br>
on that validation data and you might take that as a symptom of overfitting.</p>
<p>The things you’ve tried – augmentation, dropout, and so on – can help<br>
reduce overfitting, but they won’t turn not enough data into enough data.</p>
<p>You make this sound like it is a systematic problem for you, so the first<br>
thing that comes to my mind would be that you are systematically not using<br>
enough training data.</p>
<p>For an example of training without overfitting, you could take a look at this<br>
pytorch tutorial:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://docs.pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial">
  <header class="source">

      <a href="https://docs.pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial" target="_blank" rel="noopener nofollow ugc">docs.pytorch.org</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://docs.pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial" target="_blank" rel="noopener nofollow ugc">NLP From Scratch: Classifying Names with a Character-Level RNN — PyTorch...</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>It comes with its own data set (so you will have enough data).  It randomly<br>
splits its data into a training set and a test set so that test data is<br>
of the same character as the training data and is, so to speak, statistically<br>
equivalent.</p>
<p>It might make sense to work through the tutorial and see if that gives you any<br>
clues about what you might be doing that is causing problems.</p>
<p>Good luck!</p>
<p>K, Frank</p>

Fatih_Haslak: <p>My model: Resnet18 with dropout,batchnorm1d<br>
Dataset: Train → 1.5 million<br>
Dataset: Test/Val → 300K</p>
<p>Train AUC Score - &gt; 0.99<br>
Test AUC Score → 0.85<br>
is that okay?</p>

KFrank: <p>Hi Fatih!</p>
<aside class="quote no-group" data-username="Fatih_Haslak" data-post="4" data-topic="223329" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fatih_haslak/48/66598_2.png" class="avatar"> Fatih_Haslak:</div>
<blockquote>
<p>Dataset: Train → 1.5 million<br>
Dataset: Test/Val → 300K</p>
<p>Train AUC Score - &gt; 0.99<br>
Test AUC Score → 0.85<br>
is that okay?</p>
</blockquote>
</aside>
<p>It certainly could be fine, but without knowing the details of your use case<br>
nor what your data looks like, it’s hard to say.</p>
<p>1.5 million training examples is a goodly number and for most “ordinary”<br>
problems should be enough to train your model reasonable well.</p>
<p>It’s quite reasonable to have a very good figure of merit (such as the area<br>
under the curve) for your training set while having a merely good figure of<br>
merit for your validation set.  Such a situation would not, in and of itself,<br>
indicate overfitting.</p>
<p>One expects a trained model to perform better – sometimes by a lot – on the<br>
training set than it does on the validation set.  Again, this is not what I would<br>
call overfitting.</p>
<p>I like to track both the training loss and some figure of merit relevant to my<br>
use case for both the training and validation sets.  Generally the loss and<br>
figure of merit will behave in similar ways – getting better at the same “rate”<br>
during training, slowing down or plateauing at the same time, and for the<br>
validation loss and figure of merit, turning around and getting worse at the<br>
same time when overfitting sets in.</p>
<p>Overfitting is when your training is causing your model, figuratively speaking,<br>
to start “memorizing” examples in your training set so that the weights in your<br>
model become less good at performing the general task  for which you are trying<br>
to train them.  When this happens you will see your training loss keep going<br>
down (as the “memorization” lets the model perform well on the training set)<br>
while the validation loss starts going up (as “memorizing” training-set examples<br>
doesn’t help your model perform well in the validation set which is a proxy for<br>
the real, more general task your model is trying to learn).</p>
<p>So when your <em>validation</em> loss (and / or figure of merit) starts getting worse,<br>
that’s a signal that overfitting has set in and you should stop training and<br>
keep your final model, or perhaps a bit better, the model that has the best<br>
validation loss (and / or figure of merit) as you were coming to the end of<br>
your training.</p>
<p>Best.</p>
<p>K. Frank</p>

Fatih_Haslak: <p>Hi Frank!</p>
<p>Is it normal for a model to learn everything in just 1–2 epochs, where the training loss keeps decreasing but the validation loss increases afterward? In my current dataset, I observed this behavior. When the learning rate is very low, the model continues to learn gradually over the epochs. However, with a learning rate like 1e-4, it seems to learn everything within just one epoch and then stops improving.</p>
<p>Thank you</p>

KFrank: <p>Hi Fatih!</p>
<aside class="quote no-group quote-modified" data-username="Fatih_Haslak" data-post="6" data-topic="223329" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fatih_haslak/48/66598_2.png" class="avatar"> Fatih_Haslak:</div>
<blockquote>
<p>Is it normal for a model to learn everything in just 1–2 epochs, where the training loss keeps decreasing but the validation loss increases afterward?<br>
…<br>
When the learning rate is very low, the model continues to learn gradually over the epochs.</p>
</blockquote>
</aside>
<p>When you say that the model is “learning,” it’s not clear to me whether you<br>
mean its performance is improving on the training set or on the validation set<br>
(or maybe some combination of the two).</p>
<p>Regardless, when your training set is <em>very large</em> it would become possible for<br>
your model to “learn (almost) everything” in one or two epochs.  Note, “epoch”<br>
generally means one pass through your training set, so one or two epochs could<br>
mean training on a very large number of training examples.</p>
<p>However, the way overfitting happens is that the training process sees the same<br>
training example many times so that it has the chance to (figuratively speaking)<br>
“memorize” it.  I therefore would not expect to see overfitting over just a few<br>
epochs.</p>
<p>I could see the improvement in both the training and validation loss slowing<br>
down and flattening out as the model learns more or less as much as it can<br>
about the problem it is actually trying to solve from the very large training set,<br>
but I wouldn’t expect the validation loss to start going up because the model has<br>
not yet started to learn misleading details of individual examples in the training<br>
set – degrading the performance on the validation set – because it’s only seen<br>
those individual examples once or twice.</p>
<p>The only way I could see overfitting occurring after a small number of training<br>
epochs would be if the “large” training set contained many examples that were<br>
near duplicates of one another (and were not also near duplicates of examples<br>
in the validation set) so that, in effect, the training set was not as large as it<br>
appeared and during training the model was seeing more or less the same example<br>
many times instead of just once or twice.</p>
<p>Best.</p>
<p>K. Frank</p>