Overfitting model

username: Schroter
cooked: <p>Hi All,</p>
<p>I have a SWIN transformer model that is overfitting the data (Increasing training accuracy and stable  validation accuracy) despite using some measures to mitigate it. Such as <code>weight decay</code>, <code>batch-norm</code> and <code>dropout</code>. Each class has around 2500 images. Dataset has 8 classes. Batch size is 32.</p>
<p>Model details:</p>
<pre><code class="lang-auto">model = models.swin_v2_b(weights='IMAGENET1K_V1')

num_classes = len(image_datasets['train'].classes)

#model.head = nn.Linear(model.head.in_features, num_classes)


num_classes = model.head.in_features
model.head = nn.Sequential(
    nn.Linear(num_classes, 512),  # Add a new fully connected layer
    nn.ReLU(),
    nn.BatchNorm1d(512),  # Batch normalization
    nn.Dropout(0.5),       # Dropout with a 50% probability
    nn.Linear(512, 8)       # Output layer
)
</code></pre>
<p>Would anyone be able to help me in this regards please.<br>
Thanks &amp; Best Regards<br>
Michael</p>

username: KFrank
cooked: <p>Hi Michael!</p>
<aside class="quote no-group" data-username="Schroter" data-post="1" data-topic="190937" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/schroter/48/57964_2.png" class="avatar"> Schroter:</div>
<blockquote>
<p>model that is overfitting the data (Increasing training accuracy and stable  validation accuracy)</p>
</blockquote>
</aside>
<p>I would not necessarily say that your model has started to overfit yet.  To me,<br>
overfitting has set in when your validation performance metrics start getting<br>
<em>worse,</em> even as your training performance metrics continue to improve.</p>
<p>Consider training significantly longer to check whether your validation<br>
performance metrics might just have hit a temporary plateau and might<br>
start improving again as you continue to train.</p>
<p>I would also recommend looking at your validation loss (and training loss)<br>
in addition to your validation accuracy.  You would like to see their behavior<br>
confirm one another.  For example, it might be the case that your validation<br>
loss is still going down (perhaps slowly), even as your validation accuracy<br>
is going sideways (is “stable”).  That would be a sign that you are not yet<br>
overfitting.</p>
<p>Also, you haven’t given us any sense of how long you have trained for.</p>
<aside class="quote no-group">
<blockquote>
<p>despite using some measures to mitigate it. Such as <code>weight decay</code>, <code>batch-norm</code> and <code>dropout</code>.</p>
</blockquote>
</aside>
<p>Consider also using data augmentation to help delay overfitting.</p>
<aside class="quote no-group">
<blockquote>
<p>Each class has around 2500 images. Dataset has 8 classes. Batch size is 32.</p>
</blockquote>
</aside>
<p>How many samples are in your training and validation datasets?</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">model = models.swin_v2_b(weights='IMAGENET1K_V1')

num_classes = len(image_datasets['train'].classes)

num_classes = model.head.in_features
model.head = nn.Sequential(
    nn.Linear(num_classes, 512),  # Add a new fully connected layer
...
</code></pre>
</blockquote>
</aside>
<p>You haven’t said whether you are only fine-tuning the new <code>model.head</code> or<br>
whether you are also continuing to train the pretrained weights.</p>
<p>You can also sometimes reduce overfitting by reducing the “capacity” of your<br>
model (basically the number of parameters), generally by making the model<br>
“shallower” or “narrower.” Applying this just to <code>model.head</code>, you might consider<br>
narrowing <code>model.head</code> by reducing the number of its “hidden neurons” to <code>128</code><br>
or <code>64</code> (from the current value of <code>512</code>).</p>
<p>Best.</p>
<p>K. Frank</p>

username: Schroter
cooked: <p>Hi,<br>
Thanks for the reply. The validation set has around 250 images per class. What I am trying to do currently, is to use it as a fixed feature extractor.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="190937">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>You haven’t said whether you are only fine-tuning the new <code>model.head</code> or<br>
whether you are also continuing to train the pretrained weights.</p>
</blockquote>
</aside>
<p>Is it possible to do both at the same time?</p>
<p>I will change the hidden neurons and check.</p>
<p>Please find the attached graph of training and testing (should be validation set could not change the labels) losses and accuracy. For a better understanding of the issue.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8b078f4fb8ad239804dda94626698baa2067fab.png" data-download-href="https://discuss.pytorch.org/uploads/default/a8b078f4fb8ad239804dda94626698baa2067fab" title="image_2023-11-01_010956409"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8b078f4fb8ad239804dda94626698baa2067fab_2_690x354.png" alt="image_2023-11-01_010956409" data-base62-sha1="o4iiWIrWtCP3nyyDGPhIUz8csPx" width="690" height="354" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8b078f4fb8ad239804dda94626698baa2067fab_2_690x354.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8b078f4fb8ad239804dda94626698baa2067fab_2_1035x531.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8b078f4fb8ad239804dda94626698baa2067fab_2_1380x708.png 2x" data-dominant-color="FCFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image_2023-11-01_010956409</span><span class="informations">1438×738 74.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks &amp; Best Regards<br>
Michael</p>

username: KFrank
cooked: <p>Hi Michael!</p>
<aside class="quote no-group" data-username="Schroter" data-post="3" data-topic="190937" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/schroter/48/57964_2.png" class="avatar"> Schroter:</div>
<blockquote>
<p>Please find the attached graph of training and testing</p>
</blockquote>
</aside>
<p>At first glance, it does look like your model is overfitting – the training loss<br>
is going down while the validation loss is going up, and so on.</p>
<p>However, on closer inspection, the graphs look fishy to me.</p>
<p>You add a new, randomly-initialized head to your model.  Therefore, before<br>
any training or fine-tuning, your model should be making random guesses,<br>
<em>regardless</em> of how well your pre-trained “backbone” matches your use case.</p>
<p>With an eight-class classification problem (with about the same number of<br>
data samples in each class), random guesses will produce an accuracy of<br>
12.5%.</p>
<p>In your accuracy graph, your initial training accuracy (this is presumably the<br>
value after either no training or one epoch of training, depending on when you<br>
take your accuracy snapshot) is over 70%, which seems surprisingly high.<br>
Furthermore, your validation accuracy, although not as high, is still something<br>
like 45%, which is also much better than random guessing.</p>
<p>Your loss values also look fishy.  Your validation loss starts out worse than<br>
your training loss, and immediately goes up.  Even if your graphs start after<br>
the first epoch of training, it seems highly unlikely that your model (specifically<br>
its head) could have overfit after a single epoch of training (during which it<br>
would have seen each of the approximately 18,000 training images only once).</p>
<p>The results just don’t look sensible.</p>
<p>Again, depending on how your code is organized, the initial points on your<br>
graphs might be after training for an epoch.  If so, what are your training<br>
and validation losses and accuracies <em>before</em> training (that is, immediately<br>
after installing your randomly-initialized head)?  At this point, your training<br>
and validation losses should be nearly equal and your training and validation<br>
accuracies should both be approximately 12.5%.</p>
<p>Last, things like <code>BatchNorm</code> and <code>Dropout</code> behave differently in <code>train</code><br>
and <code>eval</code> mode.  To make sure that you are making an apples-to-apples<br>
comparison, it probably makes sense to plot training and validation losses<br>
and accuracies all computed with your model running in <code>eval</code> mode.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Schroter
cooked: <p>Hi, Thanks for the reply. You are right this graph is for a network that does not have any measures in place for overfitting. Does this mean anything? In the sense that the validation accuracy could be made better? Please find the attached images of the graphs for the network which with the measures for overfitting.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/c/fcaedb0858ee9fcbf130d339ff7f5550e3c7740b.png" data-download-href="https://discuss.pytorch.org/uploads/default/fcaedb0858ee9fcbf130d339ff7f5550e3c7740b" title="image_2023-11-02_014717216"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fcaedb0858ee9fcbf130d339ff7f5550e3c7740b_2_690x195.png" alt="image_2023-11-02_014717216" data-base62-sha1="A3kWPIZEejSUGXPseHgXueotWdB" width="690" height="195" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fcaedb0858ee9fcbf130d339ff7f5550e3c7740b_2_690x195.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fcaedb0858ee9fcbf130d339ff7f5550e3c7740b_2_1035x292.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fcaedb0858ee9fcbf130d339ff7f5550e3c7740b_2_1380x390.png 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image_2023-11-02_014717216</span><span class="informations">2410×684 70 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks &amp; Best Regards<br>
Michael</p>

username: KFrank
cooked: <p>Hi Michael!</p>
<aside class="quote no-group" data-username="Schroter" data-post="5" data-topic="190937" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/schroter/48/57964_2.png" class="avatar"> Schroter:</div>
<blockquote>
<p>You are right this graph is for a network that does not have any measures in place for overfitting.</p>
</blockquote>
</aside>
<p>I don’t know what you mean by this.</p>
<p>In general I don’t understand what you are doing here.  You’ve posted some<br>
graphs that look different than they did before, but with no explanation of<br>
what you changed.  (I hope that you’re not just blindly changing things until<br>
the results “look good.”)</p>
<p>Have you figured out why your previous graphs looked so fishy?</p>
<p>Have you figured out why – even in your current graphs – the initial values<br>
for your training and validation losses and accuracies differ from one another?</p>
<p>Why don’t your accuracies start at the 12.5% one would expect for random<br>
guessing?</p>
<aside class="quote no-group">
<blockquote>
<p>In the sense that the validation accuracy could be made better?</p>
</blockquote>
</aside>
<p>Ignoring the above-noted issues, it looks like your training has plateaued, but<br>
that overfitting has not set in.  Twenty epochs is not a lot, so I would recommend<br>
training more.  Sometimes training “gets stuck” on a temporary plateau, but gets<br>
off of it with additional training.</p>
<p>If your goal is to achieve a higher validation accuracy, I would suggest training<br>
until overfitting has clearly set in.  If your validation accuracy is then still not as<br>
high as you would like, consider various techniques for delaying overfitting,<br>
including data augmentation and reducing the capacity of your model.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Schroter
cooked: <p>Hi, Thanks for the reply. The second graph is the correct graph for this question. The first graph image was from a NN that did not have any measures (<code>WeightDecay</code>, <code>Dropout</code>, <code>BatchNorm</code>) in place in case of overfitting.<br>
Thanks &amp; Best Regards<br>
MIchael</p>