Val loss doesn't decrease after a certain number of epochs

Saswat: <p>I’m working on a classification problem (500 classes). My NN has 3 fully connected layers, followed by an LSTM layer. I use <code>nn.CrossEntropyLoss()</code> as my loss function. This is my network’s configuration</p>
<pre><code class="lang-auto">Model(
  (fc): Sequential(
    (0): Linear(in_features=800, out_features=1024, bias=True)
    (1): ReLU()
    (2): Linear(in_features=1024, out_features=512, bias=True)
    (3): ReLU()
    (4): Linear(in_features=512, out_features=256, bias=True)
    (5): ReLU()
  )
  (lstm): LSTM(256, 1024, bidirectional=True)
  (hidden2tag): Sequential(
    (0): Linear(in_features=2048, out_features=1024, bias=True)
    (1): ReLU()
    (2): Linear(in_features=1024, out_features=500, bias=True)
  )
)
</code></pre>
<p>This is what my loss looks like. It increases after reaching a particular value no matter what setting I use. I’ve used k-fold cross–validation but the loss/accuracy across all folds stays the same, so I’m assuming that there’s no issue in the distribution of train/val splits.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/7/27eb7bc3dd106c582332f3f0fe5d082568993245.png" data-download-href="https://discuss.pytorch.org/uploads/default/27eb7bc3dd106c582332f3f0fe5d082568993245" title="Screenshot 2022-10-25 at 8.20.34 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27eb7bc3dd106c582332f3f0fe5d082568993245_2_690x164.png" alt="Screenshot 2022-10-25 at 8.20.34 AM" data-base62-sha1="5H98P6Mvusvr8WzFLNjVfTgxHy5" width="690" height="164" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27eb7bc3dd106c582332f3f0fe5d082568993245_2_690x164.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27eb7bc3dd106c582332f3f0fe5d082568993245_2_1035x246.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27eb7bc3dd106c582332f3f0fe5d082568993245_2_1380x328.png 2x" data-dominant-color="FAF8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-10-25 at 8.20.34 AM</span><span class="informations">2086×496 76.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’ve tried using step-wise LR scheduling (tried OneCycleLR and Multiplicative LR as well) but the loss still doesn’t improve<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6c5faafb876b609eb41c4aa8e24b9bb3aee93956.png" data-download-href="https://discuss.pytorch.org/uploads/default/6c5faafb876b609eb41c4aa8e24b9bb3aee93956" title="Screenshot 2022-10-25 at 8.20.56 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6c5faafb876b609eb41c4aa8e24b9bb3aee93956_2_690x488.png" alt="Screenshot 2022-10-25 at 8.20.56 AM" data-base62-sha1="fsIx80cx3kwLERnbcCi9fvbekmi" width="690" height="488" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6c5faafb876b609eb41c4aa8e24b9bb3aee93956_2_690x488.png, https://discuss.pytorch.org/uploads/default/original/3X/6/c/6c5faafb876b609eb41c4aa8e24b9bb3aee93956.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/c/6c5faafb876b609eb41c4aa8e24b9bb3aee93956.png 2x" data-dominant-color="FAF9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-10-25 at 8.20.56 AM</span><span class="informations">700×496 21.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’ve tried using dropout but that gives higher validation loss values instead <a href="https://discuss.pytorch.org/t/higher-validation-loss-after-using-dropout/164160">ref</a></p>
<p>What can be done here in order to further decrease the loss?</p>

KFrank: <p>Hi Saswat!</p>
<aside class="quote no-group" data-username="Saswat" data-post="1" data-topic="164308" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saswat/48/49862_2.png" class="avatar"> Saswat:</div>
<blockquote>
<p>I’m working on a classification problem (500 classes).</p>
</blockquote>
</aside>
<p>Note that 500 classes is a fairly large number.  It is reasonable to expect<br>
that you might need a large amount of training data to train effectively.</p>
<aside class="quote no-group">
<blockquote>
<p>This is what my loss looks like. It increases after reaching a particular value no matter what setting I use.</p>
</blockquote>
</aside>
<p>The graphs you posted show your training loss continuing to go down while<br>
your validation loss turns around and starts going up.  This is a typical sign<br>
of <em>overfitting.</em></p>
<p>The best way to avoid / reduce overfitting is to train with more data.  If<br>
you simply don’t have more training data it can be helpful to use <em>data<br>
augmentation.</em>  What you do is make some “synthetic” training data by<br>
modifying copies of your real training data.  It’s worth a try.</p>
<p>If data augmentation doesn’t work, the next step is to make your model<br>
simpler (reduce its “capacity”).  You can make your model narrower or<br>
shallower or both.  The basic idea is that the more complicated your model<br>
is, the more parameters it has with which to overfit – that is to learn specific<br>
details of your training data that aren’t actually relevant to the general<br>
problem you are trying to solve.  In effect, your model “memorizes” the<br>
correct answers (predictions) for the specific samples in your training<br>
set without really learning how to make predictions for out-of-sample<br>
data.  Hence the higher loss for your validation set (that you didn’t train<br>
with).</p>
<aside class="quote no-group">
<blockquote>
<p>I’ve tried using step-wise LR scheduling (tried OneCycleLR and Multiplicative LR as well) but the loss still doesn’t improve</p>
</blockquote>
</aside>
<p>Using a learning-rate scheduler could well help you train more efficiently<br>
<em>on your training set,</em> but it won’t address the overfitting problem.</p>
<aside class="quote no-group">
<blockquote>
<p>I’ve tried using dropout but that gives higher validation loss values</p>
</blockquote>
</aside>
<p>Dropout can help prevent overfitting.  But you say that it’s not working for<br>
you, so your best approach will be to acquire more training data.  Barring<br>
that, data augmentation is worth a shot.</p>
<p>You could also try simplifying your model, but there is the risk that if you<br>
make your model too simple, it won’t have enough structure to be able to<br>
learn what sounds like a potentially challenging (500 classes) classification<br>
problem.</p>
<p>This comes back to my original comment – harder problems require more<br>
training data.</p>
<p>Good luck.</p>
<p>K. Frank</p>

Saswat: <p><a class="mention" href="/u/kfrank">@KFrank</a> After adding more training data, the validation loss plateaus at of 3.2 (lower than the previous case). What could be the possible cause for this? How can the loss be driven further down?</p>

KFrank: <p>Hi Saswat!</p>
<aside class="quote no-group" data-username="Saswat" data-post="3" data-topic="164308" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saswat/48/49862_2.png" class="avatar"> Saswat:</div>
<blockquote>
<p>How can the loss be driven further down?</p>
</blockquote>
</aside>
<p>You can’t magically lower the validation loss – training, by definition, lowers<br>
the training loss.</p>
<p>When your validation loss plateaus, is the training loss still going down?  If<br>
so, you might be seeing the beginning of overfitting.  (If you are overfitting,<br>
see my comments posted above.)</p>
<p>If the training loss has plateaued, you need to focus on figuring out how<br>
to train so as to further lower your training loss (and see whether that also<br>
lowers your validation loss).</p>
<p>First, try training (much) longer.  Complicated models do sometimes get<br>
“stuck” in plateaus, but then sometimes get “unstuck” after (a lot) more<br>
training.</p>
<p>Experiment with momentum and  various optimizers, and, as you’ve already<br>
tried, with learning-rate schedulers.  Oddly enough, lowering your learning<br>
rate can sometimes let you “sneak” out of a plateau while increasing your<br>
learning rate can sometimes “jolt” you out of a plateau.</p>
<p>You don’t say how many epochs you’ve tried training with, but the graph<br>
you posted suggests that you’ve used 500 epochs.  That’s really not a lot,<br>
especially for complicated problems (and lots of training data).  You might<br>
just need to be patient and experiment with training <em>a lot</em> more.</p>
<p>Best.</p>
<p>K. Frank</p>

Saswat: <p><a class="mention" href="/u/kfrank">@KFrank</a><br>
I tried adding more data and training longer. Previously, the number of data points was 3k and now there are 66k. I’m now training for 1500 epochs instead of 500 (before) with a very low learning rate (1e-5, before 6.4e-4). This is how the loss plots look like</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6eaad0fe03c08ed746fb59dc5f5ad0f10174e62.png" data-download-href="https://discuss.pytorch.org/uploads/default/d6eaad0fe03c08ed746fb59dc5f5ad0f10174e62" title="Screenshot 2022-10-31 at 8.29.16 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d6eaad0fe03c08ed746fb59dc5f5ad0f10174e62_2_690x160.png" alt="Screenshot 2022-10-31 at 8.29.16 AM" data-base62-sha1="uFeY7FCv5pcmS5f9LiEU2OBW5Y6" width="690" height="160" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d6eaad0fe03c08ed746fb59dc5f5ad0f10174e62_2_690x160.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d6eaad0fe03c08ed746fb59dc5f5ad0f10174e62_2_1035x240.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d6eaad0fe03c08ed746fb59dc5f5ad0f10174e62_2_1380x320.png 2x" data-dominant-color="FAF8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-10-31 at 8.29.16 AM</span><span class="informations">2136×496 64.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What would you advise me to do in this case?</p>

KFrank: <p>Hi Saswat!</p>
<aside class="quote no-group" data-username="Saswat" data-post="5" data-topic="164308" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saswat/48/49862_2.png" class="avatar"> Saswat:</div>
<blockquote>
<p>What would you advise me to do in this case?</p>
</blockquote>
</aside>
<p>Train more.</p>
<p>Your validation loss is still going down (as is your training loss), so there is<br>
likely more to be gained (and you haven’t started overfitting yet).</p>
<p>I’ve never experimented with long-short-term-memory models, but, in general<br>
for real models running on real problems, 1500 epochs is a very small number.</p>
<p>Also, try experimenting with various optimizers and their hyperparameters, as<br>
well as with the learning rate.</p>
<p>Good luck!</p>
<p>K. Frank</p>

Saswat: <p>Sure. I’ll give that a shot</p>

Messiah: <p>Did your results improve by merely training for more epochs as suggested?</p>

Saswat: <p>It improved for some time but then the model began overfitting</p>

Messiah: <p>How much lower did you manage to drive the validation loss?</p>

Saswat: <p>I was able to drive it to 3</p>

Messiah: <p>Do you think that adding more training data would drive down the loss further?</p>