Train loss and test loss increasing after +/- 600 epochs

nestlee: <p>Hello,</p>
<p>I am running a multichannel UNET on MRI images of different sequences.<br>
After +/- 600 epoch I get these losses (blue = train loss, orange = test loss) :</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a6ecd58a103f919085ed6786d1221078d964c2a9.png" data-download-href="https://discuss.pytorch.org/uploads/default/a6ecd58a103f919085ed6786d1221078d964c2a9" title="Figure_1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6ecd58a103f919085ed6786d1221078d964c2a9_2_690x204.png" alt="Figure_1" data-base62-sha1="nOGFQilDLbfS5GdPLe0yOqoyPpv" width="690" height="204" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6ecd58a103f919085ed6786d1221078d964c2a9_2_690x204.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6ecd58a103f919085ed6786d1221078d964c2a9_2_1035x306.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6ecd58a103f919085ed6786d1221078d964c2a9_2_1380x408.png 2x" data-dominant-color="FCFBF9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Figure_1</span><span class="informations">1616×480 46.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Why does the losses increase (a lot) after 600 epochs ? What should I do ?<br>
And I think that my model overfits, the gap between train loss and test loss tells us…, how can I reduce this gap ?</p>
<p>Some params :</p>
<ul>
<li>loss function = dice loss</li>
<li>optimizer = adam optim</li>
<li>batch size = 10</li>
<li>learning rate = 1e-4</li>
</ul>

KFrank: <p>Hi Nestlee!</p>
<aside class="quote no-group quote-modified" data-username="nestlee" data-post="1" data-topic="148807" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/9e8a1a/48.png" class="avatar"> nestlee:</div>
<blockquote>
<p>I am running a multichannel UNET<br>
…<br>
Why does the losses increase (a lot) after 600 epochs ?</p>
</blockquote>
</aside>
<p>It looks like your model / training is becoming unstable, which is something<br>
that can happen.  Note that the <code>Adam</code> optimizer can be a little finicky and<br>
unstable, even though it often trains faster when it’s working.</p>
<aside class="quote no-group">
<blockquote>
<p>What should I do ?</p>
</blockquote>
</aside>
<p>First, try using <code>Adam</code>’s <code>weight_decay</code> parameter (if you aren’t already).<br>
Depending on the cause of your instability, this might very well help.</p>
<aside class="quote no-group">
<blockquote>
<p>Some params :</p>
<ul>
<li>loss function = dice loss</li>
</ul>
</blockquote>
</aside>
<p>Your using UNET suggests that you are performing semantic segmentation,<br>
which is a per-pixel classification problem.</p>
<p>I would recommend <em>not</em> using pure dice loss.  I would suggest starting with<br>
pure <code>CrossEntropyLoss</code> – the go-to loss criterion for classification – with<br>
class weights (the <code>weight</code> argument), if you have class imbalances.  If you<br>
want to use dice loss, I would suggest <em>augmenting</em> <code>CrossEntropyLoss</code><br>
with dice loss.</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li>optimizer = adam optim</li>
</ul>
</blockquote>
</aside>
<p>As mentioned above, consider using <code>weight_decay</code>.</p>
<p>If you still have stability problems, consider switching to the <code>SGD</code><br>
optimizer, most likely with <code>momentum</code> and probably with <code>weight_decay</code>.<br>
<code>SGD</code> tends to be better behaved, even if it doesn’t train as fast.</p>
<p>Note that the learning rate (the <code>lr</code> argument) doesn’t map directly<br>
from <code>Adam</code> to <code>SGD</code>.  You will likely want to use a significantly higher<br>
value for <code>lr</code> when using <code>SGD</code>.</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li>batch size = 10</li>
</ul>
</blockquote>
</aside>
<p>(Unlikely to be the cause of your problem.)</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li>learning rate = 1e-4</li>
</ul>
</blockquote>
</aside>
<p>Consider reducing the learning rate significantly – perhaps by one or two<br>
(or more) orders of magnitude – before the instability sets in – perhaps<br>
around epoch 500.</p>
<aside class="quote no-group">
<blockquote>
<p>And I think that my model overfits, the gap between train loss and test loss tells us…, how can I reduce this gap ?</p>
</blockquote>
</aside>
<p>It’s not clear that your model it overfitting, per se.  Both your train and test<br>
losses go down nicely before they both plateau.  It may be that you are<br>
doing as well as you can, given your specific model and the data you have.</p>
<p>However, sometimes training does plateau for quite a while, but then<br>
starts getting better again as you keep training.  So, if you can avoid the<br>
instability, you might be able to train (much) longer, and get past your<br>
(long-lasting, but temporary) plateau.</p>
<p>As a general rule, to do better on your test set, it helps to train with more<br>
training data.  Of course, you might not have more data to train on, but<br>
you can often get significant benefit by using <a href="https://pytorch.org/vision/0.12/transforms.html" rel="noopener nofollow ugc">data augmentation</a> to “derive”<br>
some additional partially “fake” data from your “real” training data.</p>
<p>Good luck.</p>
<p>K. Frank</p>