Interpret plots... should i continue training?

username: natxo
cooked: <p>Training loss is descending, but test loss and mAP don’t… what will be the most common approach? Continue training or stop.</p>
<p>What you should do?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a130492f15686d4e532a83f34eed2198e25235e0.png" data-download-href="https://discuss.pytorch.org/uploads/default/a130492f15686d4e532a83f34eed2198e25235e0" title="plot"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a130492f15686d4e532a83f34eed2198e25235e0_2_562x499.png" alt="plot" data-base62-sha1="mZWk8lqhGPs2SaoSfk0cdfpH05i" width="562" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a130492f15686d4e532a83f34eed2198e25235e0_2_562x499.png, https://discuss.pytorch.org/uploads/default/original/3X/a/1/a130492f15686d4e532a83f34eed2198e25235e0.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/1/a130492f15686d4e532a83f34eed2198e25235e0.png 2x" data-dominant-color="F9FAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">plot</span><span class="informations">751×667 36.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks for comment</p>

username: KFrank
cooked: <p>Hi naxto!</p>
<aside class="quote no-group" data-username="natxo" data-post="1" data-topic="155681" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/natxo/48/51109_2.png" class="avatar"> natxo:</div>
<blockquote>
<p>Training loss is descending, but test loss and mAP don’t…</p>
</blockquote>
</aside>
<p>To my eye, it looks as if your training has started <em>overfitting</em> your training<br>
set.</p>
<p>I say this because your validation loss has started going up (albeit modestly)<br>
while your training loss is continuing to go down.  The idea is that your<br>
network is continuing to learn – your training loss is going down – but it’s<br>
learning details specific to your training set that aren’t actually relevant to<br>
your real problem (as represented by your validation set).</p>
<p>All else being the same, you should stop training when your network starts<br>
overfitting.</p>
<p>If you have the resources (and patience) you could double-check this by<br>
rerunning your training from scratch starting with a different random<br>
initialization of your network (and also a different random split between<br>
your training and validation datasets).  I expect that if you do so you will<br>
see the same general structure – validation loss starts going up, while<br>
the training loss continues to decrease – although the exact numbers will<br>
differ.</p>
<p>If you care, some things you can try in order to reduce / postpone overfitting:</p>
<p>Weight decay and <a href="https://pytorch.org/docs/1.12/generated/torch.nn.Dropout.html" rel="noopener nofollow ugc">dropout</a> can reduce overfitting.</p>
<p>If you still need to achieve better validation / test performance, the best<br>
way to reduce overfitting is to train with a larger training set – if you have,<br>
or can acquire, more training data.  If you don’t have more data, you can<br>
try <a href="https://pytorch.org/vision/0.13/transforms.html" rel="noopener nofollow ugc">data augmentation</a> to get some of the benefits of more actual data.</p>
<p>Best.</p>
<p>K. Frank</p>

username: oke-aditya
cooked: <p>My suggestion is to take a word from Karpathy’s advice <a href="http://karpathy.github.io/2019/04/25/recipe/" class="inline-onebox" rel="noopener nofollow ugc">A Recipe for Training Neural Networks</a></p>
<p>Overfit, regularize and then Tune <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"> I think the blog will explain better than anyone else.</p>

username: natxo
cooked: <p><a class="mention" href="/u/kfrank">@KFrank</a> Ok, got it!!</p>
<p>Even test loss increase very very small, as train loss is going down means that’s is only learning features that are exclusive from train set, as when try to use those on test set, loss increase so whats learning is not useful</p>
<p><a class="mention" href="/u/oke-aditya">@oke-aditya</a> thanks for that link is super helpful!!</p>
<p>So or i get the model at epoch 30 or try to improve training… i will try to improve <img src="https://discuss.pytorch.org/images/emoji/apple/sunglasses.png?v=12" title=":sunglasses:" class="emoji" alt=":sunglasses:" loading="lazy" width="20" height="20"></p>
<p>After reading KFrank comment and that karpathy blog im going to:</p>
<ul>
<li>reduce image input size from 1024 to 800</li>
<li>use Adam optimizer lr 0.0001 and weight_decay 0.0005</li>
<li>decrease batch size from 6 to 4</li>
<li>increase data argumentation</li>
</ul>
<p>Thanks to both, i’m learning a lot here!!</p>