Training with model.eval() works while in model.train() the model overfits in one epoch

username: stadtgiraffe
cooked: <p>Hello everyone,<br>
I have a noticed a misbehaviour of my model.<br>
The problem is quite similar to <a href="https://discuss.pytorch.org/t/training-model-in-eval-mode-gives-better-result/31004" class="inline-onebox">Training model in eval() mode gives better result ?!</a> .<br>
I just use BatchNorm2d layer instead of dropout layer.</p>
<p>If I set my model.train() outside of my epoch loop (instead of inside) my model trains in train mode<br>
only in the first epoch. After the first evaluation (which shows very low evaluation confidence while having high training confidence) the model stays in eval mode and continues training.<br>
For that configuration the loss increases after evaluation shortly and than decreases again. (evaluation each 200 batches, see image)- I assume the weights leave a local minimum. After that, the training continues perfectly and I get good evaluation and detection results (Yolov3 code).</p>
<p>But if I set the model.train() inside the loop, the “overfitting” continues since I go back to train mode after evaluation. There is no peak in the loss function and my evaluation confidence continues to decrease (even below 0.01) even more while my training confidence converges for all three detection layers to 0.999.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/4/640303ce0309adabf489ff99381c73daf7ef1458.png" alt="Unbenannt" data-base62-sha1="egKcNeLhu4pNbtEOriplZ82Tm7C" width="434" height="288"></p>
<p>I know for Dropout Layers I could just leave the Dropout Layer out or I could increase the layers. But what do I have to change for BatchNorm layers? Leaving the bn-layers completely out, leads to loss divergence.</p>
<p>Thanks in advance for the help!</p>

username: ptrblck
cooked: <p>I’m not sure I understand the issue completely.</p>
<p>Based on this statement:</p>
<aside class="quote no-group" data-username="stadtgiraffe" data-post="1" data-topic="88533">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/d6d6ee/48.png" class="avatar"> stadtgiraffe:</div>
<blockquote>
<p>But if I set the model.train() inside the loop, the “overfitting” continues since I go back to train mode after evaluation. There is no peak in the loss function and my evaluation confidence continues to decrease (even below 0.01) even more while my training confidence converges for all three detection layers to 0.999.</p>
</blockquote>
</aside>
<p>it sounds like the expected result. I.e. the training and evaluation losses decrease properly in their corresponding mode or are you using <code>model.train()</code> for both, training and evaluation?</p>

username: stadtgiraffe
cooked: <p>The training loss decreases in training mode while the evaluation loss increases in eval mode (which is the basic sign for overfitting).<br>
My issue is that I do not fully understand why training works better (much better mAP performance) if I freeze my batchnorm layer after epoch 1, meaning just continue training in model.eval() instead of model.train() after one epoch.<br>
I tried adding Dropout additional to the BN layer to prevent fast overfitting but it still shows that my training works better if I switch to model.eval() for training.</p>

username: ptrblck
cooked: <p>Thanks for the update.<br>
It sounds as is the running estimates might overfit to the training data.<br>
How large is the currently used batch size and did you change the <code>momentum</code> in the batchnorm layers?<br>
If the training batch size is small (e.g. &lt;16), the running stats might be too noisy, which would result in a bad validation performance.</p>