Model accuracy increases gradually in `model.eval()`

janandd: <p>I have modified the <code>fc</code> layer in a resnet18 model to output 4 logits in a classification problem. To test my model am using the following function.</p>
<pre><code class="lang-auto">def test_model(model, test_loader, train=False):
    with torch.no_grad():
        _ = model.train() if train else model.eval()
        labs_list, prds_list, test_corrects = [], [], 0
        for i,(inps,labs) in enumerate(test_loader):
            inps, labs = inps.to(DEVICE), labs.to(DEVICE)
            outs = model(inps)
            prds = outs.argmax(dim=1)
            test_corrects += (prds==labs).sum().item()
    print(f'model.training = {str(model.training):&gt;5},  ' \
                + f'test_acc = {test_corrects/len(test_loader.dataset):0.6f}')
</code></pre>
<p>The <code>train</code> argument in the function decides if <code>model.train()</code> is used or <code>model.eval()</code>. It is expected that <code>model.eval()</code> should always produce the same value of accuracy. However, if I run the following piece of code:</p>
<pre><code class="lang-auto">for i in range(10):
    test_model(net, test_loader)
    test_model(net, test_loader, train=True)
</code></pre>
<p>i.e. running the <code>train</code> and <code>eval</code> alternatively, accuracy in train model remains the same every time, but accuracy in <code>eval</code> mode gradually increases until it reaches a value, after which there is no further increase. This gradual saturation of accuracy is not observed if the <code>train</code> mode is removed from the <code>for</code> loop. This suggests that changes in the model after each <code>train</code> mode, affects the <code>eval</code> mode as well, which is really perplexing.</p>
<p>I suspect it might be wrong for the <code>test_model</code> function to call <code>train</code> mode inside <code>torch.no_grad()</code>. But I am still not sure how it affects the model performance in the <code>eval</code> mode.</p>

ptrblck: <aside class="quote no-group" data-username="janandd" data-post="1" data-topic="166177">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/4bbf92/48.png" class="avatar"> janandd:</div>
<blockquote>
<p>This suggests that changes in the model after each <code>train</code> mode, affects the <code>eval</code> mode as well, which is really perplexing.</p>
</blockquote>
</aside>
<p>This could be expected if you are using e.g. batchnorm layers, which update their running stats in each forward pass in training mode. The <code>eval()</code> mode would then potentially return different outputs until the running stats converged.</p>

janandd: <p>I am using Dropout in fc, so that must be the case. Thank you!<br>
I suppose then that the correct value of accuracy for the model would be the one we get upon running <code>test_model</code> once in <code>eval</code> mode, and not the one obtained after running the <code>for</code> loop above. Also, is there any case where you would have to use <code>train</code> inside <code>torch.no_grad()</code>?</p>