Worse performance when executing model.eval() than model.train()

sina_malakouti: <p>My network performance on the test set gets much worse after some iterations when applying the model. eval() statement. However, if I do the same thing without having model. eval(), the performance is much better.  Does anyone know how I can solve this problem?</p>

1chimaruGin: <p>could you share some code snippet?</p>

sina_malakouti: <p>thanks for your reply.<br>
<a class="mention" href="/u/1chimarugin">@1chimaruGin</a><br>
So my network is almost like Unet.<br>
and let’s say I train/evaluate my model in the following ways:</p>
<ol>
<li>
</li></ol>
<p>model.eval()<br>
evaluate the model on test set<br>
model.train()<br>
train the model on train set</p>
<ol start="2">
<li>
</li></ol>
<p>model.train()<br>
evaluate model on test set<br>
model.train()<br>
train the model on train set</p>
<p>my batch size is 5.<br>
So, the only difference is that in the second case, before evaluating the model on test set I don’t call model.eval().</p>
<p>In the second case I get a better performance. I have both dropout and batch norm in my network and I know that they behave differently if calling model.eval(). I think the problem is because of batchnorm and maybe running stats but I don’t know how to fix it.</p>

1chimaruGin: <aside class="quote no-group" data-username="sina_malakouti" data-post="3" data-topic="107386">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sina_malakouti/48/32830_2.png" class="avatar"> sina_malakouti:</div>
<blockquote>
<p>In the second case I get a better performance.</p>
</blockquote>
</aside>
<p>Did you mean evaluation accuracy and loss?<br>
I think if your model doesn’t include <code>Dropout</code> and <code>BatchNorm</code>, <code>model.train()</code> and <code>model.eval()</code> won’t be different. If not, each of them can effect the result. For me, evaluation with <code>model.eval()</code> seem legit. Correct me if I’m wrong.</p>
<p>Pytorch <a>source code</a></p>
<pre><code class="lang-auto">def train(self: T, mode: bool = True) -&gt; T:
        r"""Sets the module in training mode.

        This has any effect only on certain modules. See documentations of
        particular modules for details of their behaviors in training/evaluation
        mode, if they are affected, e.g. :class:`Dropout`, :class:`BatchNorm`,
        etc.

        Args:
            mode (bool): whether to set training mode (``True``) or evaluation
                         mode (``False``). Default: ``True``.

        Returns:
            Module: self
        """
        self.training = mode
        for module in self.children():
            module.train(mode)
        return self

    def eval(self: T) -&gt; T:
        r"""Sets the module in evaluation mode.

        This has any effect only on certain modules. See documentations of
        particular modules for details of their behaviors in training/evaluation
        mode, if they are affected, e.g. :class:`Dropout`, :class:`BatchNorm`,
        etc.

        This is equivalent with :meth:`self.train(False) &lt;torch.nn.Module.train&gt;`.

        Returns:
            Module: self
        """
        return self.train(False)
</code></pre>
<ul>
<li>official pytorch tutorial with <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="noopener nofollow ugc"><code>model.eval()</code></a> and <a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html" rel="noopener nofollow ugc"><code>model.train()</code></a></li>
</ul>

sina_malakouti: <p>Yes.  I mean accuracy.</p>
<p>And yes, you are right. I have batch norm and drop out.</p>
<p>However, the problem is when I exclude model.eval() ( i.e., perform evaluation without executing model.eval() ) my performance is way better than when I do the evaluation after executing model.eval().</p>
<p>The accuracy when having model.eval() doesn’t make any sense. It’s like the network is not learning at all. However, when I remove model.eval() the accuracy on the test set actually makes sense.</p>

1chimaruGin: <aside class="quote quote-modified" data-post="2" data-topic="107438">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jamesdickens/48/45317_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/saved-cnn-model-giving-different-results/107438/2">Saved CNN model giving different results</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    You could debug this by: 
def areTorchModulesEqual(module1, module2):
    for index, (p1, p2) in enumerate(zip(module1.parameters(), module2.parameters())):
        if p1.data.ne(p2.data).sum() &gt; 0:
            return False
    return True

def whichModulesHaveBeenUpdated(model1_list, model2_list):
    for index, module1 in enumerate(model1_list):
        module2 = model2_list[index]
        if areTorchModulesEqual(module1, module2):
            print("At Index " + str(index) + " Module1 and Mod…
  </blockquote>
</aside>


Llama2: <p>Did you use multiple layers in your forward method with the same name from your <strong>init</strong> method?<br>
e.g.<br>
in init,<br>
self.conv1 =  nn.Conv2d(…)<br>
in forward,<br>
…<br>
x = conv1(x)<br>
x = conv1(x)<br>
…</p>
<p>but instead do:<br>
init,<br>
self.conv1 = nn.conv2d(…)<br>
self.conv2 = nn.conv2d(…)<br>
forward,<br>
…<br>
x = conv1(x)<br>
x = conv2(x)<br>
…</p>
<p>I encountered a similar issue with eval() giving different results, try it</p>
<p>I recall a forum post that solved this issue, i believe it could be related to some pytorch internals</p>

Shima_Shahfar: <p>Quick question just to make sure, are you using <code>torch.no_grad()</code> or <code>torch.zero_grad()</code> for validation/test time when you remove <code>model.eval()</code>?</p>
<p>Also, you may want to take a look at these two discussions on this topic (<a href="https://discuss.pytorch.org/t/performance-highly-degraded-when-eval-is-activated-in-the-test-phase/3323">1</a> and <a href="https://discuss.pytorch.org/t/model-eval-gives-incorrect-loss-for-model-with-batchnorm-layers/7561">2</a>) in case you haven’t seen them.</p>