Same model,data and iteration,totally different forward output in train then eval mode,why?

AlexLuya: <p>Hello,as you can see,for same data in same iteration,when putting mode in train then eval mode,forward pass gives totally different outputs:</p>
<pre><code class="lang-auto">TRAIN output:[[  7.2314,  -9.7634], [-11.1361,  15.7153],  [ -6.0605,   7.8842],  [ -7.5571,  10.5594]]
EVAL   output:[[  9.7397, -12.8771],[  2.3994,  -3.3750], [  5.1334,  -6.9638], [  9.0693, -11.9809]]
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/f87f76eef30d7d0fc9c6a339543fae65075db678.png" data-download-href="https://discuss.pytorch.org/uploads/default/f87f76eef30d7d0fc9c6a339543fae65075db678" title="Workspace%201_810"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f87f76eef30d7d0fc9c6a339543fae65075db678_2_690x326.png" alt="Workspace%201_810" data-base62-sha1="zsjvgszc1YmsrZTf6P2t4DYWSp2" width="690" height="326" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f87f76eef30d7d0fc9c6a339543fae65075db678_2_690x326.png, https://discuss.pytorch.org/uploads/default/optimized/2X/f/f87f76eef30d7d0fc9c6a339543fae65075db678_2_1035x489.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/f/f87f76eef30d7d0fc9c6a339543fae65075db678.png 2x" data-dominant-color="3B3A3A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Workspace%201_810</span><span class="informations">1183×559 93.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
would you please tell me possible reasons?</p>

111137: <p>Hi, <a class="mention" href="/u/alexluya">@AlexLuya</a> -san,</p>
<p>It probably cause by <strong>train mode and eval mode are in same for loop.</strong><br>
After one training it updates parameters of your model has, so different result is presented on eval after the train.</p>

SoucheChapich: <p>Hello,</p>
<p>If your model has BatchNorm or DropOut modules this behavior is normal. They dont have same behavior in training and eval mode that is why you have different results</p>

AlexLuya: <p>Thanks,<br>
after changing code to:</p>
<pre><code class="lang-auto"> for batch, (images, labels) in dataIter:
            startTime = time.time()
            import torch
            with torch.no_grad():
                outputs = self.forward(images)
            # loss = self.criterion(outputs, labels.cuda())
            # self.backward(loss)
            self.backbone.eval()
            outputs_eval = self.forward(images)
            print("hello")
</code></pre>
<p>outputs is still totally different form output_eval</p>

111137: <p>My suggestion is to make different loop for train and eval. After finishing for-loop for the train you obtain trained parameters, so you can verify your parameters and your model too.</p>

AlexLuya: <p>Thanks,as you can see that,from second item,the classification results actually opposite against to each other in two modes</p>
<pre><code class="lang-auto">TRAIN output:[[  7.2314,  -9.7634], [-11.1361,  15.7153],  [ -6.0605,   7.8842],  [ -7.5571,  10.5594]]
EVAL   output:[[  9.7397, -12.8771],[  2.3994,  -3.3750], [  5.1334,  -6.9638], [  9.0693, -11.9809]]
</code></pre>
<p>and How can I use this evaluation result to evaluate the model?</p>

SoucheChapich: <p>As <a class="mention" href="/u/111137">@111137</a> suggested, I think that you should separate the two for loops. That could be something like:</p>
<pre><code class="lang-auto">for e in range(nb_epochs):
     model.train()
     for batch, (images, labels) in dataIterTrain:
           optimizer.zero_grad()
           do your stuff
    model.eval()
    with torch.no_grad(): 
     for batch, (images, labels) in dataIterVal:
          validate your model
          
</code></pre>
<p>And don’t forget that some modules have different behavior dependently of the mode of the model. So even at the end of the training loop, if you do something like:</p>
<pre><code class="lang-auto">model.train()
output_train = model(input)
model.eval()
output_eval = model(input)
</code></pre>
<p>The two output will be different.</p>

111137: <p>Let us consider about training and evaluation.</p>
<ul>
<li>Training is to update parameters by back propagation.</li>
<li>Evaluation is to verify the result of training.</li>
</ul>
<p>So these two modes should not been in same for-loop.<br>
The sequence of your code means;</p>
<ol>
<li>train the model, generates result, and update parameters.</li>
<li>eval the model using updated parameters, and generates result.</li>
<li>repeat this sequence by for-loop (going back to 1)</li>
</ol>
<p>So, different number is alway there, I think.<br>
<strong>NOTE</strong> : training and validation have not use same data set, in order to check overfitting on your model.</p>

AlexLuya: <p>Thanks,I actually do training and evaluation in different loop,and got train accuracy raising from 0.5 to 0.995 after several epochs,but evaluation accuracy stays at 0.5(a little bit higher or lower),due to this weird,I put some evaluation data in same loop to check the output in different mode,and found this,so I want to know why?</p>

111137: <p>Reason is this, you do it in  your code;</p>
<aside class="quote no-group" data-username="111137" data-post="8" data-topic="54428">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111137/48/15236_2.png" class="avatar"> 111137:</div>
<blockquote>
<ul>
<li>train the model, generates result, and update parameters.</li>
<li>eval the model using updated parameters, and generates result.</li>
<li>repeat this sequence by for-loop (going back to 1)</li>
</ul>
</blockquote>
</aside>
<p>Means your program correctly works.</p>

AlexLuya: <p>In these code:</p>
<pre><code class="lang-auto"> for batch, (images, labels) in dataIter:
            startTime = time.time()
            import torch
            with torch.no_grad():
                outputs = self.forward(images)
            # loss = self.criterion(outputs, labels.cuda())
            # self.backward(loss)
            self.backbone.eval()
            outputs_eval = self.forward(images)
            print("hello")
</code></pre>
<p>backward related code has been comment out,but outputs are still totally different.</p>

111137: <aside class="quote no-group" data-username="AlexLuya" data-post="11" data-topic="54428">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alexluya/48/15408_2.png" class="avatar"> AlexLuya:</div>
<blockquote>
<p>import torch</p>
</blockquote>
</aside>
<p>Let you try to move the import to outside of the loop.<br>
Your program does import every time in every iteration.<br>
And tell me reason why use <code>with torch.no_grad():</code>.</p>
<p>When removing the “import” and the “with” statement, what happen?</p>

SoucheChapich: <p><a class="mention" href="/u/alexluya">@AlexLuya</a> what are the modules of your backbone ?</p>

AlexLuya: <p><strong>InceptionResNetV2</strong></p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/AlexLuya/DSC-AI/blob/master/src/backbone/cnn/InceptionResNetV2.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/AlexLuya/DSC-AI/blob/master/src/backbone/cnn/InceptionResNetV2.py" target="_blank" rel="nofollow noopener">AlexLuya/DSC-AI/blob/master/src/backbone/cnn/InceptionResNetV2.py</a></h4>
<pre><code class="lang-py"># coding:utf-8
"""
Created on 15.06.2018
@author: alex
"""
from torch import nn
from torch.nn import Module

from backbone.cnn.inception.BasicConv2D import BasicConv2D
from backbone.cnn.inception.Block17 import Block17
from backbone.cnn.inception.Block35 import Block35
from backbone.cnn.inception.Block8 import Block8
from backbone.cnn.inception.ReductionA import ReductionA
from backbone.cnn.inception.ReductionB import ReductionB
from backbone.cnn.inception.Stem import Stem
import torch.nn.functional as F

# Inception-ResNet-v2
# https://github.com/rwightman/pytorch-planet-amazon/blob/master/models/inception_resnet_v2.py
# https://github.com/Cadene/pretrained-models.pytorch/blob/master/pretrainedmodels/models/inceptionresnetv2.py
</code></pre>

  This file has been truncated. <a href="https://github.com/AlexLuya/DSC-AI/blob/master/src/backbone/cnn/InceptionResNetV2.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


SoucheChapich: <p>So you are using InceptionResNetV2(module) which uses a F.dropout( 0.8 ) module by default. This module will change the behavior of your module in training or eval mode. In training mode, the dropout will supress some connections between the layers to overcome overfiting but in eval mode it will remain all the connection. So the output will change.</p>

SoucheChapich: <p>I think that if you set dropRate to 0 the output will be the same for both eval and train. (But in that case the DropOut module will be useless)</p>

AlexLuya: <p>Thanks,after moving<br>
<code>import torch</code><br>
outside of the loop,these two outputs are still totally different.</p>
<p>For why using “with torch.no_grad()”:<br>
After commenting out the backward,the computation graph won’t be deleted,and I got a CUDA out of memory exception,so I use<br>
<code>torch.no_grad():</code><br>
to tell pytorch don’t hold computation graph</p>

111137: <p>Ok,</p>
<p>Let you do the dropout rate = 0.0 as suggestion of <a class="mention" href="/u/souchechapich">@SoucheChapich</a> -san. If result is same then reason is the dropout. If it still shows different number then the train and eval in same loop makes the difference.</p>

AlexLuya: <p>Thank,I actually knew this,the problem is that the model already gives me 99.5% training accuracy,but evaluation accuracy remains at 50%,even using training data to do evaluation,the accuracy is still 0.5.and both train and eval accuracy computation used the same code.</p>

SoucheChapich: <p>That is strange because if you use exactly the same data and that you take care of the dropout, the results should be the same. Can you show us the entire model ? Not just the backbone</p>