Fluctuating performance of pre-trained model (frozen parameter) -- model.eval() doesn't work for dropout layer

username: shirui-japina
cooked: <p>I trained a model, the performance during the training is here:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/4/1407aa6efdd22b4ddb08c790460d899f0fa480a7.png" data-download-href="https://discuss.pytorch.org/uploads/default/1407aa6efdd22b4ddb08c790460d899f0fa480a7" title="train_validation_acc"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/1407aa6efdd22b4ddb08c790460d899f0fa480a7_2_517x332.png" alt="train_validation_acc" data-base62-sha1="2RbYl2xKcpgvL8jcgL6zjdKyeB9" width="517" height="332" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/1407aa6efdd22b4ddb08c790460d899f0fa480a7_2_517x332.png, https://discuss.pytorch.org/uploads/default/original/3X/1/4/1407aa6efdd22b4ddb08c790460d899f0fa480a7.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/4/1407aa6efdd22b4ddb08c790460d899f0fa480a7.png 2x" data-dominant-color="F9F7F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">train_validation_acc</span><span class="informations">700×450 37.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/7/07dc1508704aacf681678d75ade48c331aa9cfe7.png" data-download-href="https://discuss.pytorch.org/uploads/default/07dc1508704aacf681678d75ade48c331aa9cfe7" title="train_validation_loss"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/07dc1508704aacf681678d75ade48c331aa9cfe7_2_517x332.png" alt="train_validation_loss" data-base62-sha1="17wRHsV9TgYh2qBwXz3LKLdPl55" width="517" height="332" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/07dc1508704aacf681678d75ade48c331aa9cfe7_2_517x332.png, https://discuss.pytorch.org/uploads/default/original/3X/0/7/07dc1508704aacf681678d75ade48c331aa9cfe7.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/7/07dc1508704aacf681678d75ade48c331aa9cfe7.png 2x" data-dominant-color="F9F7F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">train_validation_loss</span><span class="informations">700×450 35.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I thought the parameters of epoch 44, is the best one. Then I loaded the parameters of epoch 44:</p>
<pre><code class="lang-auto">model = Model()
model.to(device=args.device)
model.load_state_dict(torch.load(args.weight_path))
criterion_reduction_none = nn.CrossEntropyLoss(reduction='none')
model.eval()
</code></pre>
<p>After that, I tried to predict <strong>the same data set</strong> to observe the performance of the model.<br>
Originally, I thought that the prediction result of the model would <em><strong>always</strong></em> be the value of epoch 44.<br>
In my opinion, it’s just like getting the value of <code>y = f(x)</code>.<br>
The <code>f</code> is just like the frozen parameter model; the <code>x</code> is just like the data set here.<br>
But in fact, the performance of the model is <em><strong>fluctuating</strong></em> within a certain range like:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/1/81d6bcd7bc2b24368aa1177121fc3f52a2d38ff8.png" data-download-href="https://discuss.pytorch.org/uploads/default/81d6bcd7bc2b24368aa1177121fc3f52a2d38ff8" title="test_acc"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/81d6bcd7bc2b24368aa1177121fc3f52a2d38ff8_2_517x332.png" alt="test_acc" data-base62-sha1="iwBFm0UbxA9Swr2sqOmILz3Ge6c" width="517" height="332" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/81d6bcd7bc2b24368aa1177121fc3f52a2d38ff8_2_517x332.png, https://discuss.pytorch.org/uploads/default/original/3X/8/1/81d6bcd7bc2b24368aa1177121fc3f52a2d38ff8.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/1/81d6bcd7bc2b24368aa1177121fc3f52a2d38ff8.png 2x" data-dominant-color="F4F8FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">test_acc</span><span class="informations">700×450 40.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f36d062da9cb730895e036a01f5bf62045ff443b.png" data-download-href="https://discuss.pytorch.org/uploads/default/f36d062da9cb730895e036a01f5bf62045ff443b" title="test_loss"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f36d062da9cb730895e036a01f5bf62045ff443b_2_517x332.png" alt="test_loss" data-base62-sha1="yJrBISypGpJTBFFBqKO4lEXmf7d" width="517" height="332" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f36d062da9cb730895e036a01f5bf62045ff443b_2_517x332.png, https://discuss.pytorch.org/uploads/default/original/3X/f/3/f36d062da9cb730895e036a01f5bf62045ff443b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/3/f36d062da9cb730895e036a01f5bf62045ff443b.png 2x" data-dominant-color="F6F9FB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">test_loss</span><span class="informations">700×450 38.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Why it came out like this?<br>
Here is my model:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def forward(self, input_image):
        out = self.conv_1(input_image) # 20 * 44 * 44
        out = self.pooling_1(out) # 20 * 22 * 22

        out = self.conv_2(out) # 50 * 16 * 16
        out = self.pooling_2(out) # 50 * 8 * 8

        out = self.conv_3(out) # 500 * 2 * 2
        out = self.pooling_3(out) # 500 * 1 * 1

        out = F.dropout(out)
        out = F.relu(out)
        out = self.conv_4(out) # 2 * 1 * 1
</code></pre>
<p>I heard that <code>model.eval()</code> is needed when there is a dropout layer in the model. I actually did it.<br>
Or is this the normal state of any deep learning model?</p>

username: shirui-japina
cooked: <p>Firstly, I verified the predicted output value of the model.<br>
1% of my data set (11 samples) as verification data.</p>
<p><strong>The first time predicted output:</strong><br>
<code>tensor([[ 2.3544, -1.6035], [-2.8536, 2.6495], [-1.4212, 0.3854], [-0.3363, -0.2809], [ 0.3346, -0.5889], [-1.1946, 2.2831], [-0.2375, -0.7653], [-0.1411, 0.0929], [-1.5674, 2.0254], [-0.0735, 0.0934], [ 1.3332, -1.3259]], device='cuda:0', grad_fn=&lt;SqueezeBackward0&gt;)</code></p>
<p><strong>The second time predicted output:</strong><br>
<code>tensor([[ 1.6338, -2.4857], [-2.7193, 2.8088], [-0.8641, 0.8308], [-0.6862, -0.8741], [-0.3699, -0.3573], [-1.7963, 1.0194], [ 0.2741, -0.1906], [-0.6578, -0.2878], [-1.3425, 2.3365], [ 0.1620, 0.3339], [ 0.8572, -1.2491]], device='cuda:0', grad_fn=&lt;SqueezeBackward0&gt;)</code></p>
<p><strong>The third time predicted output:</strong><br>
<code>tensor([[ 1.7435, -1.8670], [-2.4884, 2.8394], [-1.0321, 1.0768], [-0.5901, -0.0761], [ 0.0477, -0.6905], [-1.3724, 0.0560], [ 0.1470, -0.4054], [ 0.0067, 0.2553], [-1.7054, 1.6145], [ 0.3581, -0.2970], [ 1.3902, -1.1217]], device='cuda:0', grad_fn=&lt;SqueezeBackward0&gt;)</code></p>
<p>There are even minor changes! !<img src="https://discuss.pytorch.org/images/emoji/apple/cold_face.png?v=9" title=":cold_face:" class="emoji" alt=":cold_face:"><img src="https://discuss.pytorch.org/images/emoji/apple/cold_face.png?v=9" title=":cold_face:" class="emoji" alt=":cold_face:"><img src="https://discuss.pytorch.org/images/emoji/apple/scream.png?v=9" title=":scream:" class="emoji" alt=":scream:"><img src="https://discuss.pytorch.org/images/emoji/apple/scream.png?v=9" title=":scream:" class="emoji" alt=":scream:"><br>
How could this be?<img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=9" title=":flushed:" class="emoji" alt=":flushed:"><img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=9" title=":flushed:" class="emoji" alt=":flushed:"><img src="https://discuss.pytorch.org/images/emoji/apple/fearful.png?v=9" title=":fearful:" class="emoji" alt=":fearful:"><img src="https://discuss.pytorch.org/images/emoji/apple/fearful.png?v=9" title=":fearful:" class="emoji" alt=":fearful:"></p>

username: mailcorahul
cooked: <p>Are you saying that given a fixed data and a fixed model, you get different outputs on every different forward pass?<br>
To answer the dropout part, yes <code>model.eval()</code> is necessary to turn off dropout during inference.</p>
<p>Can you see if the model is loaded in the first place and that there are no random initialization of params during every forward pass?</p>

username: shirui-japina
cooked: <p>Secondly, I tried to <em><strong>remove</strong></em> the <code>dropout layer</code>.<br>
I mean, now the model is like:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def forward(self, input_image):
        out = self.conv_1(input_image) # 20 * 44 * 44
        out = self.pooling_1(out) # 20 * 22 * 22

        out = self.conv_2(out) # 50 * 16 * 16
        out = self.pooling_2(out) # 50 * 8 * 8

        out = self.conv_3(out) # 500 * 2 * 2
        out = self.pooling_3(out) # 500 * 1 * 1

        # out = F.dropout(out) # removed
        out = F.relu(out)
        out = self.conv_4(out) # 2 * 1 * 1
</code></pre>
<p>Then the phenomenon of fluctuating <em><strong>has completely disappeared</strong></em>:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/eeae9305c147eea9d312b39f48b5714e1dd85f49.png" data-download-href="https://discuss.pytorch.org/uploads/default/eeae9305c147eea9d312b39f48b5714e1dd85f49" title="validation_acc_no_dropout"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/eeae9305c147eea9d312b39f48b5714e1dd85f49_2_517x332.png" alt="validation_acc_no_dropout" data-base62-sha1="y3tF1y8Al3gzHlmgHEQn52aLoSJ" width="517" height="332" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/eeae9305c147eea9d312b39f48b5714e1dd85f49_2_517x332.png, https://discuss.pytorch.org/uploads/default/original/3X/e/e/eeae9305c147eea9d312b39f48b5714e1dd85f49.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/e/eeae9305c147eea9d312b39f48b5714e1dd85f49.png 2x" data-dominant-color="FAFBFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">validation_acc_no_dropout</span><span class="informations">700×450 12 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/6/46c71dbf36607fdd1d18815585eb24e3f224ed44.png" data-download-href="https://discuss.pytorch.org/uploads/default/46c71dbf36607fdd1d18815585eb24e3f224ed44" title="validation_loss_no_dropout"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/46c71dbf36607fdd1d18815585eb24e3f224ed44_2_517x332.png" alt="validation_loss_no_dropout" data-base62-sha1="a681HoZd6yNIlZHCpHgOKdCGjVW" width="517" height="332" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/46c71dbf36607fdd1d18815585eb24e3f224ed44_2_517x332.png, https://discuss.pytorch.org/uploads/default/original/3X/4/6/46c71dbf36607fdd1d18815585eb24e3f224ed44.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/6/46c71dbf36607fdd1d18815585eb24e3f224ed44.png 2x" data-dominant-color="FAFBFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">validation_loss_no_dropout</span><span class="informations">700×450 9.99 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>But I don’t know the essential reason, because I <em><strong>have set</strong></em> <code>model.eval()</code><img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji only-emoji" alt=":thinking:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji only-emoji" alt=":thinking:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji only-emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>

username: shirui-japina
cooked: <p>Thanks for your reply.</p>
<aside class="quote no-group" data-username="mailcorahul" data-post="3" data-topic="59032">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mailcorahul/48/15919_2.png" class="avatar"> mailcorahul:</div>
<blockquote>
<p>Are you saying that given a fixed data and a fixed model, you get different outputs on every different forward pass?</p>
</blockquote>
</aside>
<p>Yes, it’s incredible, but the output of the model does have small fluctuations.</p>
<aside class="quote no-group" data-username="mailcorahul" data-post="3" data-topic="59032">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mailcorahul/48/15919_2.png" class="avatar"> mailcorahul:</div>
<blockquote>
<p>To answer the dropout part, yes  <code>model.eval()</code>  is necessary to turn off dropout during inference.</p>
</blockquote>
</aside>
<p>Yes, but it doesn’t seem to work.</p>
<aside class="quote no-group" data-username="mailcorahul" data-post="3" data-topic="59032">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mailcorahul/48/15919_2.png" class="avatar"> mailcorahul:</div>
<blockquote>
<p>Can you see if the model is loaded in the first place and that there are no random initialization of params during every forward pass?</p>
</blockquote>
</aside>
<ul>
<li>I’m sorry, I don’t know what means that <em><strong>the model is loaded in the first place</strong></em>.<br>
I save and load the model like:</li>
</ul>
<pre><code class="lang-auto"># save
path_save_model = os.path.join('model', 'weight_{epoch}.pth'.format(epoch=(epoch + 1)))
torch.save(model.state_dict(), path_save_model)

# load
model = Model()
model.to(device=args.device)
model.load_state_dict(torch.load(args.weight_path))
criterion_reduction_none = nn.CrossEntropyLoss(reduction='none')
model.eval()
</code></pre>
<ul>
<li>The model is just a simple CNN, even there is no <code>fully connection layer</code>.<br>
I’m sure there is <strong>NO</strong> random initialization of params during every forward pass.</li>
</ul>

username: markl
cooked: <p>I don’t know what the issue would be, but I wanted to point out you don’t have to use real data for problems like this. You can do something like</p>
<pre><code class="lang-auto">zeros = torch.zeros((your_input_shape))
print(model.forward(zeros).sum())
print(model.forward(zeros).sum())
print(model.forward(zeros).sum())
</code></pre>
<p>If there is nondeterminacy, it should show up.</p>
<p>You could probably even go into your model’s <code>forward</code> definition, and do <code>print(out.sum())</code> at multiple points, to see where the nondeterminacy is showing up. But I guess you already found out it has something to do with the dropout.</p>

username: shirui-japina
cooked: <p>Thanks for your reply.</p>
<pre><code class="lang-auto">zeros = torch.zeros((your_input_shape))
print(model.forward(zeros).sum())
print(model.forward(zeros).sum())
print(model.forward(zeros).sum())
</code></pre>
<p>I got a new PyTorch skill.<br>
Thank you so much.<img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"><img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"></p>

username: shirui-japina
cooked: <p>Maybe I got answer  here.<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://forums.fast.ai/images/default-favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://forums.fast.ai/t/fluctuating-performance-of-pre-trained-model-frozen-parameter/57052/2?u=shirui" target="_blank" title="07:00AM - 24 October 2019" rel="nofollow noopener">Deep Learning Course Forums – 24 Oct 19</a>
  </header>
  <article class="onebox-body">
    <img src="https://forums.fast.ai/letter_avatar_proxy/v3/letter/t/c37758/200.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://forums.fast.ai/t/fluctuating-performance-of-pre-trained-model-frozen-parameter/57052/2?u=shirui" target="_blank" rel="nofollow noopener">Fluctuating performance of pre-trained model (frozen parameter)</a></h3>

<p>Yeah, that looks normal. Note the scale, it’s only fluctuating in a pretty small range (only the last couple of epochs are in that range, so it’s preserved the final accuracy).   You’re missing the fact that f(x) may not be a deterministic function....</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>