If my model has dropout, do I have to alternate between model.eval() and model.train() during training?

Asa-Nisi-Masa: <p>So if a model has a dropout layer (or a batch-norm layer), then doing</p>
<p><code>model(x)</code></p>
<p>will/may yield a different result compared to</p>
<pre><code class="lang-auto">model.eval()
model(x)
</code></pre>
<p>But now for training what we have is an inference step and a training step. So do we actually have to do something like the following?</p>
<pre><code class="lang-auto">...
for epoch in range(epochs):
    model.eval()   # make sure we are in .eval() mode
    ...
    y_pred = model(x)
    ...
    model.train()  # make sure we are in .train() mode
    loss = loss_fun(y_pred, target)
    loss.backward()
    ...
</code></pre>
<p>What’s the correct way to train a network in this case?</p>

Nikronic: <p>Hi,</p>
<aside class="quote no-group" data-username="Asa-Nisi-Masa" data-post="1" data-topic="83007">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/asa-nisi-masa/48/24803_2.png" class="avatar"> Asa-Nisi-Masa:</div>
<blockquote>
<p>will/may yield a different result compared to</p>
<pre><code class="lang-auto">model.eval()
model(x)
</code></pre>
</blockquote>
</aside>
<p>About this question the answer is yes. The reason is that when you set <code>model.eval()</code> PyTorch removes all dropout layers (do not update mean/variance in batch norm). Here is a small snippet to test:</p>
<pre><code class="lang-auto">class Test(nn.Module):
    def __init__(self, pool_size=(4, 4)):
        super(Test, self).__init__()
       
        self.layer = nn.Linear(1, 1, bias=False)
        self.dropout = nn.Dropout()
   
    def forward(self, x):
        out = self.dropout(self.layer(x))
        return out

model.train()
for i in range(10):
    print(model(x))
</code></pre>
<p>If you run this code, few times (about 1/2) you will get 0 in the output. The reason is that I put 1 single neuron (<code>self.layer</code>) and a dropout with probability of 0.5, so, it will zero the output of tensor with probability of 0.5.</p>
<p>About</p>
<aside class="quote no-group" data-username="Asa-Nisi-Masa" data-post="1" data-topic="83007">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/asa-nisi-masa/48/24803_2.png" class="avatar"> Asa-Nisi-Masa:</div>
<blockquote>
<p>But now for training what we have is an inference step and a training step</p>
</blockquote>
</aside>
<p>This is not true in your example. Only inference time during training is using validation set which is still validation not training. If you set model.eval() then get prediction of your models, you are not using any dropout layers or updating any batchnorm so, we can literally remove all of these layers. As you know, in case of dropout, it is a regularization term to control weight updating, so by setting model in eval mode, it will have no effect.</p>
<p>Bests<br>
Nik</p>

Asa-Nisi-Masa: <p>Thank you for the response. So basically just use <code>model.train()</code> before training and <code>model.eval()</code> after initializing a trained model and it will have no side-effects.</p>

Nikronic: <p>Yes, exactly. Let’s say you are using a pretrained model. If you do not set <code>model.eval</code>, then some of values in resulted tensor will be zero which is not desired as we already trained the model and want a stable values not some random zeros where happens differently in every run.</p>
<p>Although remember that model.eval only has effect on dropout and batchnorm as we discussed, but still you need to use <code>torch.no_grad()</code> to loop over network for inference stage, otherwise, even if you have had set model.eval, gradients will be updated.</p>

Asa-Nisi-Masa: <blockquote>
<p>Although remember that model.eval only has effect on dropout and batchnorm as we discussed, but still you need to use  <code>torch.no_grad()</code>  to loop over network for inference stage, otherwise, even if you have had set model.eval, gradients will be updated.</p>
</blockquote>
<p>I see. But when would this be necessary? During training, it is customary to zero out the gradients with something like <code>optimizer.zero_grad()</code> before backpropagation, so that’s not an issue during training.</p>
<p>For pure inference with a pretrained model, do the gradients matter in any way? I guess keeping track of them might introduce a slight overhead.</p>

Nikronic: <p>drop out does not zero gradients, it zeros tensors in feed forward. Zeroing gradients is not good idea as it has been aggregated over all batches so far and may lead to underfitting. <code>optimizer.zero_grad()</code> is used to zero gradients after each forward and backward propagations.</p>
<p>If you do not use <code>torch.no_grad()</code>, gradients in pretrained model will be updated and the result will be overfitted on your test set.</p>

Asa-Nisi-Masa: <blockquote>
<p>If you do not use  <code>torch.no_grad()</code> , gradients in pretrained model will be updated and the result will be overfitted on your test set.</p>
</blockquote>
<p>I’m not sure I completely follow. If I am using a pretrained model and make predictions without <code>torch.no_grad()</code>, then even if the gradients are computed, no weights are actually being updated. There is no overfitting happening because there is no loss defined when doing just the inference, and no gradient descent being done.</p>

Nikronic: <aside class="quote no-group" data-username="Asa-Nisi-Masa" data-post="7" data-topic="83007">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/asa-nisi-masa/48/24803_2.png" class="avatar"> Asa-Nisi-Masa:</div>
<blockquote>
<p>There is no overfitting happening because there is no loss defined when doing just the inference, and no gradient descent being done.</p>
</blockquote>
</aside>
<p>Oh my bad, I considered same code for training as your original post was about swaping between <code>model.eval</code> and <code>model.train</code>. But using <code>torch.no_grad</code> will save a lot of time and memory!</p>

duddal: <p><a class="mention" href="/u/nikronic">@Nikronic</a> Is it important to use <code>model.eval()</code> in the training loop?</p>