'model.eval()' vs 'with torch.no_grad()'

spnova12: <p>When i test my model, do I have to use model.eval() even though I am using 'with torch.no_grad() ?</p>

albanD: <p>Hi,</p>
<p>These two have different goals:</p>
<ul>
<li>
<code>model.eval()</code> will notify all your layers that you are in eval mode, that way, batchnorm or dropout layers will work in eval mode instead of training mode.</li>
<li>
<code>torch.no_grad()</code> impacts the autograd engine and deactivate it. It will reduce memory usage and speed up computations but you won’t be able to backprop (which you don’t want in an eval script).</li>
</ul>

spnova12: <p>Thank you very much for your quick and clear explanation.</p>

Jk749: <p>Hey, this implies I should definitely do <code>"model.eval"</code> while validating.</p>
<p>And, if memory and speed are not constraints; <code>"torch.no_grad()"</code> can be ignored. Right?</p>

Naman-ntc: <p>Ahh with torch.no_grad() you’ll have much higher speeds and can use larger validation batch sizes so it’s useful if not recommended</p>

albanD: <p>Why do something that takes 5x more memory (the 5 here is for the example, not actual number in practice) and is slow, if you can just add one extra line to avoid it?<br>
<a class="mention" href="/u/naman-ntc">@Naman-ntc</a> using <code>torch.no_grad()</code> is actually the recommended way to perform validation !</p>

Naman-ntc: <p>yeah alright I meant to write “if not compulsory” <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=5" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

Jk749: <p>Thank you for the explanation!</p>

kasperfred: <p>Does <code>torch.no_grad()</code> also disable dropout layers?</p>

michaelklachko: <p>So why is torch.no_grad() is not enabled by default inside model.eval() function? Is there a situation where we want to compute some gradients when in evaluation mode? Even if that’s the case, it seems like no_grad() method should be made an optional argument to eval(), and set to True by default.</p>

albanD: <p><a class="mention" href="/u/kasperfred">@kasperfred</a>  No it does not.</p>
<p><a class="mention" href="/u/michaelklachko">@michaelklachko</a> Some user can have a use case for this. The problem with doing this I guess is that no_grad is a context manager to work with the autograd engine while eval() is changing the state of an nn.Module.</p>

Victor_Tan: <p>Hello, do you know how exactly the eval mode  affect the dropout layer in the test? What are the differences of the dropout behavior between the eval and training mode?</p>

ptrblck: <p>During eval <code>Dropout</code> is deactivated and just passes its input.<br>
During the training the probability <code>p</code> is used to drop activations. Also, the activations are scaled with <code>1./p</code> as otherwise the expected values would differ between training and eval.</p>
<pre><code class="lang-python">drop = nn.Dropout()
x = torch.ones(1, 10)

# Train mode (default after construction)
drop.train()
print(drop(x))

# Eval mode
drop.eval()
print(drop(x))
</code></pre>

Victor_Tan: <p>Thanks a lot. Your answer is the same as what I thought.</p>

BramVanroy: <p>Could someone please confirm whether this means that you handle evaluating and testing similarly? In both cases you set the model to <code>.eval()</code> and use <code>with torch.no_grad()</code>? (A bit more explanation as to why we treat them similarly is also welcome; I am a beginner.)</p>

two_four: <p>In <code>Dropout</code> documentation, it says the probability <code>p</code> is used to drop activations. At the same time, the activations not be dropped are scaled with <code>1/(1-p)</code>, I am not sure why it uses <code>1/(1-p)</code> as a factor to scale the activations, could you give some explanation?</p>

ptrblck: <p>Have a look at <a href="https://discuss.pytorch.org/t/unclear-behaviour-of-dropout/22890/4?u=ptrblck">this post</a> for an example why we are scaling the activations.<br>
Note that the <code>p</code> in my explanation refert to the keep probability not the drop probability.</p>

two_four: <p>Thanks for your explanation, now I am clear about that</p>

yilong: <p>hi, are you sure bn and dropout work in eval model?  I think bn and dropout work in trainning mode, not working in validation and test mode.</p>

albanD: <p>Hi,</p>
<p>There is no such thing as “test mode”.<br>
Only train() and eval().<br>
Both bn and dropout will work in both cases but will have different behaviour as you expect them to have different behaviours during training and evaluation. For example, during evaluation, dropout should be disabled and so is replaced with a no op. Similarly, bn should use saved statistics instead of batch data and so that’s what it’s doing in eval mode.</p>