Where should I put train and eval

DSX: <p>Hi,<br>
I saw so many methods the training/validation part of a CNN and here is mine actually :</p>
<pre><code class="lang-auto">model.train()
for e in range(epoch):
    train_sum_loss=0.0
    validation_sum_loss=0.0
    for inputs, labels in train_loader:
        inputs = inputs.to(device)
        labels = labels.to(device)
        optimizer.zero_grad()
        outputs = model.forward(inputs)
        batch_loss = loss(outputs.squeeze(),labels)
        batch_loss.backward()
        optimizer.step()
        train_sum_loss += batch_loss.item()

    model.eval()
    for inputs, labels in validation_loader:
        inputs = inputs.to(device)
        labels = labels.to(device)
        outputs = model.forward(inputs)
        batch_loss = loss(outputs.squeeze(),labels)
        validation_sum_loss += batch_loss.item()
</code></pre>
<p>Results seems good, the network converge, no overfitting.<br>
But if I put the model.train() just before the for training loop (like the validation loop), the NN overfit really fastly.</p>
<p>Which one of these methods is the good one ? On which result I can work ?</p>
<p>Thanks for helping.</p>

pchandrasekaran: <pre><code class="lang-auto">model.train()
for inputs, labels in train_loader:
</code></pre>
<p>The <code>model.train()</code> needs to go there. If you put it outside as in your snippet, the model will only be in training mode for the first epoch. All subsequent epochs will be in evaluation mode.</p>

DSX: <p>Ok thanks.</p>
<p>What about this instruction : torch.no_grad() ?</p>
<p>Some people use it, but not everyone. When I read the documention, it seems to me that this command does the same thing as model.eval(), doesn’t it ?</p>

pchandrasekaran: <p>No. They aren’t the same. I’ve copied a reply from another user that explains the difference.</p>
<aside class="quote no-group">
<blockquote>
<p>These two have different goals:</p>
<ul>
<li>
<p><code>model.eval()</code> will notify all your layers that you are in eval mode, that way, batchnorm or dropout layers will work in eval mode instead of training mode.</p>
</li>
<li>
<p><code>torch.no_grad()</code> impacts the autograd engine and deactivates it. It will reduce memory usage and speed up computations but you won’t be able to backprop (which you don’t want in an eval script).</p>
</li>
</ul>
</blockquote>
</aside>
<p>In essence, simply using torch.no_grad() will not impact layers such as dropout or batchnorm, they will still be used during inference.</p>

DSX: <p>Ok thanks for details.</p>
<p>So I can use it together during validation loop.</p>
<p>But I haven’t find a way to do the opposite of no_grad() (to be back in training mode).</p>
<p>Edit:<br>
Maybe I’ve just found it : torch.set_grad_enabled(True)<br>
Is it this instruction ?</p>

pchandrasekaran: <p>Yes. I was just going to post that.</p>

pchandrasekaran: <p><a class="mention" href="/u/dsx">@DSX</a></p>
<p>I apologize. Just double checked the documentation. You do not need to use set_grad_enabled(). torch.no_grad() only sets the requires_grad=False temporarily.</p>
<pre><code class="lang-auto">x = torch.randn(3, requires_grad=True)

print(x.requires_grad)

print((x ** 2).requires_grad)

with torch.no_grad():

    print((x ** 2).requires_grad)

print((x ** 2).requires_grad)
</code></pre>
<pre><code class="lang-auto">True
True
False
True
</code></pre>

DSX: <p>Thanks for your double check.<br>
It will simpler a bit my code <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"></p>