Model.eval() for train accuracy

username: hyperlight
cooked: <p>If my model includes BatchNorm or Dropout layers, do I need to set my model in evaluation mode before computing the training accuracy?</p>
<pre><code class="lang-auto">def accuracy(batch, model):
    x, y = batch
    y_pred = (model.predict(x)).type(torch.FloatTensor) 
    y = y.unsqueeze(1)
    correct = (y_pred == y).type(torch.FloatTensor) 
    return correct.mean()
    
def train(data, model, optimizer, criterion, epochs):
    
    train_dl, val_dl = data
    
    train_epoch_loss = []
    train_epoch_acc = []
    
    val_epoch_loss = []
    val_epoch_acc = []
    
    for epoch in range(epochs): # start training 
        # start epoch
        train_batch_loss = []
        train_batch_acc = []
        
        val_batch_loss = []
        val_batch_acc = []
        
        for batch in train_dl: 
            # start train batch
            model.train() # enter train mode
            x, y = batch
            y = torch.unsqueeze(y, 1)
            y_pred = model.forward(x)
            loss = criterion(y_pred, y)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            train_batch_loss.append(loss)
            model.eval() # enter evaluation mode
            with torch.no_grad():
                train_batch_acc.append(accuracy(batch, model)) # evaluate mini-batch train accuracy in evaluation
            # train_batch_acc.append(accuracy(batch, model)) # evaluate mini-batch train accuracy in train mode
        train_epoch_loss.append(torch.tensor(train_batch_loss).mean())
        train_epoch_acc.append(torch.tensor(train_batch_acc).mean())
        
        model.eval() # enter evaluation mode
        with torch.no_grad():
            for batch in val_dl:
                # start validation batch
                x, y = batch
                y = torch.unsqueeze(y, 1)
                y_pred = model.forward(x)
                loss = criterion(y_pred, y)
                val_batch_loss.append(loss)
                val_batch_acc.append(accuracy(batch, model))
        val_epoch_loss.append(torch.tensor(val_batch_loss).mean())
        val_epoch_acc.append(torch.tensor(val_batch_acc).mean())
</code></pre>

username: ptrblck
cooked: <p>Usually you would just calculate the training accuracy on-the-fly without setting the model to <code>eval()</code> and recalculate the “real” training accuracy for this epoch. Mostly because it’s just too expensive and the running loss estimate is often good enough.</p>
<p>That being said, there are some issues in your code:</p>
<ul>
<li>you are currently saving the <code>loss</code> in <code>train_batch_loss</code> with out detaching it. This will store the whole computation graph and can eventually yield an out of memory error. If you don’t need to call <code>backward()</code> on the loss anymore, you should rather store it via <code>train_batch_loss.append(loss.item())</code>, which will properly detach the loss.</li>
<li>instead of calling the <code>model.forward</code> method, call the model directly as: <code>y_pred = model(x)</code>. this will make sure to call al registered hooks (if any) and will avoid potentially other issues.</li>
</ul>

username: hyperlight
cooked: <p>Thank you so much <a class="mention" href="/u/ptrblck">@ptrblck</a> for your explanations, I end-up plotting the training losses both ways and the losses calculated in <code>model.train()</code> is indeed follow the losses calculated in <code>model.eval()</code>.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bb624125680eb4993d0437454b1c522b5b2fbf69.png" data-download-href="https://discuss.pytorch.org/uploads/default/bb624125680eb4993d0437454b1c522b5b2fbf69" title="16%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bb624125680eb4993d0437454b1c522b5b2fbf69_2_690x325.png" alt="16%20PM" data-base62-sha1="qJFNmFypkiNb7qC5GaOp5DppIRX" width="690" height="325" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bb624125680eb4993d0437454b1c522b5b2fbf69_2_690x325.png, https://discuss.pytorch.org/uploads/default/original/3X/b/b/bb624125680eb4993d0437454b1c522b5b2fbf69.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/b/bb624125680eb4993d0437454b1c522b5b2fbf69.png 2x" data-dominant-color="F5F6F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">16%20PM</span><span class="informations">757×357 67.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Also, thanks for pointing out the issues in my code!</p>