Torch.no_grad() in Validation Set

mrityu: <p>Below is a code snippet that I am implementing:-</p>
<pre><code class="lang-auto">def loss_fn(Y_prob, Y_true):
    criterion = nn.BCELoss()
    loss = criterion(Y_prob, Y_true)
    return loss
</code></pre>
<pre><code class="lang-auto">def loss_for_batch(model, X, Y_true, optimizer=None):
    Y_prob = model(X)
    loss = loss_fn(Y_prob, Y_true)
    if optimizer is not None:
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
    return (loss.item())*X.shape[0])
</code></pre>
<pre><code class="lang-auto">def train():
    .
    .
    .
    model.train()
    for X_train, Y_train in train_dl:
        train_loss_batch = loss_for_batch(
                    model, X_train, Y_train, optimizer)
         .
         .
         .

     model.eval()
     for X_val, Y_valin train_dl:
     val_loss_batch = loss_for_batch(
                    model, X_train, Y_train, optimizer)     
</code></pre>
<p>My question is:-<br>
Since I haven’t used <code>torch.no_grad()</code> for backward calculation and I am aware of the fact that I should do it considering the computational constraints. But, even if I am not doing it, will it affect my model performance?<br>
My doubt is when I am iterating through the <code>val_dl</code>, the computation graph which is being made while iterating through the <code>val_dl</code> will have the <code>loss</code> variable too, which is having <code>requires_grad</code> set to <code>True</code> because I haven’t used <code>torch.no_grad()</code>, will this graph affect the “next” graph and gradients which will be created on the next iteration through <code>train_dl</code> and backpropagated through <code>loss.backward</code>?<br>
Is the <code>loss</code> tensor being created is the “same” for all the iterations of <code>dataloaders</code> irrespective of <code>train_dl</code> or <code>val_dl</code>?</p>

tom: <p>I don’t know if this is from posting here, but you would want the <code>model.eval()</code> to happen outside the training loop, right? Also, you would want the validation to happen on the validation loop, probably.</p>
<p><code>torch.no_grad()</code> saves memory and, sometimes, time, but the recorded autograd graphs should be deallocated when the validation leaves <code>loss_for_batch</code> because all tensors (Y_prob and loss in particular) go out of scope regardless of whether you have done the backward, so at the end of each iteration you will be in the same situation with or without <code>no_grad</code> in the <code>loss_for_batch</code> function. There used to be bugs around this a longish time ago, but I think the above would work.</p>
<p>That said, I would probably add the no_grad context (or a <code>torch.set_grad_enabled(torch.is_grad_enabled() and optimizer is not None)</code> context if you want to handle with gradient and without in one bit of code).</p>
<p>Best regards</p>
<p>Thomas</p>

mrityu: <p>The <code>model.eval()</code> should have been outside the training loop. I wrote that erroneously.</p>
<p>Also, thanks for the clarification.</p>
<p>I am adding the <code>torch.no_grad()</code> in the training loop just after the <code>model.eval()</code> part. I guess that won’t raise an alarm.</p>