How to clear autograd information between iterations when using eval mode?

username: tueboesen
cooked: <p>My code requires the autograd to be on even during eval mode, since I need the gradient information of my output with regards to one of my input variables. However this accumulates gradient information somehow when using eval mode. (In training mode this is not a problem, since it also calls loss.backward() and optimizer.step(), which frees up the memory along with optimizer.zero_grad())</p>
<p>How do I free the memory in a similar fashion but without actually running loss.backward()</p>
<p>Here is a simplified version of my train/eval mode:</p>
<pre><code class="lang-auto">def use_model(model,dataloader,train,optimizer,device,batch_size=1):
    aloss = 0.0
    if train:
        model.train()
    else:
        model.eval()
    for i, (Ri, Fi, Ei, zi) in enumerate(dataloader):
        Ri.requires_grad_(True)
        xn, xe, G = getIterData_MD17(Ri.squeeze(), device=device)

        if train:
            optimizer.zero_grad()
        xnOut, xeOut = model(xn, xe, G)

        E_pred = torch.sum(xnOut)
        F_pred = -grad(E_pred, Ri, create_graph=True)[0].requires_grad_(True)
        loss = F.mse_loss(F_pred, Fi)
        if train:
            loss.backward()
            optimizer.step()
        aloss += loss.detach()
    return aloss
</code></pre>

username: ptrblck
cooked: <p>I don’t fully understand the use case.<br>
Based on this statement:</p>
<aside class="quote no-group" data-username="tueboesen" data-post="1" data-topic="118205">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tueboesen/48/17097_2.png" class="avatar"> tueboesen:</div>
<blockquote>
<p>I need the gradient information of my output with regards to one of my input variables</p>
</blockquote>
</aside>
<p>You are calculating the gradients during validation, while</p>
<aside class="quote no-group" data-username="tueboesen" data-post="1" data-topic="118205">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tueboesen/48/17097_2.png" class="avatar"> tueboesen:</div>
<blockquote>
<p>How do I free the memory in a similar fashion but without actually running loss.backward()</p>
</blockquote>
</aside>
<p>claims that you are never using the <code>backward</code> operation. How are these gradients calculated?</p>
<p>Note that you could also freeze all parameters, which shouldn’t get gradients and which should thus also save memory.</p>

username: tueboesen
cooked: <p>You are right, I am using the backward operation implicitly when I use:</p>
<pre><code class="lang-auto">F_pred = -grad(E_pred, Ri, create_graph=True)[0].requires_grad_(True)
</code></pre>
<p>I guess my question is how to clear the gradient information after I have invoked this call?<br>
(So basically I need the autograd to compute a derivative through my network at each iteration in the validation/testing step, but I do not want to actually update/change my network parameters, and after each iteration I wish to ensure that I have discarded all the gradient information that the network might have built up.)</p>
<p>Does that make sense?</p>

username: ptrblck
cooked: <p><code>torch.autograd.grad</code> should remove all intermediate tensors after its operation, if <code>retain_graph</code> is not set to <code>True</code> (in the same way <code>tensor.backward()</code> operates).</p>

username: tueboesen
cooked: <p>It really was that simple, thank you.</p>
<p>I just needed to remove the requires_grad_ from F_pred when running in validation mode.</p>