Error: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time

Rajat_Sharma1: <p>I have a situation where for each mini-batch, I have multiple nested data, for which model need to be trained.</p>
<pre><code class="lang-auto">for idx, batch in enumerate(train_dataloader):
    data = batch.get("data").squeeze(0)
    op = torch.zeros(size) #zero_initializations
    for i in range(data.shape[0]):
        optimizer.zero_grad()
        current_data = data[i, ...]
        start_to_current_data = data[:i+1, ...]
        target =  some_transformation_func(start_to_current_data)
        op = model(current_data, op)
        loss = criterion(op, target)
        loss.backward()
        optimizer.step()
</code></pre>
<p>But when I start training, I get the following error <strong>RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time</strong>. Setting retain_graph=True increase the memory usage and I can not train the model. How can I fix this.</p>

eqy: <p>I’m not sure this is the cause of your issue, but it looks like your model output <code>op</code> isn’t used in the loss calculation at all?</p>

Rajat_Sharma1: <p>It was a typo. I fixed it.</p>

eqy: <p>I think there might be an issue because the same data tensor is used multiple times, and I am wondering if there is a gradient calculation here. Can you see what happens when you set <code>data.requires_grad = False</code> when it is returned by <code>batch.get</code>?</p>
<p>A miscellaneous note: You can try timing to see if there is an actual difference, but I don’t think the line <code>op = torch.zeros(size) #zero_initializations</code> will actually save time based on how allocations are done. If anything it might just make autograd more confusing if the assignment happens with the actual tensor values rather than just the object references.</p>

Rajat_Sharma1: <p>Setting op = torch.zeros(size) <span class="hashtag">#zero_initializations</span> does not have anything to do with the timing. It is the network architecture where output of the model is fed back as input along with other input. For the very first time it is initialized as zeros.</p>

Rajat_Sharma1: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Can you help me out in this case.</p>

ptrblck: <aside class="quote no-group" data-username="Rajat_Sharma1" data-post="5" data-topic="124751">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rajat_sharma1/48/39351_2.png" class="avatar"> Rajat_Sharma1:</div>
<blockquote>
<p>It is the network architecture where output of the model is fed back as input along with other input. For the very first time it is initialized as zeros.</p>
</blockquote>
</aside>
<p>This might be the culprit, as the computation graph might be kept alive using this approach.<br>
Assuming you don’t want to backpropagate through the previous iterations, you could detach the tensor via <code>op.detach_()</code> in each iteration.</p>