"RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time" while using custom loss function

username: paduvi
cooked: <p>I keep running into this error:</p>
<blockquote>
<p>RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.</p>
</blockquote>
<p>I’m searching in forum, but still can’t know what I have wrong in my custom loss function.<br>
I’m using nn.GRU, here is my Loss function:</p>
<pre><code>def _loss(outputs, session, items):  # `items` is a dict() contains embedding of all items
    def f(output, target):
        pos = torch.from_numpy(np.array([items[target["click"]]])).float()
        neg = torch.from_numpy(np.array([items[idx] for idx in target["suggest_list"] if idx != target["click"]])).float()
        if USE_CUDA:
            pos, neg = pos.cuda(), neg.cuda()
        pos, neg = Variable(pos), Variable(neg)

        pos = F.cosine_similarity(output, pos)
        if neg.size()[0] == 0:
            return torch.mean(F.logsigmoid(pos))
        neg = F.cosine_similarity(output.expand_as(neg), neg)

        return torch.mean(F.logsigmoid(pos - neg))

    loss = map(f, outputs, session)
return -torch.mean(torch.cat(loss))
</code></pre>
<p>Training code:</p>
<pre><code>        # zero the parameter gradients
        model.zero_grad()

        # forward + backward + optimize
        outputs, hidden = model(inputs, hidden)
        loss = _loss(outputs, session, items)
        acc_loss += loss.data[0]

        loss.backward()
        # Add parameters' gradients to their values, multiplied by learning rate
        for p in model.parameters():
            p.data.add_(-learning_rate, p.grad.data)
</code></pre>

username: jpeg729
cooked: <p>I don’t think error is in your loss function. I think any loss function would cause this error.</p>
<p>Am I right in saying that your training loop doesn’t detach or repackage the hidden state in between batches? If so, then <code>loss.backward()</code> is trying to back-propagate all the way through to the start of time, which works for the first batch but not for the second because the graph for the first batch has been discarded.</p>
<p>If I am right then there are two possible solutions.</p>
<ol>
<li>
<p>detach/repackage the hidden state in between batches. There are (at least) three ways to do this.</p>
<ol>
<li><code>hidden.detach_()</code></li>
<li><code>hidden = hidden.detach()</code></li>
<li><code>hidden = Variable(hidden.data, requires_grad=True)</code></li>
</ol>
</li>
<li>
<p>replace <code>loss.backward()</code> with <code>loss.backward(retain_graph=True)</code> but know that each successive batch will take more time than the previous one because it will have to back-propagate all the way through to the start of the first batch.</p>
</li>
</ol>

username: paduvi
cooked: <p>Thanks. I have tried 2nd solution before, but it ran really slow. Your 1st solution helped me to fix it.</p>

username: thongnguyen
cooked: <p><a class="mention" href="/u/jpeg729">@jpeg729</a>, can you elaborate a bit more:</p>
<p>Mathematically speaking, what would be the difference between solution 1 and solution 2? Or they are mathematically equivalent but not as same computationally efficient ?</p>
<p>Thanks</p>

username: Yongjie_Shi
cooked: <p>The same question~Thank you~</p>

username: pp18
cooked: <p><a class="mention" href="/u/jpeg729">@jpeg729</a> Thank you for your answer. I am adding this comment which can further help and elaborate on the statement <strong>"Am I right in saying that your training loop doesn’t detach or <code>repackage</code> the hidden state in between batches?"</strong></p>
<p>Detach is very well explained but <strong><code>repackage</code></strong> is what caught my eyes and wanted an explanation. I guess I figured that out and which in my understanding is as follows:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.tensor([1,2,3.], requires_grad = True)
&gt;&gt;&gt; out = a.sigmoid()
&gt;&gt;&gt; out.sum().backward()
&gt;&gt;&gt; a.grad # This will output following
tensor([0.1966, 0.1050, 0.0452])
</code></pre>
<p>But when we run the following code again :</p>
<pre><code class="lang-auto">&gt;&gt;&gt; out.sum().backward() # when we run this again then following runtime error is in output:
[RuntimeError]: Trying to backward through the graph a second time, 
but the buffers have already been freed. 
Specify retain_graph=True when calling backward the first time.
</code></pre>
<p>This <strong>error</strong> because the graph has been cleared as indicated and hence can not backpropagate. Thus, if we want to backpropagate again then we must use &lt;<strong>retrain_graph=True</strong>&gt; or create the graph again (which is like calling the <strong>forward</strong>  function of the model to create the graph again) by using the following:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; out = a.sigmoid()
&gt;&gt;&gt; out.sum().backward()
&gt;&gt;&gt; a.grad # This will output following. One extra thing to notice here is 
# that gradient has been added over the last one as we did not perform
# &lt; a.grad.data.zero_()&gt; to reset the gradient thus is accumulated. 
tensor([0.3932, 0.2100, 0.0904])
</code></pre>
<p>I hope this further explains and make thing more clear.</p>
<p>Please comment on this if something is not fine.</p>

username: Amy_Lapi
cooked: <p>Could you elaborate on the line <code>hidden = Variable(hidden.data, requires_grad=True)</code>? I don’t understand what Variable is in the code, but I have the same problem in my own network using a GRU. I think your solution will work, but I don’t understand how to apply that specific line. Thanks!</p>

username: J_Johnson
cooked: <p>Would be great if they included this or a link to this in the GRU/RNN docs. Judging by how many have liked your answer, this is pertinent info.</p>

username: J_Johnson
cooked: <p><code>Variable</code> is a specific Pytorch class, which has been deprecated. I think they just use only Tensor now.<br>
<a href="https://pytorch.org/docs/stable/autograd.html#variable-deprecated" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/autograd.html#variable-deprecated</a></p>
<p>For example:<br>
<code>hidden = torch.Tensor(hidden.data, requires_grad=True)</code></p>

username: yinyee
cooked: <p>Hello, hoping for some help as I have been stuck for a long time.  This is a snippet of the code:</p>
<pre><code class="lang-auto">optimiser.zero_grad()
loss.backward()
for param in agent.parameters(): param.grad.data.clamp_(-1, 1) 
optimiser.step()
for param in agent.parameters(): param = param.detach()
</code></pre>
<p>As you can see, I am already ‘repackaging’ the model parameters.  For context, I am trying to implement DQN, following as closely to the PyTorch tutorial as possible (different environment and use case).</p>
<p>I randomly get one of two errors on each run:</p>
<p>“one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3, 1]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!”</p>
<p>or</p>
<p>“Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling .backward() or autograd.grad() the first time.”</p>
<p>Any suggestions as to what the bug might be?</p>
<p>Also, print(param._version) at the very beginning of the training loop (first line), some of the parameters (weights) are already at version 1, whilst others (biases) are at version 0.  Is that normal?</p>

username: as3eem
cooked: <p>This helped. I was consuming too much memory on GPU w/ retain_graph=True parameter</p>