Error while doing backpropagation in case of batch gradient descent

Jeet: <p>I am trying to implement the gradient descent algorithm in a neural network. The network accumulates the loss for all the samples in the training set and then updates the model once like in a simple gradient descent algorithm. However, I am getting a Runtime Error that says-</p>
<blockquote>
<p>“RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.”</p>
</blockquote>
<pre><code class="lang-auto">for epoch in range(num_epochs):
  totallossg=0
  totallossd=0
  disc.zero_grad()
  gen.zero_grad()

  for iteration in range(4):

     for face in faces:
     
        noise=torch.randn(z_dim, z_dim)
        fake = gen(noise)
        disc_real = disc(face.to(torch.float32))
        disc_fake = disc(fake)
        lossD_real = criterion(disc_real, torch.ones_like(disc_real))
        lossD_fake = criterion(disc_fake, torch.zeros_like(disc_fake))  
        lossD = (lossD_real + lossD_fake) / 2
        totallossd=totallossd+lossD

     totallossd=torch.div(totallossd,5)
     disc.zero_grad()    
     totallossd.backward()  #This line is generating the above said error
     opt_disc.step()
</code></pre>
<p>I have tried doing <em>retain_graph=True</em>. That changes the error to -</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [4, 1]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True)</p>
</blockquote>
<p>Is it something wrong I am trying to do here? Why am I getting the errors when it is a simple gradient descent implementation?</p>

KFrank: <p>Hi Jeet!</p>
<aside class="quote no-group quote-modified" data-username="Jeet" data-post="1" data-topic="158444" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jeet/48/47500_2.png" class="avatar"> Jeet:</div>
<blockquote>
<blockquote>
<p>“RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed).”</p>
</blockquote>
<pre><code class="lang-auto">     disc.zero_grad()    
     totallossd.backward()  #This line is generating the above said error
     opt_disc.step()
</code></pre>
</blockquote>
</aside>
<p>Based on the indentation of the code you posted, <code>totallossd.backward()</code><br>
is <em>inside</em> the <code>for iteration in range(4):</code> loop.</p>
<p>Consider:</p>
<p>In the first iteration of this loop you compute <code>lossD</code> – let’s call this<br>
<code>lossD</code>-1.  You then accumulate <code>lossD</code>-1 into <code>totallossd</code> and call<br>
<code>totallossd.backward()</code>.  This backpropagates through <code>lossD</code>-1’s<br>
computation graph and then frees the graph.  But <code>lossD</code>-1 is still part<br>
of <code>totallossd</code>.</p>
<p>In the second iteration, you calculate a new <code>lossD</code> (Let’s call it <code>lossD</code>-2.)<br>
and create its computation graph.  You then accumulate <code>lossD</code>-2 into<br>
<code>totallossd</code> and call <code>totallossd.backward()</code>.  This backpropagates<br>
through <code>lossD</code>-2’s computation graph, which is fine, but because<br>
<code>totallossd</code> still contains <code>lossD</code>-1, the <code>.backward()</code> call tries to<br>
backpropagate a second time through <code>lossD</code>-1’s computation graph,<br>
which has already been freed.  Hence the error.</p>
<p>If I understand your use case correctly, you want to move all of:</p>
<pre data-code-wrap="python"><code class="lang-python">     totallossd=torch.div(totallossd,5)
     disc.zero_grad()    
     totallossd.backward()  #This line is generating the above said error
     opt_disc.step()
</code></pre>
<p>out of the <code>for iteration in range(4):</code> loop, which you can do by<br>
reducing the indentation of that section of code by one unit.</p>
<p>As an aside, I would think that in <code>totallossd=torch.div(totallossd,5)</code><br>
you would actually want to divide by <code>4</code>, as <code>totallossd</code> has four quantities<br>
summed into it in the  <code>for iteration in range(4):</code> loop, rather than five.</p>
<p>Best.</p>
<p>K. Frank</p>

srishti-git1110: <p>Hello Jeet,<br>
<a class="mention" href="/u/kfrank">@KFrank</a> has excellently answered and solved your query.</p>
<p>I’d like to add some more details that’ll help you understand a bit more about pytorch’s autograd engine-<br>
For the first iteration of <code>for iteration in range(4):</code> , <code>totallossd</code> is basically composed of one <code>lossD</code> value that K. Frank refers to as <code>lossD</code>-1.</p>
<p>Now, in this first iteration, right after <code>totallossd.backward()</code> backpropagates gradients, the references to saved tensors (essentially, references to tensor variables needed to calculate the gradient of <code>totallossd</code>) are freed, but <em>the graph still remains in the memory</em>.</p>
<p>Now, in the second iteration, <code>lossD</code>-2 is formed which eventually becomes a part of <code>totallossd</code>; and now when <code>totallossd.backward()</code> is called, an error is thrown as autograd also tries to backprop through <code>lossD</code>-1’s <em>graph that is hanging in the memory</em> but has no references to saved tensors = tensors needed to compute the gradient have already been freed.</p>
<p>When you did <code>retain_graph=True</code>, no saved tensors were freed. Now, note that in the second iteration those same tensors viz <code>disc_real, disc_fake, lossD_real, lossD</code> etc. are being used to eventually create <code>totallossd</code> - they tensor you are calling <code>backward</code> on.</p>
<p>Since references to these very tensors weren’t freed in the first iteration, autograd will be able to monitor the “change” in the values of these tensors in the 2nd iteration as it modifies the values in place.<br>
This monitoring is done as a part of a tensor attribute called <code>version</code> which keeps getting updated as the values keep changing.</p>
<p>Hence, as the second backward call is made and autograd tries to backpropagate gradients through the graph, these tensors are at <code>version 2</code> which is different from the <code>version</code> (1) they had when the forward graph was constructed.</p>
<p>Hope this helps,<br>
S</p>