Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time

Came: <pre><code class="lang-auto">for epoch in range(num_epochs):
    # For each batch in the dataloader
    for i, data in enumerate(dataloader, 0):

        ############################
        # (1) Update D network: maximize log(D(x)) + log(1 - D(G(z)))
        ###########################
        ## Train with all-real batch
        netD.zero_grad()
        # Format batch
        real_cpu = data[0].to(device)
        b_size = real_cpu.size(0)
        label = torch.full((b_size,), real_label, device=device)
        # Forward pass real batch through D
        output = netD(real_cpu).view(-1)
        # Calculate loss on all-real batch
        errD_real = criterion(output, label)
        # Calculate gradients for D in backward pass
        errD_real.backward()
        D_x = output.mean().item()

        ## Train with all-fake batch
        # Generate batch of latent vectors
        noise = torch.randn(b_size, nz, 1, 1, device=device)
        # Generate fake image batch with G
        fake = netG(noise)
        label.fill_(fake_label)
        # Classify all fake batch with D
        output = netD(fake.detach()).view(-1)
        # Calculate D's loss on the all-fake batch
        errD_fake = criterion(output, label)
        # Calculate the gradients for this batch
        errD_fake.backward()
        D_G_z1 = output.mean().item()
        # Add the gradients from the all-real and all-fake batches
        errD = errD_real + errD_fake
        # Update D
        optimizerD.step()

        ############################
        # (2) Update G network: maximize log(D(G(z)))
        ###########################
        netG.zero_grad()
        label.fill_(real_label)  # fake labels are real for generator cost
        # Since we just updated D, perform another forward pass of all-fake batch through D
        output = netD(fake).view(-1)
        # Calculate G's loss based on this output
        errG = criterion(output, label)
        # Calculate gradients for G
        errG.backward()
        D_G_z2 = output.mean().item()
        # Update G
        optimizerG.step()
</code></pre>
<p>This is the code from the tutorial DCGAN<br>
<a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html</a></p>
<p>I can’t  understand why we need detach() here.<br>
And once I remove detach, it would arise the error.</p>
<pre><code class="lang-auto"># Classify all fake batch with D
        output = netD(fake.detach()).view(-1)
</code></pre>

MariosOreo: <p>Hi,</p>
<p>Calling <code>.backward()</code> will free the computational graph for saving memory.</p>
<p><code>fake</code> is used to compute the loss of netD and also used in updating netG, after performing <code>.backward()</code> on loss of netD, <code>fake</code> will be freed without <code>.detach()</code> (here detach is used to exclude fake from the computational graph so that calling <code>.backward()</code> on loss of netD will not free <code>fake</code>).</p>
<p>Thus, if you remove <code>.detach()</code> (it will free <code>fake</code>) and call <code>.backward()</code> on loss of netG, it will raise the error that needs <code>retain_graph=True</code>.</p>