DCGAN - Trying to backward through the graph a second time

username: dev1
cooked: <p>I know there are similar posts. I nearly checked them all. Tried <code>.detach()</code>. and <code>retain_graph=True</code> options. However I couldn’t solve it for my training loop of DCGAN. This is a fairly complex training loop for me so I wouold be glad for any help. I also checked the PyTorch tutorial and created this version.</p>
<pre><code class="lang-auto">criterion = nn.BCELoss()

discriminator_optimizer = optim.Adam(netD.parameters(), lr=lr, betas=(beta1, 0.999))
generator_optimizer = optim.Adam(netG.parameters(), lr=lr, betas=(beta1, 0.999))

img_list = []
G_losses = []
D_losses = []
iters = 0

start = timeit.default_timer()
for epoch in tqdm(range(EPOCHS), position=0, leave=True):
    generator.train()
    discriminator.train()
    for idx, data in enumerate(tqdm(dataloader, position=0, leave=True)):
        img_data = data[0].to(device) # size [1024, 3, 64, 64]
        dummy_labels = data[1] # size [1024]
        
        real_labels = torch.full((dummy_labels.size()), 1., dtype=torch.float).to(device) # size [1024]
        fake_labels = torch.full((dummy_labels.size()), 0., dtype=torch.float).to(device)
        noise = torch.randn(dummy_labels.size()[0], INPUT_VECTOR_DIM, 1, 1).to(device) # size [1024, 100, 1, 1]
        
        discriminator.zero_grad() 
        
        discriminator_real_out = discriminator(img_data).view(-1).to(device) # size [1024] .view(-1) to rid unnecessary dimensions
        discriminator_real_loss = criterion(discriminator_real_out, real_labels)
        discriminator_real_loss.backward()
        
        generator_fake_out = generator(noise)  # size [1024, 3, 64, 64]
        discriminator_fake_out = discriminator(generator_fake_out.detach()).view(-1).to(device)
        discriminator_fake_loss = criterion(discriminator_fake_out, fake_labels)
        discriminator_fake_loss.backward()
        discriminator_loss = discriminator_real_loss.item() + discriminator_fake_loss.item()
        discriminator_optimizer.step()

        generator_loss = criterion(discriminator_fake_out, real_labels)
        generator_loss.backward()
        generator_optimizer.step()
        generator.zero_grad()
</code></pre>
<p>Problem is from this line <code>generator_loss.backward()</code>. Also when I remove the line <code>discriminator_optimizer.step()</code> the problem disappears.</p>
<p>Full error: <code>RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.</code></p>

username: KFrank
cooked: <p>Hi Uygar!</p>
<aside class="quote no-group" data-username="dev1" data-post="1" data-topic="191651" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/cdc98d/48.png" class="avatar"> dev1:</div>
<blockquote>
<p>Tried <code>.detach()</code>. and <code>retain_graph=True</code> options.</p>
</blockquote>
</aside>
<p>As a general principle, don’t just try things.  Analyze the actual cause<br>
of <em>your</em> specific issue and use the “option” that actually addresses that<br>
cause.</p>
<p>Please look at the comments that I’ve added in line to your quoted code:</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">criterion = nn.BCELoss()

discriminator_optimizer = optim.Adam(netD.parameters(), lr=lr, betas=(beta1, 0.999))
generator_optimizer = optim.Adam(netG.parameters(), lr=lr, betas=(beta1, 0.999))

img_list = []
G_losses = []
D_losses = []
iters = 0

start = timeit.default_timer()
for epoch in tqdm(range(EPOCHS), position=0, leave=True):
    generator.train()
    discriminator.train()
    for idx, data in enumerate(tqdm(dataloader, position=0, leave=True)):
        img_data = data[0].to(device) # size [1024, 3, 64, 64]
        dummy_labels = data[1] # size [1024]
        
        real_labels = torch.full((dummy_labels.size()), 1., dtype=torch.float).to(device) # size [1024]
        fake_labels = torch.full((dummy_labels.size()), 0., dtype=torch.float).to(device)
        noise = torch.randn(dummy_labels.size()[0], INPUT_VECTOR_DIM, 1, 1).to(device) # size [1024, 100, 1, 1]
        
        discriminator.zero_grad() 
        
        discriminator_real_out = discriminator(img_data).view(-1).to(device) # size [1024] .view(-1) to rid unnecessary dimensions
        discriminator_real_loss = criterion(discriminator_real_out, real_labels)
        discriminator_real_loss.backward()
        
        generator_fake_out = generator(noise)  # size [1024, 3, 64, 64]
        discriminator_fake_out = discriminator(generator_fake_out.detach()).view(-1).to(device)
        ##### at this point, discriminator is included in the computation graph for discriminator_fake_out
        ##### note that discriminator_fake_out's computation graph does not include generator (because you called .detach() on generator_fake_out)
        discriminator_fake_loss = criterion(discriminator_fake_out, fake_labels)
        discriminator_fake_loss.backward()
        ##### this frees (most of) the computation graph that runs through discriminator
        ##### note, if you call .backward (retain_graph = True), you won't free the graph
        discriminator_loss = discriminator_real_loss.item() + discriminator_fake_loss.item()
        discriminator_optimizer.step()

        generator_loss = criterion(discriminator_fake_out, real_labels)
        ##### the computation graph for generator_loss contains discriminator_fake_out and therefore contains the mostly-freed graph for discriminator
        generator_loss.backward()
        ##### you now try to backpropagate through discriminator's mostly-freed graph, causing the "backward a second time" error
        generator_optimizer.step()
        ##### leaving aside the "backward a second time"" error, this won't do anything because you called .detach() on generator_fake_out, preventing any of the .backward() calls from populating generator's parameters with gradients
        generator.zero_grad()
</code></pre>
</blockquote>
</aside>
<p>The key issue is that you are using <code>discriminator_fake_out</code> to<br>
optimize both <code>discriminator</code> and <code>generator</code> (leading in this particular<br>
attempt to the “backward a second time” error).  If you try using<br>
<code>retain_graph = True</code> without doing things just so, you are<br>
likely to get an inplace-modification error.</p>
<p>Probably the simplest way to address this issue with training GANs is<br>
to rebuild the <code>discriminator</code> computation graph by calling – at added<br>
computational cost – <code>discriminator_fake_out = discriminator (...)</code><br>
twice, once for the <code>discriminator_fake_loss.backward()</code><br>
backpropagation and then again for the <code>generator_loss.backward()</code><br>
backpropagation.</p>
<aside class="quote no-group">
<blockquote>
<p>Problem is from this line <code>generator_loss.backward()</code>.</p>
</blockquote>
</aside>
<p>Yes, as explained in the in-line comments I added to your code.</p>
<aside class="quote no-group">
<blockquote>
<p>Also when I remove the line <code>discriminator_optimizer.step()</code> the problem disappears.</p>
</blockquote>
</aside>
<p>I don’t believe this.  For the version of the code you posted, you will get the<br>
“backward a second time” error after calling <code>generator_loss.backward()</code>,<br>
regardless of whether <code>discriminator_optimizer.step()</code> was called or not.</p>
<p>Perhaps in a different version of your code you had an inplace-modification<br>
error that removing <code>discriminator_optimizer.step()</code> appeared to fix.</p>
<p>If you have (or do) come across inplace-modification errors, it will be<br>
because <code>discriminator_optimizer.step()</code> is modifying <code>discriminator</code>’s<br>
parameters inplace.</p>
<p>A discussion about fixing inplace-modification errors that includes a<br>
toy-GAN example can be found in this post:</p>
<aside class="quote quote-modified" data-post="7" data-topic="171826">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-torch-floattensor-64-1-which-is-output-0-of-asstridedbackward0-is-at-version-3-expected-version-2-instead-hint-the-backtrace-further-a/171826/7">"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further a</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi Fahmyadan and Sangyoon! 

Here are some suggestions about how to track down (and maybe fix) 
inplace-modification errors.  Note that an inplace modification in the forward 
pass is not necessarily* an error – it depends on whether and how the tensor 
that was modified is used in the backward pass.  Note that inplace operations 
can be useful for saving memory – if you replace an innocent inplace operation 
with an out-of-place equivalent, your training will use more memory (and, to a 
minor e…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>

username: dev1
cooked: <p>Hi Frank. Thank you for the comprehensive answer.</p>
<p>Great catch! the problem is because of the <code>discriminator_fake_out</code> since I use it at both <code>discriminator</code> and <code>generator</code>.</p>
<p>Calculating the computation graph for <code>discriminator_fake_out</code> for a second time solves the problem. I checked the PyTorch tutorial again and that’s also how they implemented it.</p>
<p>Now the first question comes into mind is can we solve it without calculating the <code>discriminator_fake_out</code> for a second time?</p>
<p>I did<br>
<code>discriminator_fake_out = torch.clone(discriminator_fake_out).detach().requires_grad_()</code><br>
and the dummy version run without any problems. I’ll look at the actual results to see if it works.</p>
<p>Do you think this is also a correct approach?</p>

username: J_Johnson
cooked: <aside class="quote no-group" data-username="dev1" data-post="3" data-topic="191651">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/cdc98d/48.png" class="avatar"> dev1:</div>
<blockquote>
<p>Now the first question comes into mind is can we solve it without calculating the <code>discriminator_fake_out</code> for a second time?</p>
</blockquote>
</aside>
<p>Why would you want to, though? The results of the discriminator are different(i.e. better) after that first optimizerD.step(), and you want those better results to optimize the generator.</p>

username: dev1
cooked: <p>You’re right. However we’re feeding the same input <code>generator_fake_out</code>. So I couldn’t be sure if it worths it or becomes just an extra operation. What do you think?</p>

username: KFrank
cooked: <p>Hi Uygar!</p>
<aside class="quote no-group" data-username="dev1" data-post="3" data-topic="191651" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/cdc98d/48.png" class="avatar"> dev1:</div>
<blockquote>
<p>can we solve it without calculating the <code>discriminator_fake_out</code> for a second time?</p>
</blockquote>
</aside>
<p>Yes.  One approach to not recalculating <code>discriminator_fake_out</code> is<br>
illustrated in the toy-GAN example in the post I linked to above.</p>
<p>The basic idea is to modify <code>discriminator</code> so that its forward pass uses<br>
<code>clone()</code>s (where necessary) of its parameters because they get modified<br>
inplace when you call <code>discriminator_optimizer.step()</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>I did <code>discriminator_fake_out = torch.clone(discriminator_fake_out).detach().requires_grad_()</code><br>
…<br>
Do you think this is also a correct approach?</p>
</blockquote>
</aside>
<p>No.  This will not backpropagate through <code>discriminator</code> (and will therefore<br>
not backpropagate through <code>generator</code>).  After calling <code>.detach()</code>, the new<br>
tensor referred to by <code>discriminator_fake_out</code> is not part of any computation<br>
graph, so any backpropagation stops at that point.</p>
<p>Calling <code>.requires_grad_()</code> does not reconnect <code>discriminator_fake_out</code><br>
to the computation graph.  (As a general rule resetting <code>requires_grad</code> to<br>
<code>True</code> to “fix” an issue is an error.  Doing so can suppress reporting of some<br>
error messages, but this doesn’t mean your code it correct.)</p>
<p>Best.</p>
<p>K. Frank</p>