Retain_graph for GAN discriminator

dragonHyeon: <p>When we try to implement GAN training code, we should do backward for discriminator twice for real image case and fake image case. However we don’t use retain_graph=True for discriminator backward. As I know we should use retain_graph=True to prevent removing the computational graph but my code (also pytorch official dcgan tutorial code) works without retain_graph=True option. Do you know how this works?</p>

ptrblck: <p>Using <code>retain_graph=True</code> will keep the computation graph alive and would allow you to call <code>backward</code> and thus calculate the gradients multiple times.<br>
The discriminator is trained with different inputs, in the <a href="https://github.com/pytorch/examples/blob/54f4572509891883a947411fd7239237dd2a39c3/dcgan/main.py#L240">first step</a> <code>netD</code> will get the <code>real_cpu</code> inputs and the corresponding gradients will be computed <a href="https://github.com/pytorch/examples/blob/54f4572509891883a947411fd7239237dd2a39c3/dcgan/main.py#L242">afterwards</a> using <code>errD_real.backward()</code>. The computation graph is freed.<br>
In the <a href="https://github.com/pytorch/examples/blob/54f4572509891883a947411fd7239237dd2a39c3/dcgan/main.py#L246-L249">next step</a> the generator will create a fake image which is then passed to the discriminator as <code>netD(fake.detach())</code>. A new computation graph is created and the corresponding gradients will be computed <a href="https://github.com/pytorch/examples/blob/54f4572509891883a947411fd7239237dd2a39c3/dcgan/main.py#L251">afterwards</a> and accumulated to the <code>.grad</code> attributes.<br>
Finally, <code>netD</code>’s parameter will be updated via <code>optimizerD.step()</code> <a href="https://github.com/pytorch/examples/blob/54f4572509891883a947411fd7239237dd2a39c3/dcgan/main.py#L254">here</a>.</p>
<p>There is thus no need to keep any computation graphs alive as they are not reused.</p>

dragonHyeon: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Oh thank you so much! You’ve solved a problem I’ve been struggling with for hours. Now I clearly understand!</p>

dragonHyeon: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I have additional question. <a href="https://github.com/pytorch/examples/blob/54f4572509891883a947411fd7239237dd2a39c3/dcgan/main.py#L249" rel="noopener nofollow ugc">Here</a> we detached to prevent removing the computational graphs for training generator. However <a href="https://github.com/pytorch/examples/blob/54f4572509891883a947411fd7239237dd2a39c3/dcgan/main.py#L261" rel="noopener nofollow ugc">here</a> we feed forward again and doesn’t this making new computational graphs for generator whether we detach or not?</p>
<p>Is this because we focus on <code>fake</code>? Which are related to <code>netG</code> and it feed forward only once <a href="https://github.com/pytorch/examples/blob/54f4572509891883a947411fd7239237dd2a39c3/dcgan/main.py#L247" rel="noopener nofollow ugc">here</a>?</p>

ptrblck: <aside class="quote no-group" data-username="dragonHyeon" data-post="4" data-topic="176619">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/e8c25b/48.png" class="avatar"> dragonHyeon:</div>
<blockquote>
<p><a href="https://github.com/pytorch/examples/blob/54f4572509891883a947411fd7239237dd2a39c3/dcgan/main.py#L249">Here</a> we detached to prevent removing the computational graphs for training generator. However <a href="https://github.com/pytorch/examples/blob/54f4572509891883a947411fd7239237dd2a39c3/dcgan/main.py#L261">here</a> we feed forward again and doesn’t this making new computational graphs for generator whether we detach or not?</p>
<p>Is this because we focus on <code>fake</code>? Which are related to <code>netG</code> and it feed forward only once <a href="https://github.com/pytorch/examples/blob/54f4572509891883a947411fd7239237dd2a39c3/dcgan/main.py#L247">here</a>?</p>
</blockquote>
</aside>
<p>We are detaching <code>fake</code> in <code>netD(fake.detach())</code> since we only want to train the discriminator to learn how to detect fake input images. Note that the <code>label</code> is also filled with <code>fake_label</code>.<br>
The <code>errD_fake.backward()</code> call will only compute the gradients for parameters of <code>netD</code>, since the computation graph in <code>netG</code> used to create <code>fake</code> is detached from the previous forward pass.</p>
<p>The second usage via <code>output = netD(fake)</code> keeps <code>fake</code> attached to <code>netG</code> since now we are training the generator. The <code>label</code> tensor is filled with <code>real_label</code> and <code>errG.backward()</code> will now compute the gradients for parameters in the generator <code>netG</code> (as well as <code>netD</code>, but we will zero them out and won’t use them). This step allows the generator to learn how to “fool” the discriminator in “thinking” the <code>fake</code> input is a real image.</p>

dragonHyeon: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I see. We detach <code>fake</code> since discriminator should train just to discriminate whether the image is fake or real but not how generator make image looks realistic.</p>
<p>So we detach <code>fake</code> because we don’t need to track generators parameter while training discriminator. But also if we don’t detach, still there’s nothing matter for discriminator to train (maybe just some performance isses) but it matters for generator. Does this due to <code>fake</code> have already been backward <a href="https://github.com/pytorch/examples/blob/54f4572509891883a947411fd7239237dd2a39c3/dcgan/main.py#L249" rel="noopener nofollow ugc">here</a> if we don’t detach so that we can no longer use this computational graph to train generator unless we feed forward again to create new <code>fake</code> again?</p>

ptrblck: <p>You don’t want to reuse the same computation graph since the generator would be trained to help the discriminator detect fake images. Take another look at the losses and how the targets are defined to train the discriminator vs. the generator.</p>

dragonHyeon: <p>Oh that’s right. Now I got your point. I really appreciate your help!</p>