GAN only learns properly when using detach()

ryuser: <p>Hello,</p>
<p>I am not quite sure how to categorize this question, so I am sorry in advance for this.</p>
<p>I am currently training a GAN to generate medical images. My code looks like this:</p>
<pre><code class="lang-auto">            # CREATE GENERATED OUTPUT
            fake = gen_model(data).to(device)

            # TRAIN DISCRIMINATOR
            discriminator.zero_grad()
            disc_pred_fake = nn.Sigmoid()(discriminator(fake.detach()).to(device))
            disc_pred_real = nn.Sigmoid()(discriminator(target_img.to(device)).to(device))
            real_label = torch.ones(data.shape[0], 1).to(device)
            fake_label = torch.zeros(data.shape[0], 1).to(device)
            loss_disc_fake = nn.BCELoss()(disc_pred_fake, fake_label)
            loss_disc_real = nn.BCELoss()(disc_pred_real, real_label)
            loss_disc = (loss_disc_real + loss_disc_fake) / 2
            loss_disc.backward()
            opt_disc.step()

            # TRAIN GENERATOR

            gen_model.zero_grad()
            disc_pred_fake = nn.Sigmoid()(discriminator(fake.detach()).to(device))
            L1loss = nn.L1Loss()(fake, target_img)
            loss_gen = loss_func(disc_pred_fake, real_label)

            full_loss = L1loss + loss_gen

            full_loss.backward()
            opt_gen.step()
</code></pre>
<p>This generates very good results, but what I find strange is that if I remove</p>
<pre><code class="lang-auto">disc_pred_fake = nn.Sigmoid()(discriminator(fake.detach()).to(device)) 
</code></pre>
<p>and instead use</p>
<pre><code class="lang-auto">disc_pred_fake = nn.Sigmoid()(discriminator(fake).to(device))
</code></pre>
<p>my results are becoming complete nonsense, i.e. returning images that are almost only 1 etc. (I normalize between -1 and 1).<br>
But when browsing through implementations of GANs I only see people using:</p>
<pre><code class="lang-auto">disc_pred_fake = nn.Sigmoid()(discriminator(fake).to(device))
</code></pre>
<p>instead of the version with detach(). Am I doing it wrong?<br>
If so, what does the detach() do in my case?<br>
Am I implementing the GAN in a wrong manner when using the version with .detach(), i.e. doing strange things with the gradient and thus influencing the result in a good manner by coincidence?</p>
<p>I am really grateful for any kind of help since I am really stuck on this <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>Best regards</p>

ptrblck: <aside class="quote no-group" data-username="ryuser" data-post="1" data-topic="187285">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/839c29/48.png" class="avatar"> ryuser:</div>
<blockquote>
<p>Am I doing it wrong?<br>
If so, what does the detach() do in my case?</p>
</blockquote>
</aside>
<p>Yes, since your generator is now detached from the computation graph and its parameters (used to create <code>fake</code>) won’t get any gradients. You can double check it by printing their <code>.grad</code> attribute, which should show <code>None</code> for all parameters of the generator.<br>
Also, since gradients were never computed the optimizer also won’t update anything in the generator unless you are using weight decay.</p>

ryuser: <p>Thanks a lot for your answer!</p>
<p>I took another look at the example implementation from this site: <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">DCGAN Tutorial — PyTorch Tutorials 2.0.1+cu117 documentation</a><br>
and below in the code there is this line:</p>
<pre><code class="lang-auto">with torch.no_grad():
                fake = netG(fixed_noise).detach().cpu()
</code></pre>
<p>Why is the detach() called here? My understanding was now: .detach() removes the gradients, but torch.no_grad() tells pytorch to not calculate gradients anyways.<br>
I am asking since in my evaluation im not using detach(), I just use with.torch.no_grad().</p>
<p>Sorry if that question is rudimentary, but I really became unsure with the pitfalls .detach() has.</p>

ptrblck: <p>The posted code snippet is used for testing only as mentioned in the comment:</p>
<pre><code class="lang-python"># Check how the generator is doing by saving G's output on fixed_noise
</code></pre>
<p>and detaching the output won’t be needed since the <code>no_grad</code> context already avoids creating a computation graph.</p>