How to turn off gradient during GAN training

Zhenlan_Wang: <p>I am going through the DCGAN tutorials <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="nofollow noopener">tutorials</a>.</p>
<p>One question I have is how do you turn off the gradient history tracking for discriminator when you are training the generator. In the tutorial, it is not turned off as shown below.</p>
<pre><code class="lang-auto">...
# this part trains generator
        netG.zero_grad()
        label.fill_(real_label)  # fake labels are real for generator cost
        # Since we just updated D, perform another forward pass of all-fake batch through D
        output = netD(fake).view(-1)
        # Calculate G's loss based on this output
        errG = criterion(output, label)
        # Calculate gradients for G
        errG.backward()
...
</code></pre>
<p>I see the grad tracking is turned off for generator when training the discriminator by calling detach on fake image. But not the other way around. Thanks in advance for your help <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=6" title=":smiley:" class="emoji" alt=":smiley:"></p>

ptrblck: <p>Since the discriminator’s optimizer won’t be called for the generator updates, nothing bad will happen.<br>
You could set the <code>requires_grad</code> attributes of the discriminator’s parameters to <code>False</code> and reset them to <code>True</code> after the generator updates, but this is not really necessary, as you are using different optimizers.</p>

Zhenlan_Wang: <p>Thanks for your reply. But my understanding of how optimizer works is that they take .grad  and current weight and then update them. So even through discriminator’s optimizer does not get called, .grad gets calculated for discriminator’s weight when you call .backward and on the forward pass, cache is saved as require_grad==True. Let me know if that is the case or I mis-understand something.</p>

ptrblck: <p>Your understanding is basically correct. Since <code>netD.zero_grad()</code> is called before updating the discriminator, these gradients will be cleared.</p>

Zhenlan_Wang: <p>In other words, even though discriminator’s optimizer won’t be called for the generator updates, you will still save some time and memory by turning requires_grad off. This is because regardless of whether or not there will be optimizer call on weight, weight.grad get calculated (if requires_grad is on).</p>

ptrblck: <p>I’m not sure about the computation and memory usage, since you need the gradients to backpropagate to the generator.</p>

Zhenlan_Wang: <p>I will benchmark it and report back.</p>

Zhenlan_Wang: <p>There is a slight improvement in terms of time. I single out the part of the code that trains generator as shown below.</p>
<pre><code class="lang-auto">label = torch.full((batch_size,), real_label, device=device)

def train():
    netG.zero_grad()
    noise = torch.randn(batch_size, nz, 1, 1, device=device)
    fake = netG(noise)
    output = netD(fake).view(-1)
    errG = criterion(output, label)
    errG.backward()
    optimizerG.step()
    
%timeit train()
# -&gt; 44.3 ms ± 1.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
for p in netD.parameters():
    temp.requires_grad_(False)


%timeit train()
# -&gt; 41 ms ± 253 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
</code></pre>

ptrblck: <p>Thanks for the debugging!<br>
<a href="https://discuss.pytorch.org/t/update-only-a-middle-layer-of-a-neural-network/35302/4?u=ptrblck">This post</a> might explain the benefits you are seeing.</p>

Zhenlan_Wang: <p>Thanks for the reference and your help. Much appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=7" title=":smiley:" class="emoji" alt=":smiley:"></p>

saba: <p>Hi Ptrblck,</p>
<p>in this line based on which error the updating is done?</p>
<pre><code class="lang-auto">errD_penalty = errD_fake - errD_real + grad_penalty
errD = errD_fake - errD_real
optimizerD.step()
</code></pre>

ptrblck: <p>In the posted code snippet, no <code>backward()</code> operation is called, thus no gradients are calculated, and <code>optimizerD.step()</code> will use any gradients, which are already assigned to the passed parameters.</p>
<p>I guess. you’ve forgotten to post the code containing the <code>backward()</code> operation.<br>
In that case the <code>loss.backward()</code> is used to calculate the gradients.</p>

saba: <p>That’s the sample code, the self.d_optimizer.step() is optimized with which error? with   d_loss = d_loss_fake - d_loss_real? does it need backward()?</p>
<pre><code class="lang-auto">                # Train discriminator
                # WGAN - Training discriminator more iterations than generator
                # Train with real images
                d_loss_real = self.D(images)
                d_loss_real = d_loss_real.mean(0).view(1)
                d_loss_real.backward(one)

                # Train with fake images
                if self.cuda:
                    z = Variable(torch.randn(self.batch_size, 100, 1, 1)).cuda()
                else:
                    z = Variable(torch.randn(self.batch_size, 100, 1, 1))
                fake_images = self.G(z)
                d_loss_fake = self.D(fake_images)
                d_loss_fake = d_loss_fake.mean(0).view(1)
                d_loss_fake.backward(mone)
                d_loss = d_loss_fake - d_loss_real
                Wasserstein_D = d_loss_real - d_loss_fake
                self.d_optimizer.step()</code></pre>

ptrblck: <p>Assuming <code>d_optimizer</code> contains the parameters of <code>D</code>, then <code>d_optimizer.step()</code> is using the gradients of <code>d_loss_real.backward(one)</code> and <code>d_loss_fake.backward(mone)</code> to update the parameters.<br>
Both losses will create gradients in <code>D</code> and these gradients will be accumulated in all parameters of <code>D</code>, which require gradients.</p>
<p><code>Wasserstein_D = d_loss_real - d_loss_fake</code> is not used to create the gradients in this example, but based on the formula could yield the same gradients as seen here:</p>
<pre><code class="lang-python">torch.manual_seed(2809)
model = models.resnet18()

out_real = model(torch.randn(1, 3, 224, 224))
out_real = out_real.mean()
out_real.backward()

out_fake = model(torch.randn(1, 3, 224, 224))
out_fake = out_fake.mean()
out_fake.backward(torch.ones_like(out_fake) * -1)

print(model.conv1.weight.grad.abs().sum())

torch.manual_seed(2809)
model = models.resnet18()

out_real = model(torch.randn(1, 3, 224, 224))
out_real = out_real.mean()

out_fake = model(torch.randn(1, 3, 224, 224))
out_fake = out_fake.mean()

loss = out_real - out_fake
loss.backward()

print(model.conv1.weight.grad.abs().sum())
</code></pre>
<p>You could check the <code>.grad</code> attributes of some layer and compare the value to the current approach and to the gradient created by <code>Wasserstein_D.backward()</code>.</p>

saba: <p>Great, I appreciate your help</p>