GAN loss is not changing, has gradients but is stuck

Joshua_Clancy: <p>Hello,</p>
<p>I am new to GAN models and I have hit a weird problem.</p>
<p>The model does not seem to be updating properly. The discriminator and generator BCE loss stay at approximately 0.69 the entire time, its almost as if the two optimizers are not working.  I have been banging my head against this problem for three days now so any help would be very appreciated.</p>
<p><strong>Code:</strong></p>
<pre><code class="lang-auto">##### Print #####
model = GAN_Generator(run)
disc = GAN_Discriminator(run)
printNetworkSummary(disc, False)
printNetworkSummary(model, False)
device = "cuda"

data_loader = torch.utils.data.DataLoader(train_set, batch_size=run.batch, shuffle=run.shuf)
val_data_loader = torch.utils.data.DataLoader(val_set, batch_size=run.batch, shuffle=run.shuf)

model.to(device)
disc.to(device)
wandb.watch(model, log='all')

optimizer = getOptimizer(model.parameters(), run)
disc_optimizer = getOptimizer(disc.parameters(), run)
        
# Epoch loop  
for epoch in range(run.E_num):
            model.train()
            disc.train()
            lossy = 0
            # Training Loop ###########################################################################################################
            for image_batch, labels in data_loader:
                disc_optimizer.zero_grad()
                optimizer.zero_grad()
                
                label_real = torch.ones(image_batch.size(0), device=dev)
                label_fake = torch.zeros(image_batch.size(0), device=dev)

                image_batch = image_batch.to(device)
                labels = labels.to(device).float()
                
                recon_x = model(outputDict, image_batch)

                real_pred = disc(outputDict, image_batch)
                fake_pred = disc(outputDict, recon_x.detach())

                disc_lossy = 0.5 * (F.binary_cross_entropy(real_pred, label_real) + F.binary_cross_entropy(fake_pred, label_fake))
                disc_lossy.backward(retain_graph=True)
                disc_optimizer.step()

                fake_pred = disc(outputDict, recon_x)
                lossy = F.binary_cross_entropy(fake_pred, label_real)

                lossy.backward()              
                optimizer.step()
</code></pre>
<p><strong>Models:</strong></p>
<p>The discriminator model is simply a set of convolution relus and batchnorms ending in a linear classifier with a sigmoid activation.</p>
<p>The generator model is actually a convolutional autoencoder which also ends in a sigmoid activation.</p>
<p>(note I am using the F.binary_cross_entropy loss which plays nice with sigmoids)<br>
<strong>Tests:</strong></p>
<p>I have messed around with batchsize and step size and have not gotten any improvement. Generally, the loss initially goes from .7 to .69 and then oscillates between 0.685 and 0.695, with no improvement in the generated image.</p>
<p>To make sure the parameters were updating at all I used:</p>
<pre><code class="lang-auto">a = list(disc.parameters())[0].clone()
disc_lossy.backward(retain_graph=True)
disc_optimizer.step()
b = list(disc.parameters())[0].clone()
print((a == b))
</code></pre>
<p>This printed false, so the parameters <strong>are</strong> updating.</p>
<p>I then used this function to check the gradients.</p>
<pre><code class="lang-auto">def checkGrad(model, loss):
    print("loss grad : ", loss.grad)
    for name, param in model.named_parameters():
        if param.grad is not None:
            print(name, param.grad.sum())
        else:
            print(name, param.grad)
</code></pre>
<p>This printed:<br>
loss grad :  1<br>
conv1.conv.weight tensor(0.3062, device=‘cuda:0’)<br>
conv1.conv.bias tensor(0.1214, device=‘cuda:0’)<br>
conv1.norm.weight tensor(-0.0040, device=‘cuda:0’)<br>
conv1.norm.bias tensor(-0.0181, device=‘cuda:0’)<br>
…etc.</p>
<p>That all looks fine to me, so I am confused by what is going wrong.</p>
<p>Once again any help would be greatly appreciated, I have found nothing online that matches my problem. Thanks</p>

Joshua_Clancy: <p>Ahhhhh finally figured it out. For anyone else with a similar problem, it was because I had retain_graph = True on the first backward function. In other contexts that is useful with multiple losses/models but not in this instance.</p>

Joshua_Clancy: <p>Actually sneaky, sneaky… there was another problem. This one I do not understand. With batchNorm on each layer, the Discriminator loss gets stuck at ~0.69. I do not know why but when I switch my normalization method to dropout everything starts working.</p>
<p>This person has had a similar experience:<br>
</p><aside class="onebox stackexchange">
  <header class="source">
      <a href="https://datascience.stackexchange.com/questions/56860/dc-gan-with-batch-normalization-not-working/56891#56891" target="_blank" rel="nofollow noopener">datascience.stackexchange.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://datascience.stackexchange.com/users/62938/nagabhushan-s-n" target="_blank" rel="nofollow noopener">
    <img alt="Nagabhushan S N" src="https://www.gravatar.com/avatar/b096725512960a65abe37b76c383de67?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://datascience.stackexchange.com/questions/56860/dc-gan-with-batch-normalization-not-working/56891#56891" target="_blank" rel="nofollow noopener">DC GAN with Batch Normalization not working</a>
</h4>

<div class="tags">
  <strong>training, gan, mnist</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://datascience.stackexchange.com/users/62938/nagabhushan-s-n" target="_blank" rel="nofollow noopener">
    Nagabhushan S N
  </a>
  on <a href="https://datascience.stackexchange.com/questions/56860/dc-gan-with-batch-normalization-not-working/56891#56891" target="_blank" rel="nofollow noopener">02:21PM - 03 Aug 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>