Begginner help with DCGAN tutorial

lobachevscki: <p>Hi!</p>
<p>I followed the DCGAN tutorial that is shared in the webpage here <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">DCGAN Tutorial — PyTorch Tutorials 2.2.0+cu121 documentation</a> (including reading the papers and other materials).</p>
<p>After checking my implementation again and again im pretty sure there are no errors or problems, however, when it comes to training both the error of the the discriminator is always zero and the network doesnt get trained.</p>
<p>For example printing how the network evolve i get this:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/2/52c1fb4c6f214442ceefe7ddbfeeec65b82b7eb4.png" data-download-href="https://discuss.pytorch.org/uploads/default/52c1fb4c6f214442ceefe7ddbfeeec65b82b7eb4" title="Figure_1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52c1fb4c6f214442ceefe7ddbfeeec65b82b7eb4_2_690x345.png" alt="Figure_1" data-base62-sha1="bO6KTgBj6XyWhYWCtLfKDu8d52c" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52c1fb4c6f214442ceefe7ddbfeeec65b82b7eb4_2_690x345.png, https://discuss.pytorch.org/uploads/default/original/3X/5/2/52c1fb4c6f214442ceefe7ddbfeeec65b82b7eb4.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/2/52c1fb4c6f214442ceefe7ddbfeeec65b82b7eb4.png 2x" data-dominant-color="FCFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Figure_1</span><span class="informations">1000×500 15.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And this is part of the log output in Pycharm:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>[0/0][950/1583]</th>
<th>Loss_D: 0.0000</th>
<th>Loss_G: 100.0000</th>
<th>D(x): 1.0000</th>
<th>D(G(z)): 0.0000 / 0.0000</th>
</tr>
</thead>
<tbody>
<tr>
<td>[0/0][1000/1583]</td>
<td>Loss_D: 0.0000</td>
<td>Loss_G: 100.0000</td>
<td>D(x): 1.0000</td>
<td>D(G(z)): 0.0000 / 0.0000</td>
</tr>
</tbody>
</table>
</div><p>(the forum doesnt let me upload more than one picture).</p>
<p>If you plot the faces is just noise</p>
<p>Im pretty sure is something that my lack of experience is not letting me debug correctly, i would appreciate any help.</p>
<p>Find below the script im using, if you need to test you need (as the tutorial requires) to have the celeba collection in the root folder.</p>
<pre><code class="lang-auto">#%matplotlib inline
import argparse
import os
import random
import torch
import torch.nn as nn
import torch.nn.parallel
import torch.optim as optim
import torch.utils.data
import torchvision.datasets as dset
import torchvision.transforms as transforms
import torchvision.utils as vutils
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from IPython.display import HTML

def weights_init(m):

    classname = m.__class__.__name__

    if classname.find('Conv') != -1:
        nn.init.normal_(m.weight.data, 0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        nn.init.normal_(m.weight.data, 1.0, 0.02)
        nn.init.constant_(m.bias.data, 0.0)



class Generator(nn.Module):
    def __init__(self, ngpu, nz, ngf, nc):
        super(Generator, self).__init__()
        self.ngpu = ngpu

        self.generator = nn.Sequential(

            nn.ConvTranspose2d(nz, ngf*8, 4, 1, 0, bias = False),
            nn.BatchNorm2d(ngf*8),
            nn.ReLU(True),

            nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 4),
            nn.ReLU(True),

            nn.ConvTranspose2d(ngf * 4, ngf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 2),
            nn.ReLU(True),

            nn.ConvTranspose2d(ngf * 2, ngf, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf),
            nn.ReLU(True),

            nn.ConvTranspose2d(ngf, nc, 4, 2, 1, bias=False),
            nn.Tanh(),
        )

    def forward(self, input):
        return self.generator(input)


class Discriminator(nn.Module):
    def __init__(self, ngpu, nc, ndf):

        super(Discriminator, self).__init__()
        self.ngpu = ngpu

        self.discriminator = nn.Sequential(

            nn.Conv2d(nc, ndf, 4, 2, 1, bias = False),
            nn.LeakyReLU(0.2, inplace = True),

            nn.Conv2d(ndf, ndf*2, 4,2,1, bias = False),
            nn.BatchNorm2d(ndf*2),
            nn.LeakyReLU(0.2, inplace = True),

            nn.Conv2d(ndf*2, ndf*4, 4,2, 1, bias = False),
            nn.BatchNorm2d(ndf*4),
            nn.LeakyReLU(0.2, inplace = True),

            nn.Conv2d(ndf*4, ndf*8, 4, 2, 1, bias = False),
            nn.BatchNorm2d(ndf*8),
            nn.LeakyReLU(ndf*8),
            nn.LeakyReLU(0.2, inplace = True),

            nn.Conv2d(ndf*8,1, 4, 1, 0, bias= False),
            nn.Sigmoid()

        )
    def forward(self, input):
        return self.discriminator(input)
def main():

    data_path = os.getcwd()
    #images_path = os.path.join(data_path, 'celeba').replace('\\', '/')
    images_path = 'celeba'
    #print(images_path)

    manual_seed = 42

    random.seed(manual_seed)
    torch.manual_seed(manual_seed)
    torch.use_deterministic_algorithms(True)

    workers = 2
    batch_size = 128
    image_size = 64
    nc = 3
    nz = 100
    ngf = 64
    ndf = 64
    epochs = 1
    lr = 0.0002

    beta1 = 0.5
    ngpu = 1

    data_transforms = transforms.Compose([transforms.Resize(size=image_size), transforms.CenterCrop(image_size),
                                          transforms.ToTensor(),
                                          transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

    dataset = dset.ImageFolder(root=images_path, transform=data_transforms)

    data_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, shuffle=True, num_workers=workers)

    device = torch.device('cuda:0' if (torch.cuda.is_available() and ngpu &gt; 0) else 'cpu')


    #-- Test plotting
    """
    real_batch = next(iter(data_loader))
    plt.figure(figsize=(8, 8))
    plt.axis('off')
    plt.title('Training Images')
    plt.imshow(np.transpose(vutils.make_grid(real_batch[0].to(device)[:64], padding=2, normalize=True).cpu(), (1, 2, 0)))
    plt.show()
    """

    #-- Apply Generator

    netG = Generator(ngpu, nz, ngf, nc).to(device)

    if (device.type == 'cuda') and (ngpu &gt; 1):
        netG = nn.DataParallel(netG, list(range(ngpu)))

    netG.apply(weights_init)


    netD = Discriminator(ngpu, nc, ndf).to(device)

    if (device.type == 'cuda') and (ngpu &gt; 1):
        netG = nn.DataParallel(netG, list(range(ngpu)))

    netD.apply(weights_init)

    #print(netG)
    #print(netD)

    criterion = nn.BCELoss()

    fixed_noise = torch.randn(64, nz, 1, 1, device = device)

    real_label = 1
    fake_label = 0

    optimizerD = optim.Adam(netD.parameters(), lr = lr, betas = (beta1, 0.999))
    optimizerG = optim.Adam(netG.parameters(), lr=lr, betas=(beta1, 0.999))


    #print(optimizerD)
    #print(optimizerG)

    img_list = []
    G_losses = []
    D_losses = []
    iters = 0

    print('Starting Training Loop...')


    for epoch in range(epochs):

        for i, data in enumerate(data_loader, 0):

            netD.zero_grad()

            real_cpu = data[0].to(device)
            b_size = real_cpu.size(0)
            label = torch.full((b_size,), real_label, dtype = torch.float, device = device)

            output = netD(real_cpu).view(-1)
            errD_real = criterion(output, label)

            errD_real.backward()
            D_x = output.mean().item()

            noise = torch.randn(b_size, nz, 1, 1, device = device)

            fake = netG(noise)
            label.fill_(fake_label)

            output = netD(fake.detach()).view(-1)

            errD_fake = criterion(output, label)

            errD_fake.backward()
            D_G_z1 = output.mean().item()

            errD = errD_real + errD_fake

            optimizerD.step()

            #print('{} {} {} '.format(errD_real, errD_fake, errD))

            ###############################################
            ###############################################

            netG.zero_grad()

            label.fill_(real_label)

            output = netD(fake).view(-1)

            errG = criterion(output, label)

            errG.backward()
            D_G_z2 = output.mean().item()

            optimizerG.step()

            if i % 50 == 0:
                print('[%d/%d][%d/%d]\tLoss_D: %.4f\tLoss_G: %.4f\tD(x): %.4f\tD(G(z)): %.4f / %.4f'
                      % (epoch, epochs - 1, i, len(data_loader),
                         errD.item(), errG.item(), D_x, D_G_z1, D_G_z2))

            G_losses.append(errG.item())
            D_losses.append(errD.item())

            if (iters % 500 == 0) or ((epoch == epochs-1) and (i == len(data_loader)-1)):
                with torch.no_grad():
                    fake = netG(fixed_noise).detach().cpu()

                img_list.append(vutils.make_grid(fake, padding = 2, normalize = True))

            iters += 1


    plt.figure(figsize=(10,5))
    plt.title("Generator and Discriminator Loss During Training")
    plt.plot(G_losses, label = 'G')
    plt.plot(D_losses, label='D')
    plt.xlabel("iterations")
    plt.xlabel("Loss")
    plt.legend()
    plt.show()
    """
    real_batch = next(iter(data_loader))
    plt.figure(figsize = (15,15))
    plt.subplot(1, 2, 1)
    plt.axis('off')
    plt.title('Real Images')
    plt.imshow(np.transpose(vutils.make_grid(real_batch[0].to(device)[:64], padding = 5, normalize= True).cpu(),
                            (1, 2, 0)))

    plt.subplot(1, 2, 2)
    plt.axis('off')
    plt.title('Fake Images')
    plt.imshow(np.transpose(img_list[-1], (1, 2, 0)))
    plt.show()
    """

if __name__ == '__main__':
    main()
</code></pre>

KFrank: <p>Hi Julián!</p>
<aside class="quote no-group" data-username="lobachevscki" data-post="1" data-topic="192201" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lobachevscki/48/64389_2.png" class="avatar"> lobachevscki:</div>
<blockquote>
<p>After checking my implementation again and again im pretty sure there are no errors or problems</p>
</blockquote>
</aside>
<p>It does look like you are following the tutorial code quite closely, but just<br>
to be sure, you might want to double-check things one more time …</p>
<aside class="quote no-group">
<blockquote>
<p>however, when it comes to training both the error of the the discriminator is always zero and the network doesnt get trained.</p>
</blockquote>
</aside>
<p>Assuming that some copy-paste error hasn’t crept into your code<br>
somewhere, the fact that you don’t get the same results as those<br>
shown in the tutorial is quite surprising.</p>
<p>Some things you might try:</p>
<p>You are using <code>42</code> for your random-number <code>manual_seed()</code>, while the<br>
tutorial uses <code>999</code>.  Try using <code>999</code> and see if you can reproduce the<br>
tutorial’s results line by line.</p>
<p>I can’t tell – neither for you nor the tutorial – which version of pytorch<br>
is being used and whether / which version of a gpu is being used.</p>
<p>Note that many recent gpus silently use reduced precision (the “dreaded<br>
TensorFloat-32”).  GAN training tends to be finicky.  It’s hypothetically<br>
possible that TensorFloat-32 could cause not-quite-unstable GAN training<br>
to become unstable, giving you the results you see.  So if you are using<br>
a gpu, you might try running on your cpu (or turning off TF32, if that’s<br>
applicable).</p>
<p>You could also try running the tutorial’s Colab version.  (See the “Run in<br>
Google Colab” link at the top of the tutorial.)  Does the Colab version still<br>
get good results?  If so, you could print out intermediate results in both your<br>
version and the tutorial’s Colab and see where they first differ (by more<br>
than some small round-off error), using a systematic divide-and-conquer<br>
approach, if it comes to that.</p>
<aside class="quote no-group">
<blockquote>
<p>|[0/0][950/1583]|Loss_D: 0.0000|Loss_G: 100.0000|D(x): 1.0000|D(G(z)): 0.0000 / 0.0000|</p>
</blockquote>
</aside>
<p>It’s noteworthy that your generator loss is “exactly” <code>100.0000</code>.  This<br>
suggests that the output of the <code>Sigmoid</code> is saturating to zero when<br>
the <code>Generator</code> is trying to get the <code>Discriminator</code> <em>not</em> to predict a<br>
small value for <code>real_label = 1</code>.  If, in this situation, <code>Discriminator</code>,<br>
correctly, from its perspective, predicts a very small value for the image<br>
<code>fake</code>, when <code>BCELoss</code> compares that prediction to <code>real_label = 1</code>,<br>
<code>BCELoss</code> will diverge and cap its divergent value at <code>100.0</code>), likely<br>
causing <code>Generator</code> to quit training.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">class Discriminator(nn.Module):
    def __init__(self, ngpu, nc, ndf):

        super(Discriminator, self).__init__()
        self.ngpu = ngpu

        self.discriminator = nn.Sequential(

            nn.Conv2d(nc, ndf, 4, 2, 1, bias = False),
            nn.LeakyReLU(0.2, inplace = True),
            ...
            nn.Conv2d(ndf*8,1, 4, 1, 0, bias= False),
            nn.Sigmoid()
        )
...
    criterion = nn.BCELoss()
</code></pre>
</blockquote>
</aside>
<p>Your <code>Discriminator</code> passes its output through <code>Sigmoid</code> and you then<br>
use <code>BCELoss</code> as your loss criterion.  Now this seems to be working for<br>
the tutorial, but it can have problems with numerical stability, in particular<br>
where <code>Sigmoid</code> saturates, as described above, causing your training to<br>
stop.</p>
<p>Try getting rid of the <code>Sigmoid</code> and use <code>BCEWithLogitsLoss</code> to avoid<br>
this potential problem.  You will no longer have the possibility of <code>Sigmoid</code><br>
saturating (underflowing) to zero.</p>
<p>The thinking is that if the GAN training is on the edge of instability, it<br>
might work for the tutorial, but get pushed over the edge into instability<br>
by <code>BCELoss</code> when you run it, just because of truly minor differences in<br>
what you are doing.</p>
<p>If none of the easier ideas work, try – assuming that the tutorial’s Colab<br>
still works – the suggestion above of running your code side-by-side with<br>
Colab until you locate the source of the discrepancy.</p>
<p>Good luck!</p>
<p>K. Frank</p>