DCGAN generator doesn't seem to be learning anything

username: CaptainCyber
cooked: <p>I’ve been building a DCGAN following the <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="noopener nofollow ugc">PyTorch tutorial</a> with some modifications.</p>
<p>The data I’m training with is Minecraft skins that I scraped from the internet. The images’ dimensions are 64x32x3. See the first image below for an example. (Sorry about having to combine all the images into one. As a new user of the forum, I can only include one image in my post.)</p>
<p>No matter what I do, after the first few epochs of training, the generator seems to keep generating images with weird patterns. See the second image below for an example.</p>
<p>As I continue training, it seems to trend towards making the whole image black except for a few pixels. See the third image below. This effect becomes even more pronounced as I train the model for longer. The fourth image below shows the generator and discriminator losses as training progresses.</p>
<p>I also tried training it on only one image, expecting it to over fit and always generate the same picture, so I could see if the model was working correctly. Surprisingly, the model seems to continue to output garbage. The image I used to train is the first image below. See the fifth image below for an example of a generated image and the sixth image for a graph of the losses.</p>
<p>In all cases, the generator’s loss appears to continuously increase as the discriminator’s loss decreases. This leads me to believe that the generator isn’t training properly. I think I have an error somewhere either in my training loop or in the generator’s architecture. I’ve checked both multiple times, but I can’t seem to find any errors. I couldn’t seem to find any information on using ConvTranspose2d to create rectangular images, and I may have introduced an error when I tried to figure out how to do it myself. Any help would be appreciated.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3d1d03b7bc6224434e901e2020b3599ffcc712c4.png" data-download-href="https://discuss.pytorch.org/uploads/default/3d1d03b7bc6224434e901e2020b3599ffcc712c4" title="forum post"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d1d03b7bc6224434e901e2020b3599ffcc712c4_2_207x500.png" alt="forum post" data-base62-sha1="8IDhV0HD1pj7IJxW4qREl4CDwGM" width="207" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d1d03b7bc6224434e901e2020b3599ffcc712c4_2_207x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d1d03b7bc6224434e901e2020b3599ffcc712c4_2_310x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d1d03b7bc6224434e901e2020b3599ffcc712c4_2_414x1000.png 2x" data-dominant-color="CFCDCB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">forum post</span><span class="informations">609×1466 93.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here is the code for the models:</p>
<pre><code class="lang-auto">class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()

        self.conv_transpose_1 = nn.ConvTranspose2d(50, 64, 4, 4)
        self.batch_norm_1 = nn.BatchNorm2d(64)

        self.conv_transpose_2 = nn.ConvTranspose2d(64, 32, 2, 2)
        self.batch_norm_2 = nn.BatchNorm2d(32)

        self.conv_transpose_3 = nn.ConvTranspose2d(32, 16, 2, 2)
        self.batch_norm_3 = nn.BatchNorm2d(16)

        self.conv_transpose_4 = nn.ConvTranspose2d(16, 8, 2, 2)
        self.batch_norm_4 = nn.BatchNorm2d(8)
                                           
        self.conv_transpose_5 = nn.ConvTranspose2d(8, 3, 1, 1)

    def forward(self, x):
        x = x.view(-1, 50, 1, 2)

        x = self.conv_transpose_1(x)
        x = self.batch_norm_1(x)
        x = F.relu(x)

        x = self.conv_transpose_2(x)
        x = self.batch_norm_2(x)
        x = F.relu(x)

        x = self.conv_transpose_3(x)
        x = self.batch_norm_3(x)
        x = F.relu(x)

        x = self.conv_transpose_4(x)
        x = self.batch_norm_4(x)
        x = F.relu(x)
        
        x = self.conv_transpose_5(x)
        x = (F.hardtanh(x) + 1) / 2
        
        return x


class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()

        self.conv_1 = nn.Conv2d(3, 64, 4, 2)
        self.dropout_1 = nn.Dropout()

        self.conv_2 = nn.Conv2d(64, 128, 2, 2)
        self.batch_norm_1 = nn.BatchNorm2d(128)
        self.dropout_2 = nn.Dropout()

        self.conv_3 = nn.Conv2d(128, 64, 2, 2)
        self.batch_norm_2 = nn.BatchNorm2d(64)
        self.dropout_3 = nn.Dropout()
        
        self.linear_1 = nn.Linear(64 * 3 * 7, 128)
        self.linear_2 = nn.Linear(128, 1)

    def forward(self, x):
        x = self.conv_1(x)
        x = F.leaky_relu(x, 0.2)
        x = self.dropout_1(x)

        x = self.conv_2(x)
        x = self.batch_norm_1(x)
        x = F.leaky_relu(x, 0.2)
        x = self.dropout_2(x)

        x = self.conv_3(x)
        x = self.batch_norm_2(x)
        x = F.leaky_relu(x, 0.2)
        x = self.dropout_3(x)

        x = self.linear_1(x.view(x.shape[0], -1))
        x = F.leaky_relu(x, 0.2)
        x = self.linear_2(x)
        x = torch.sigmoid(x)

        return x
</code></pre>
<p>Here’s the code for the training loop:</p>
<pre><code class="lang-auto">discriminator = Discriminator().to('cuda')
generator = Generator().to('cuda')

criterion = nn.BCELoss()
optimizerDiscriminator = optim.Adam(discriminator.parameters(), lr=0.0001)
optimizerGenerator = optim.Adam(generator.parameters(), lr=0.01)

d_loss = []
g_loss = []

for epoch in range(NUM_EPOCHS):
    for i, (images, _) in enumerate(loader, 0):
        batch_size = images.size(0)
        
        # Train discriminator
        discriminator.zero_grad()

        # Train the discriminator with real images
        labels = torch.full((batch_size,), REAL_LABEL, dtype=torch.float).to('cuda')
        output = discriminator(images.to('cuda')).view(-1)
        err = criterion(output, labels)
        err.backward()
        d_loss.append(err)
        print(err)
        
        # Train the discriminator with generated images
        labels = torch.full((batch_size,), FAKE_LABEL, dtype=torch.float).to('cuda')
        noise = torch.randn(batch_size, 100).to('cuda')
        with torch.no_grad():
            fake_images = generator(noise)
        output = discriminator(fake_images.detach()).view(-1)
        err = criterion(output, labels)
        err.backward()
        d_loss[-1] += err
        print(err)
        
        optimizerDiscriminator.step()
        
        # Train generator
        generator.zero_grad()
        labels = torch.full((batch_size,), REAL_LABEL, dtype=torch.float).to('cuda')
        noise = torch.randn(batch_size, 100).to('cuda')
        fake_images = generator(noise)
        output = discriminator(fake_images).view(-1)
        err = criterion(output, labels)
        err.backward()
        g_loss.append(err)
        print(err)
        optimizerGenerator.step()
        
        print()
    print(epoch)
</code></pre>

username: CaptainCyber
cooked: <p>I’ve continued trying to debug this, and I’ve realized that even after only 10 epochs, the generator starts to output similar images no matter the input. I’m not sure what could cause this, and I have no idea how to fix it. Any suggestions?</p>

username: Henry_Chibueze
cooked: <p>Try detaching the gradients of the generator when u train the discriminator on generated images</p>
<p><code>generator(noise).detach()</code></p>

username: CaptainCyber
cooked: <p>I call detach before passing <code>fake_images</code> into the discriminator:</p>
<pre><code class="lang-auto">output = discriminator(fake_images.detach()).view(-1)
</code></pre>
<p>I’m also using <code>torch.no_grad()</code>, so I don’t <em>think</em> it should be necessary anyways, but I’m not 100% sure about that.</p>

username: hossein_rajoli_Nowde
cooked: <p>In this tutorial being in equilibrium point within Training process is very sensitive to the Batch Size, try to find the optimum batch size. In my case, changing the batch size resulted in a meaningful collaboration between Generator and Discriminator.</p>

username: Max_Muster
cooked: <aside class="quote no-group" data-username="CaptainCyber" data-post="1" data-topic="122740">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/e47c2d/48.png" class="avatar"> CaptainCyber:</div>
<blockquote>
<p>after the first few epochs of training, the generator seems to keep generating images with weird pattern</p>
</blockquote>
</aside>
<p>I had a similar Problem when following the tutorial with CIFAR10 dataset which in which the images were stored as [0, 255]. For me, scaling the images to [-1, 1] helped.</p>