GAN does not converge even after 5000 epochs

username: venmadh
cooked: <p>I’m trying to train a GAN on a relatively small dataset of clothing. My dataset is very specific and made up of small girl dresses from one particular brand. It consists of 206 items of dimension 96x72x3. Here are a few examples:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/f/f0a3f8673b36c15d758936c75cc0bde04499f0c3.jpg" alt="IMG_Gap_58" data-base62-sha1="ykNTENfXuFZnNiEfSz2yXXfxscz" width="72" height="96"><img src="https://discuss.pytorch.org/uploads/default/original/2X/f/fc4149abe94cdfb52d6d9a79c0468c54dd455672.jpg" alt="IMG_Gap_6" data-base62-sha1="zZycsy4vNaeYm0PcrIKttvxtktk" width="72" height="96"><img src="https://discuss.pytorch.org/uploads/default/original/2X/f/f8c580e939e509b2e5fa3a7ca6370781410c8e07.jpg" alt="IMG_Gap_37" data-base62-sha1="zuJyQGQj3KJtmGkRWi6XlqYPpRB" width="72" height="96"><img src="https://discuss.pytorch.org/uploads/default/original/2X/f/f3a9eb11f4508c16708aebdcea09674ea1408d63.jpg" alt="IMG_Gap_48" data-base62-sha1="yLy4yrbCcy13lXPpWHZ3T2aRZ1V" width="72" height="96"><img src="https://discuss.pytorch.org/uploads/default/original/2X/b/b0697c068e23ce7fccd6ddba1c73fa2669e3807a.jpg" alt="IMG_Gap_4" data-base62-sha1="paC27DpSK26ohtJqOQMwFKn8YVc" width="72" height="96"><img src="https://discuss.pytorch.org/uploads/default/original/2X/8/84de56e13b56a2e43d97edafcd3fe7cc104dbfba.jpg" alt="IMG_Gap_44" data-base62-sha1="iXpnVPooZhv73UKk2UajGUHC80q" width="72" height="96"></p>
<p>I’ve tried a standard architecture for generator and discriminator as below on a vanilla GAN and LSGAN.</p>
<p>LSGAN Generator: ( For a vanilla GAN, the last sigmoid layer is swapped with tanh. cnnresize and view are my own classes that help with resizing))</p>
<pre><code class="lang-auto">Sequential(
  (0): Linear(in_features=20, out_features=304384, bias=True)
  (1): unView(
  )
  (2): ConvTranspose2d(64, 64, kernel_size=(5, 5), stride=(1, 1))
  (3): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True)
  (4): ReLU()
  (5): ConvTranspose2d(64, 128, kernel_size=(5, 5), stride=(1, 1))
  (6): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True)
  (7): ReLU()
  (8): ConvTranspose2d(128, 128, kernel_size=(3, 3), stride=(1, 1))
  (9): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True)
  (10): ReLU()
  (11): ConvTranspose2d(128, 3, kernel_size=(5, 5), stride=(1, 1))
  (12): View(
  )
  (13): Sigmoid()
)
</code></pre>
<p>LSGAN Discriminator: ( For a vanilla GAN, sigmoid is added to the end here. cnnresize and view are my own classes that help with resizing)</p>
<pre><code class="lang-auto">Sequential(
  (0): cnnresize(
  )
  (1): Conv2d(3, 128, kernel_size=(5, 5), stride=(2, 2))
  (2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True)
  (3): ReLU()
  (4): Conv2d(128, 48, kernel_size=(5, 5), stride=(2, 2))
  (5): BatchNorm2d(48, eps=1e-05, momentum=0.1, affine=True)
  (6): ReLU()
  (7): Conv2d(48, 64, kernel_size=(5, 5), stride=(2, 2))
  (8): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True)
  (9): ReLU()
  (10): Conv2d(64, 128, kernel_size=(5, 5), stride=(2, 2))
  (11): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True)
  (12): ReLU()
  (13): View(
  )
  (14): Dropout(p=0.5)
  (15): Linear(in_features=384, out_features=256, bias=True)
  (16): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True)
  (17): ReLU()
  (18): Dropout(p=0.5)
  (19): Linear(in_features=256, out_features=128, bias=True)
  (20): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True)
  (21): ReLU()
  (22): Dropout(p=0.5)
  (23): Linear(in_features=128, out_features=1, bias=True)
)
</code></pre>
<p>Here is my model for vanilla GAN:</p>
<pre><code class="lang-auto"># FINAL MODEL TO RUN
# ------------------


def model_gan(x,epochs,mbsize,gen_net,dis_net,gen_lr_rate,dis_lr_rate,gn_mul,no_latent):

    #optimizer = torch.optim.Adam(net.parameters(), lr=lr_rate)
    #optimizer = torch.optim.RMSprop(net.parameters(), lr=lr_rate)
    #optimizer = torch.optim.SGD(net.parameters(), lr=lr_rate, momentum=0.9)
    optimizerG = torch.optim.Adam(gen_net.parameters(), lr=gen_lr_rate)
    optimizerD = torch.optim.Adam(dis_net.parameters(), lr=dis_lr_rate)
    
    criterion = nn.BCELoss()

    global gen_loss
    gen_loss = []    
    global dis_loss
    dis_loss = []
    
    # Setting up minibatch features
    m = x.size()[0]
    mb_list = []
    mb_list = list(range(int(m/mbsize)))
    if m % mbsize == 0: # if the minibatches can be split up perfectly.
        'do nothing'
    else:
        mb_list.append(mb_list[len(mb_list)-1] + 1)
    
    #noise = sample_noise(m,nz,noise_mode)
    #noise = noise.cuda()
    
    
    for i in range(epochs): 
        for p in mb_list:
            
            # Mini batch operations
            start_index = p*mbsize
            end_index = m if p == mb_list[len(mb_list)-1] else p*mbsize + mbsize
            
            # 1. Training the discriminator - Spot real vs fake
            # -------------------------------------------------
            # 1.1 real data
            # -------------
            X_mb = x[start_index:end_index]
            m_curr = end_index - start_index
            y_mb_real = Variable(torch.Tensor(m_curr).fill_(1).float()) # real label = 1
            y_mb_real = y_mb_real.cuda()
            dis_net.zero_grad()
            output = dis_net(X_mb)
            output = output.view(-1,)
            errD_real = criterion(output, y_mb_real)
            errD_real.backward()
            
            # 1.2 fake data
            # -------------
            noise = Variable(torch.randn(m_curr,no_latent)).cuda()
            noise = noise.cuda()
            fake_data = gen_net(noise)
            
            y_mb_fake = Variable(torch.Tensor(m_curr).fill_(0).float()) # fake label = 0
            y_mb_fake = y_mb_fake.cuda()
            output = dis_net(fake_data.detach())
            output = output.view(-1,)
            errD_fake = criterion(output, y_mb_fake)
            errD_fake.backward()
            
            # Updating D network
            errD = errD_real + errD_fake
            dis_loss.append(errD.data[0])
            optimizerD.step()

            
            # 2. Training the generator - making it generate more realistic images
            # --------------------------------------------------------------------
            
            for g in range(gn_mul):
                gen_net.zero_grad()
                output = dis_net(fake_data)
                output = output.view(-1,)
                errG = criterion(output, y_mb_real)
                gen_loss.append(errG.data[0])
                errG.backward(retain_graph=True)
                optimizerG.step()
            
            # Printing loss
            print('Epoch ' + str(i+1) + ', minibatch ' + str(p+1) + ' of '  +  str(len(mb_list)) + ' -- Dis_loss: ' + str(errD.data[0]) + ', Gen_loss: ' + str(errG.data[0]))
</code></pre>
<p>Here is my LSGAN model:</p>
<pre><code class="lang-auto"># FINAL MODEL TO RUN
# ------------------


def model_lsgan(x,epochs,mbsize,gen_net,dis_net,gen_lr_rate,dis_lr_rate,gn_mul,no_latent):

    #optimizer = torch.optim.Adam(net.parameters(), lr=lr_rate)
    #optimizer = torch.optim.RMSprop(net.parameters(), lr=lr_rate)
    #optimizer = torch.optim.SGD(net.parameters(), lr=lr_rate, momentum=0.9)
    optimizerG = torch.optim.Adam(gen_net.parameters(), lr=gen_lr_rate)
    optimizerD = torch.optim.Adam(dis_net.parameters(), lr=dis_lr_rate)
    
    m = x.shape[0]
    #initial_noise = Variable(torch.randn(m,no_latent)).cuda()
    
    global gen_loss
    gen_loss = []    
    global dis_loss
    dis_loss = []
    
    # Setting up minibatch features
    m = x.size()[0]
    mb_list = []
    mb_list = list(range(int(m/mbsize)))
    if m % mbsize == 0: # if the minibatches can be split up perfectly.
        'do nothing'
    else:
        mb_list.append(mb_list[len(mb_list)-1] + 1)
        
    
    for i in range(epochs): 
        for p in mb_list:
            
            # Mini batch operations
            start_index = p*mbsize
            end_index = m if p == mb_list[len(mb_list)-1] else p*mbsize + mbsize

            
            # 1. Training the discriminator - Spot real vs fake
            # -------------------------------------------------
            # 1.1 real data
            # -------------
            X_mb = x[start_index:end_index]
            m_curr = end_index - start_index
            y_mb_real = Variable(torch.Tensor(m_curr).fill_(1).float()) # real label = 1
            y_mb_real = y_mb_real.cuda()
            dis_net.zero_grad()
            output = dis_net(X_mb)
            output = output.view(-1,)
            errD_real = torch.mean((output-1)**2)

            
            # 1.2 fake data
            # -------------
            noise = Variable(torch.randn(m_curr,no_latent)).cuda()
            fake_data = gen_net(noise)
            y_mb_fake = Variable(torch.Tensor(m_curr).fill_(0).float()) # fake label = 0
            y_mb_fake = y_mb_fake.cuda()
            output = dis_net(fake_data.detach())
            output = output.view(-1,)
            errD_fake = torch.mean(output**2)
            
            # Updating D network
            errD = 0.5 * (errD_real + errD_fake)
            errD.backward()
            dis_loss.append(errD.data[0])
            optimizerD.step()

            
            # 2. Training the generator - making it generate more realistic images
            # --------------------------------------------------------------------
            
            for g in range(gn_mul):
                gen_net.zero_grad()
                output = dis_net(fake_data)
                output = output.view(-1,)
                errG = 0.5 * torch.mean((output-1)**2)
                gen_loss.append(errG.data[0])
                errG.backward(retain_graph=True)
                optimizerG.step()
            
            # Printing loss
            print('Epoch ' + str(i+1) + ', minibatch ' + str(p+1) + ' of '  +  str(len(mb_list)) + ' -- Dis_loss: ' + str(errD.data[0]) + ', Gen_loss: ' + str(errG.data[0]))
            
</code></pre>
<p>Even after 5000 epochs, the generator does not converge. Here’s a snapshot of generator and discriminator losses.</p>
<blockquote>
<p>Generator Loss:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/1/1d468e7dc89cb567a0c6c9d5b34b5b0db24849ea.png" alt="genloss" data-base62-sha1="4aZ0PNGSpo3kvtbUrqMNwUfFX3s" width="401" height="278"></p>
</blockquote>
<blockquote>
<p>Discriminator loss:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/8/825d9821039ba02894189fede1565052f59fbbad.png" alt="disloss" data-base62-sha1="iBgAWIXcvq8ZMvc8afxuNj7QwqN" width="410" height="278"></p>
</blockquote>
<p>Results sampling code:</p>
<pre><code class="lang-auto"># GAN Sampling
# ------------
gen_model = gen_model.eval()
dis_model = dis_model.eval()
h,w,c = 96,72,3
gen_image = gen_model(fixed_noise[0:10].cuda()).view(-1,c,h,w)
gen_image = gen_image.cpu().data.numpy()
gen_image = np.swapaxes(gen_image,1,3)
gen_image = np.swapaxes(gen_image,1,2)
</code></pre>
<blockquote>
<p>All results are the same as the below yellow blob:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/6/62aeacc402b2dfad36960108c35d6472f1d8471f.png" alt="download" data-base62-sha1="e4Z1Wet2PVM7D2nuEnbaKOlGm6j" width="200" height="252"></p>
</blockquote>
<p>I know that my training set is small but I also know that it is possible to train a GAN on a training set of similar size (reference : Pokemon GAN at this <a href="https://github.com/llSourcell/Pokemon_GAN" rel="noopener nofollow ugc">link</a>. Apparently a WGAN trained on a dataset of around 150 pokemons)</p>
<p>I’m new to GANS and my future work relies heavily on GANs. It seems like I’m missing something fundamental. Would greatly appreciate any help in here please.</p>

username: zikai
cooked: <p>I think if the gradient vanishing?</p>

username: zikai
cooked: <p>Sigmoid function will cause gradient vanishing.</p>

username: TigerMan
cooked: <p>I think Goodfellow’s reply is the answer that you looking for<br>
<a href="http://qr.ae/TUTmq1" class="onebox" target="_blank" rel="nofollow noopener">http://qr.ae/TUTmq1</a></p>

username: Neel_Chaudhary
cooked: <p>Hey buddy, mine have same problem in epochs 1000 and mini_batch=128 with mnist but i change epochs to 4000 and mini_batch to 32. so decrease mini-batch and increase epochs probably work i think.</p>