Model does not change during training

username: lifeblack
cooked: <p>Hi I implemented a Gan network. the discriminator does not change  during training… loss is BCEloss and the last layer  has a sigmoid activation function…<br>
list(discriminator.parameters())[0].grad is all zeros…</p>
<pre><code class="lang-auto">import torch.nn.functional as F
class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator,self).__init__()
        self.conv1= nn.Sequential(
            nn.Conv2d(2,128, kernel_size=(4,4), stride=2,padding=5),
            nn.BatchNorm2d(128),
            nn.LeakyReLU(0.2)
        )#32
        self.conv2= nn.Sequential(
            nn.Conv2d(128,256,kernel_size=(4,4), stride=2,padding=5),
            nn.BatchNorm2d(256),
            nn.LeakyReLU(0.2)
        )#16
        self.conv3= nn.Sequential(
            nn.Conv2d(256,1, kernel_size=(4,4), stride=2,padding=5),
            nn.Sigmoid()
        )#8
    def forward(self,kr):
        
        out_conv1=self.conv1(kr)
        out_conv2=self.conv2(out_conv1)
        out_conv3=self.conv3(out_conv2)        
        return out_conv3
</code></pre>
<p>inputsize=(batch,2,32,32)<br>
outputsize=(batch,1,8,8)<br>
loss=nn.BCELoss()</p>

username: Dwight_Foster
cooked: <p>Can you show your training code.</p>

username: lifeblack
cooked: <p>thanks for reply <a class="mention" href="/u/dwight_foster">@Dwight_Foster</a> ! Ofcourse…</p>
<pre><code class="lang-auto">generator = Generator()
discriminator = Discriminator()
loss_discriminator=nn.BCELoss()
discriminator_optimizer = torch.optim.Adam(discriminator.parameters(), lr=0.0002, betas=(0.5,0.999))
generator_optimizer = torch.optim.Adam(generator.parameters(), lr=0.0002, betas=(0.5,0.999))
for iteration,(x,r) in enumerate(Train_Loader):
        X_real,y_real=generate_real_samples(x.to(device))
        # 1. Train D on Real+Fake data
        discriminator.train()
        generator.eval()
        X_fake=generator(r.to(device))
        discriminator_optimizer.zero_grad()
        # 1.A Train D on Real data
        out_diss=discriminator(X_real)
        d_loss1=loss_discriminator(out_diss,y_real)
        d_loss1.backward()
        discriminator_optimizer.step()
        # 1.B Train D on Fake data
        d_loss2=loss_discriminator(discriminator(torch.cat([x.to(device),X_fake], dim=1)),torch.zeros(x.shape).to(device))
        d_loss2.backward()
        discriminator_optimizer.step()
        # 2. Train D on Real+Fake data
        #generator_optimizer.zero_grad()
        #discriminator.eval()
        #generator.train()
        #X_fake=generator(r.to(device))
        #loss_g=gen_loss(x.to(device),X_fake,discriminator(torch.cat([x.to(device),X_fake], dim=1)))
        #loss_g.backward()
        #generator_optimizer.step()
</code></pre>

username: Dwight_Foster
cooked: <p>One thing you could try is add d_loss1 and d_loss2 together and then use that loss for the backward step. Also it doesn’t seem like you run it for multiple epochs. You should do that if you want it to improve. If you do use epochs can you show the full code for that thanks.</p>

username: lifeblack
cooked: <p><a class="mention" href="/u/dwight_foster">@Dwight_Foster</a> thanks again… I added d_loss1 to d_loss2 as you said but it did not make any changes… yes I  run it for multiple epochs… just a simple ‘for i in rage(100):’ at fist</p>

username: Dwight_Foster
cooked: <p>Is the code you provided above in the epoch for loop or is some of it outside it because if you keep defining a new model every epoch that wouldn’t work.</p>

username: lifeblack
cooked: <p>I defined the model outside the for loop… I double checked the code… discriminator loss decreases during the training but when I print the weights they are constant… How this can happen?</p>

username: Dwight_Foster
cooked: <p>That is interesting. For this training loop are you using the generator. It could be that the generator is just getting worse so the discriminator is getting better without changing. It’s good that your loss is decreasing though. If your model trains than you shouldn’t have to worry about the weights not changing when you print them out. Can you just show how you are printing them out?</p>