Model weights apparently not being modified

Ghostv: <p>Hi!<br>
I’ve been using pytorch for a couple weeks, so i’m not sure if i’m doing something wrong, but it seems the model i’m building is not updating the weights in every pass.</p>
<p>My model is as follows:</p>
<pre><code class="lang-auto">class NN(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.m0 = torch.nn.Sequential(
        torch.nn.Conv1d(402, 1, ks[0], stride=st),
        torch.nn.ReLU(True))
        self.m1 = torch.nn.Sequential(
        torch.nn.Conv1d(402, 1, ks[1], stride=st),
        torch.nn.ReLU(True))
        self.m2 = torch.nn.Sequential(
        torch.nn.Conv1d(402, 1, ks[2], stride=st),
        torch.nn.ReLU(True))
        self.m3 = torch.nn.Sequential(
        torch.nn.Conv1d(402, 1, ks[3], stride=st),
        torch.nn.ReLU(True))
        self.m4 = torch.nn.Sequential(
        torch.nn.Conv1d(402, 1, ks[4], stride=st),
        torch.nn.ReLU(True))
        self.m5 = torch.nn.Sequential(
        torch.nn.Conv1d(402, 1, ks[5], stride=st),
        torch.nn.ReLU(True))

        self.m0[0].weight.data = torch.ones(1,1,ks[0])*0.1
        self.m1[0].weight.data = torch.ones(1,1,ks[1])*0.1
        self.m2[0].weight.data = torch.ones(1,1,ks[2])*0.1
        self.m3[0].weight.data = torch.ones(1,1,ks[3])*0.1
        self.m4[0].weight.data = torch.ones(1,1,ks[4])*0.1
        self.m5[0].weight.data = torch.ones(1,1,ks[5])*0.1

        self.dense = torch.nn.Linear(11124,102,bias=False)
        self.softmax = torch.nn.Softmax(dim=0)

    def forward(self,x):
        y0 = self.m0(x)
        y1 = self.m1(x)
        y2 = self.m2(x)
        y3 = self.m3(x)
        y4 = self.m4(x)
        y5 = self.m5(x)
    
        y = torch.cat((y0,y1,y2,y3,y4,y5),dim=2)

        yf = torch.relu(self.dense(y))
        ys = self.softmax(yf)

        return ys
</code></pre>
<p>Which is basically 6 different conv layers with different kernel sizes applied in parallel, the results then get concatenated and passed through a dense layer to fit the labels (402x1 numerical categories or 402*102 one-hot encoded categories)</p>
<p>And my main loop is:</p>
<pre><code class="lang-auto">criterion = torch.nn.CrossEntropyLoss().cuda()
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)
for epoch in range(0,10):
    optimizer.zero_grad()
    outputs = model(xt)
    loss = criterion(outputs, yt)
    loss.backward()
    optimizer.step()
    print('epoch {}, loss {}'.format(epoch, loss.item()))
    print(outputs[1:5,0,0])
</code></pre>
<p>The input data is a matrix with hyperspectral values for a set of frequencies and the labels are categories for the different curve shapes (i’ve tried using it as numerical categories and one-hot encoder both with the same result).</p>
<p>Now, when i run it, i just get:</p>
<pre><code class="lang-auto">epoch 0, loss -0.0
tensor([0.0025, 0.0025, 0.0025, 0.0025], device='cuda:0',grad_fn=&lt;SelectBackward0&gt;)
epoch 1, loss -0.0
tensor([0.0025, 0.0025, 0.0025, 0.0025], device='cuda:0',grad_fn=&lt;SelectBackward0&gt;)
epoch 2, loss -0.0
tensor([0.0025, 0.0025, 0.0025, 0.0025], device='cuda:0',grad_fn=&lt;SelectBackward0&gt;)
epoch 3, loss -0.0
tensor([0.0025, 0.0025, 0.0025, 0.0025], device='cuda:0',grad_fn=&lt;SelectBackward0&gt;)
epoch 4, loss -0.0
tensor([0.0025, 0.0025, 0.0025, 0.0025], device='cuda:0',grad_fn=&lt;SelectBackward0&gt;)
epoch 5, loss -0.0
tensor([0.0025, 0.0025, 0.0025, 0.0025], device='cuda:0',grad_fn=&lt;SelectBackward0&gt;)
epoch 6, loss -0.0
tensor([0.0025, 0.0025, 0.0025, 0.0025], device='cuda:0',grad_fn=&lt;SelectBackward0&gt;)
epoch 7, loss -0.0
tensor([0.0025, 0.0025, 0.0025, 0.0025], device='cuda:0',grad_fn=&lt;SelectBackward0&gt;)
epoch 8, loss -0.0
tensor([0.0025, 0.0025, 0.0025, 0.0025], device='cuda:0',grad_fn=&lt;SelectBackward0&gt;)
epoch 9, loss -0.0
tensor([0.0025, 0.0025, 0.0025, 0.0025], device='cuda:0',grad_fn=&lt;SelectBackward0&gt;)
</code></pre>
<p>Which apparently is not updating the weights and just keeping the random init ones (doesn’t matter which timestep i use). I’m not sure if i’m missing something, or if i’m doing something wrong, as i said, i’ve been using pytorch for barely two weeks. Any ideas?</p>
<p>Thanks in advance!</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> expects raw logits so remove the <code>self.softmax</code> from your model. Once this is done, also remove the <code>torch.relu</code> and return the logits created by <code>self.dense(y)</code> directly.</p>

Ghostv: <p>I’m gonna try that and report back, thanks!.</p>
<p>I saw another post of a dude with exactly the same issue and he solved it by using a sigmoid instead of a softmax and a MSELoss for the loss fnx, i tried that and worked too. I’m gonna test your solution.</p>
<p>EDIT: Removing the softmax and the relu for the final output and using only the dense does the same, now updating weights.</p>

Ghostv: <p>I’m not sure if it could also affect, but pytorch didn’t let me use weights with dim &lt; 3, so i had to reshape my input from (n,w) to (n,1,w) since there is only one channel (same for the labels).</p>

ptrblck: <p>Could you describe which weights are used and were failing?<br>
Just want to make sure your shapes are correct and e.g. the loss calculation is correct.</p>

Ghostv: <p>Sorry, i didn’t explain myself correcly!</p>
<p>My data is 402 observations, each one made of 2151 features with length 1 (only one channel). I was a bit confused on how to use conv1d, so, i tried to use the input as a (402,2151) tensor and code all the NN accordingly (using an (1,n) conv. kernel), but i got the error that the kernel must be 3 dimensions at least, so i reshaped the input tensor to (402,1,2151) and the same with the labels. I wasn’t sure if that may be the issue, maybe when there is only one channel the reshape must be done differently, etc.</p>
<p>I tried your solution, but i get similar results, the weights are not updating with each step.</p>
<p>About the same, for categorical labels, i’m using a one-hot encoder with shape (402,1,102), for the 102 classes i have, but i think i read that pytorch can work with numerical categories also, is that correct?</p>
<p>I checked this post: <a href="https://discuss.pytorch.org/t/model-weights-not-being-updated/1842" class="inline-onebox">Model weights not being updated</a></p>
<p>And the solution posted there, changing the losses to MSEloss and the softmax to sigmoid worked for me, by doing that, the weights update on each step, but i’m not sure (based on what you wrote), if what i’m doing is correct.</p>

ptrblck: <aside class="quote no-group" data-username="Ghostv" data-post="6" data-topic="200997">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/7ea924/48.png" class="avatar"> Ghostv:</div>
<blockquote>
<p>but i got the error that the kernel must be 3 dimensions at least, so i reshaped the input tensor to (402,1,2151) and the same with the labels. I wasn’t sure if that may be the issue, maybe when there is only one channel the reshape must be done differently, etc.</p>
</blockquote>
</aside>
<p><code>nn.Conv1d</code> expects an input in the shape <code>[batch_size, channels, seq_len]</code> and will add a batch dimension with the size of 1 if it’s missing. Unsqueezing the tensor and adding the channel dimension might be the right approach depending on your use case.</p>
<aside class="quote no-group" data-username="Ghostv" data-post="6" data-topic="200997">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/7ea924/48.png" class="avatar"> Ghostv:</div>
<blockquote>
<p>I tried your solution, but i get similar results, the weights are not updating with each step.</p>
</blockquote>
</aside>
<p>I assume the weights receive valid gradients but the updates are just small or did you verify that no gradients are received at all? The latter case would not make sense since it seems other loss functions work.</p>
<aside class="quote no-group" data-username="Ghostv" data-post="6" data-topic="200997">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/7ea924/48.png" class="avatar"> Ghostv:</div>
<blockquote>
<p>About the same, for categorical labels, i’m using a one-hot encoder with shape (402,1,102), for the 102 classes i have, but i think i read that pytorch can work with numerical categories also, is that correct?</p>
</blockquote>
</aside>
<p>By default <code>nn.CrossEntropyLoss</code> expects class indices, but in newer PyTorch versions also accepts “soft” targets, which can be a one-hot encoded tensor (although it would be wasteful compared to class indices since you would store a lot of zeroes without taking the advantage of using soft targets, i.e. <code>probabilities != 0/1</code>).</p>
<aside class="quote no-group" data-username="Ghostv" data-post="6" data-topic="200997">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/7ea924/48.png" class="avatar"> Ghostv:</div>
<blockquote>
<p>And the solution posted there, changing the losses to MSEloss and the softmax to sigmoid worked for me, by doing that, the weights update on each step, but i’m not sure (based on what you wrote), if what i’m doing is correct.</p>
</blockquote>
</aside>
<p>Different loss functions could work, but <code>nn.CrossEntropyLoss</code> is commonly used for multi-class classification use cases.</p>

Ghostv: <p>How can i verify that the gradients are being received? I was comparing the model-parameters() and they were the same before-after the optimizer.step.</p>

ptrblck: <p>You could either compare <strong>copies</strong> of the parameters before and after the <code>optimizer.step()</code> call or you could also check the <code>.grad</code> attributes of all trainable parameters after the <code>backward</code> call to confirm tat gradients are indeed computed.</p>

Ghostv: <p>Just added in the main loop:</p>
<pre><code class="lang-auto">    a = list(model.parameters())[0].clone()
    optimizer.step()
    b = list(model.parameters())[0].clone()
    print(torch.equal(a.data,b.data))
</code></pre>
<p>And the output is True in every iteration. That’s following your suggestion (removing the relu and just using the output of the dense as the forward-pass output) and using a CrossEntropy loss.</p>
<p>Edit: And i just checked the grad in the last step and it’s all 0. The same happens regardless of the learning rate.</p>

ptrblck: <p>In this case you could start by checking your loss and making sure it’s <code>!=0</code>.</p>

Ghostv: <p>Yep, the loss is indeed 0, didn’t think of that…</p>
<pre><code class="lang-auto">tensor(-0., device='cuda:0', grad_fn=&lt;DivBackward1&gt;)
</code></pre>
<p>What could be causing it? Could it be maybe some issue with the data/label format?</p>
<p>My data -and its shape- is as follows:</p>
<pre><code class="lang-auto">tensor([[[0.3310, 0.3310, 0.3330,  ..., 0.6970, 0.6970, 0.6960]],
        [[0.2010, 0.2010, 0.2010,  ..., 0.5790, 0.5800, 0.5810]],
        [[0.1000, 0.0980, 0.0980,  ..., 0.3560, 0.3560, 0.3570]],
        ...,
        [[0.3040, 0.3110, 0.3120,  ..., 0.4890, 0.4950, 0.4990]],
        [[0.1830, 0.1920, 0.1980,  ..., 0.2700, 0.2740, 0.2770]],
        [[0.2120, 0.2250, 0.2450,  ..., 0.1970, 0.1980, 0.2000]]],
       device='cuda:0')

torch.Size([402, 1, 2151])
</code></pre>
<p>And my labels (i switched them to numerical):</p>
<pre><code class="lang-auto">tensor([[[  0.]],
        [[  0.]],
        [[  0.]],
        [[  0.]],
        [[  0.]],
        [[  0.]],
        [[  0.]],
        [[  0.]],
        [[  0.]],
        [[  0.]],
        [[  1.]],
        [[  1.]],
        [[  1.]],
        ...,
torch.Size([402, 1, 1])
</code></pre>
<p>Using them as-this or with a one hot encoder yields the same result.</p>
<p>Do i have to specify something when i declare the losses and/or the optimizer?<br>
I’m initializing them like this:</p>
<pre><code class="lang-auto">criterion = torch.nn.CrossEntropyLoss().cuda()
optimizer = torch.optim.Adam(model.parameters(), lr=0.00001)
</code></pre>
<p>Maybe i’m missing something <img src="https://discuss.pytorch.org/images/emoji/apple/worried.png?v=12" title=":worried:" class="emoji" alt=":worried:" loading="lazy" width="20" height="20"></p>
<p>EDIT: I just checked and the forward pass is indeed generating a result, so, the issue seems to be the loss.</p>

ptrblck: <p>Based on the shape of the model output and target you are working with a single class only, so all samples belong to class0 and the network cannot be wrong. <code>nn.CrossEntropyLoss</code> expects a model output in the shape <code>[batch_size, nb_classes, *]</code> where <code>*</code> denotes additional dimensions. In your case <code>nb_classes=1</code> which would explain the issue.</p>

Ghostv: <p>Could you explain that to me again?</p>
<p>I was just using one-hot encoder which was (402,1,102), hence, each label was a [0,0,0,0,… …,1,0,0] array with the var hot-encoded, i just changed it to numerical cat because of what we were talking, being now (402,1,1).</p>
<p>How could i exactly fix this?</p>

Ghostv: <p>This is the label shape i was using up to yesterday, with the same results, i just changed it to numerical cats. I understood it was the same.</p>
<pre><code class="lang-auto">tensor([[[1., 0., 0.,  ..., 0., 0., 0.]],
        [[1., 0., 0.,  ..., 0., 0., 0.]],
        [[1., 0., 0.,  ..., 0., 0., 0.]],
        ...,
        [[0., 0., 0.,  ..., 0., 0., 1.]],
        [[0., 0., 0.,  ..., 0., 0., 1.]],
        [[0., 0., 0.,  ..., 0., 0., 1.]]], device='cuda:0')
&gt;&gt;&gt; dim(yt)
torch.Size([402, 1, 102])
</code></pre>
<p>Could you please expand a little on what i’m doing wrong?</p>
<p>My classes are 102, how can i reshape that (if i’m using numerical cats) into a (402,102,1) array or into a (402,102,102) array with one-hot?</p>

ptrblck: <p>I don’t know where these additional dimensions come from but this code shows the expected shapes for a muli-class classification use case:</p>
<pre data-code-wrap="python"><code class="lang-python">batch_size = 402
nb_classes = 102

criterion = nn.CrossEntropyLoss()

output = torch.randn(batch_size, nb_classes, requires_grad=True)
# as class indices
target = torch.randint(0, nb_classes, (batch_size,))

loss = criterion(output, target)
print(loss)
# tensor(5.2173, grad_fn=&lt;NllLossBackward0&gt;)

# as soft targets
target = F.one_hot(target, 102)
target = target.float()
print(target.shape)
# torch.Size([402, 102])

loss = criterion(output, target)
print(loss)
# tensor(5.2173, grad_fn=&lt;DivBackward1&gt;)
</code></pre>

Ghostv: <p>The problem is that my dense layer produces a (402,1,102) output, not (402,102). I think my issue is in the Conv1d args.</p>
<p>This was trivial to do in tensorflow, but somehow here i’m not sure if what i’m doing is correct or not. I wanna apply a standard convolution to a nx1 array, that’s what my conv1d layers aim to be (varying the kernel size en each case), am i doing it right?</p>
<p>My base input is a 402 (observations/rows/entries/batch_size) x 2151 (features) with only one channel per observation. I wanna apply n different convolutions to those 2151 observations, how would that be using Conv1d in pytorch? (Let’s say… one with a conv kernel… [0.1,0.1,0.1,0.1])</p>

Ghostv: <p>I’m totally lost with this, i really have no clue what i’m doing and the more i search for input shapes of every layer i’m using, the more contradicting information i find. I’ve tried a million different ways but nothing seems to work.</p>

Ghostv: <p>I think i solved it… i just added a reshape((402,102)) to the final output of the NN (which was (402,1,102)) and now it’s working <em>apparently</em> properly.</p>

Arun_Sandy: <p>Hello,<br>
I’m working on Conditional GAN to generate medical images and for testing purpose I ran the code on an resized image of 128x128 for 7500 epoch and this is working fine.  But now when I change the code to train on a resized image of 512x512, The noise image generated on the first epoch is not being updated. I have no idea what’s going wrong.</p>
<p><strong>Code for 128x128 image feed:</strong></p>
<blockquote>
<p><strong>1. Loading the data :</strong><br>
train_transform = transforms.Compose([<br>
transforms.Resize((128,128)),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.5,0.5,0.5], [0.5,0.5,0.5])])<br>
train_dataset = datasets.ImageFolder(root=‘data/KB_Images/’, transform=train_transform)<br>
train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=2, shuffle=True)</p>
</blockquote>
<p><strong>2. GAN Architecture :</strong><br>
def weights_init(m):<br>
classname = m.<strong>class</strong>.<strong>name</strong><br>
if classname.find(‘Conv’) != -1:<br>
torch.nn.init.normal_(m.weight, 0.0, 0.02)<br>
elif classname.find(‘BatchNorm’) != -1:<br>
torch.nn.init.normal_(m.weight, 1.0, 0.02)<br>
torch.nn.init.zeros_(m.bias)</p>
<p>class Generator(nn.Module):</p>
<pre><code>def __init__(self):
    super(Generator, self).__init__()
    self.label_conditioned_generator =nn.Sequential(nn.Embedding(3, 100),
                  nn.Linear(100, 16))
    self.latent =nn.Sequential(nn.Linear(100, 4*4*512),
                               nn.LeakyReLU(0.2, inplace=True))
    self.model =nn.Sequential(nn.ConvTranspose2d(513, 64*8, 4, 2, 1, bias=False),
                  nn.BatchNorm2d(64*8, momentum=0.1,  eps=0.8),
                  nn.ReLU(True),
                  nn.ConvTranspose2d(64*8, 64*4, 4, 2, 1,bias=False),
                  nn.BatchNorm2d(64*4, momentum=0.1,  eps=0.8),
                  nn.ReLU(True), 
                  nn.ConvTranspose2d(64*4, 64*2, 4, 2, 1,bias=False),
                  nn.BatchNorm2d(64*2, momentum=0.1,  eps=0.8),
                  nn.ReLU(True), 
                  nn.ConvTranspose2d(64*2, 64*1, 4, 2, 1,bias=False),
                  nn.BatchNorm2d(64*1, momentum=0.1,  eps=0.8),
                  nn.ReLU(True), 
                  nn.ConvTranspose2d(64*1, 3, 4, 2, 1, bias=False),
                  nn.Tanh())
def forward(self, inputs):
    noise_vector, label = inputs
    label_output = self.label_conditioned_generator(label)
    label_output = label_output.view(-1, 1, 4, 4)
    latent_output = self.latent(noise_vector)
    latent_output = latent_output.view(-1, 512,4,4)
    concat = torch.cat((latent_output, label_output), dim=1)
    image = self.model(concat)
    #print(image.size())
    return image
</code></pre>
<p>class Discriminator(nn.Module):</p>
<pre><code>def __init__(self):
    super(Discriminator, self).__init__()

    self.label_condition_disc = nn.Sequential(nn.Embedding(3, 100),
                  nn.Linear(100, 3*128*128))       
    
    self.model = nn.Sequential(nn.Conv2d(3, 64, 4, 2, 1, bias=False),
                  nn.LeakyReLU(0.2, inplace=True),
                  nn.Conv2d(64, 64*2, 4, 3, 2, bias=False),
                  nn.BatchNorm2d(64*2, momentum=0.1,  eps=0.8),
                  nn.LeakyReLU(0.2, inplace=True),
                  nn.Conv2d(64*2, 64*4, 4, 3,2, bias=False),
                  nn.BatchNorm2d(64*4, momentum=0.1,  eps=0.8),
                  nn.LeakyReLU(0.2, inplace=True),
                  nn.Conv2d(64*4, 64*8, 4, 3, 2, bias=False),
                  nn.BatchNorm2d(64*8, momentum=0.1,  eps=0.8),
                  nn.LeakyReLU(0.2, inplace=True), 
                  nn.Flatten(),
                  nn.Dropout(0.4),
                  nn.Linear(9216, 1),
                  nn.Sigmoid()
                 )

def forward(self, inputs):
    img, label = inputs
    label_output = self.label_condition_disc(label)
    label_output = label_output.view(-1, 3, 128, 128)
    concat = torch.cat((img, label_output), dim=-1)
    #print(concat.size())
    output = self.model(concat)
    return output
</code></pre>
<p>def noise(n, n_features=128): # this used to be 128</p>
<pre><code>return Variable(torch.randn(n, n_features)).to(device)
</code></pre>
<p><strong>3. Optimizer and Losses :</strong><br>
device = ‘cuda’<br>
generator = Generator().to(device)<br>
generator.apply(weights_init)<br>
discriminator = Discriminator().to(device)<br>
discriminator.apply(weights_init)<br>
learning_rate = 0.0001<br>
G_optimizer = optim.Adam(generator.parameters(), lr = learning_rate, betas=(0.5, 0.999))<br>
D_optimizer = optim.Adam(discriminator.parameters(), lr = learning_rate, betas=(0.5, 0.999))<br>
images = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
test_noise = noise(100)<br>
adversarial_loss = nn.BCELoss()</p>
<p>def generator_loss(fake_output, label):</p>
<pre><code>gen_loss = adversarial_loss(fake_output, label)
#print(gen_loss)
return gen_loss
</code></pre>
<p>def discriminator_loss(output, label):</p>
<pre><code>disc_loss = adversarial_loss(output, label)
return disc_loss
</code></pre>
<p><strong>4. Training :</strong><br>
num_epochs = 7500<br>
for epoch in range(1, num_epochs+1):</p>
<pre><code>D_loss_list, G_loss_list = [], []
g_error, d_error=0.0, 0.0
print("Epoch :",epoch)
for index, (real_images, labels) in enumerate(train_loader):
    D_optimizer.zero_grad()
    real_images = real_images.to(device)
    labels = labels.to(device)
    labels = labels.unsqueeze(1).long()

   
    real_target = Variable(torch.ones(real_images.size(0), 1).to(device))
    fake_target = Variable(torch.zeros(real_images.size(0), 1).to(device))
   
    D_real_loss = discriminator_loss(discriminator((real_images, labels)), real_target)
    # print(discriminator(real_images))
    #D_real_loss.backward()
 
    noise_vector = torch.randn(real_images.size(0), 100, device=device)  
    noise_vector = noise_vector.to(device)
     
    
    generated_image = generator((noise_vector, labels))
    output = discriminator((generated_image.detach(), labels))
    D_fake_loss = discriminator_loss(output,  fake_target)

 
    # train with fake
    #D_fake_loss.backward()
   
    D_total_loss = (D_real_loss + D_fake_loss) / 2
    D_loss_list.append(D_total_loss)
   
    D_total_loss.backward()
    D_optimizer.step()

    # Train generator with real labels
    G_optimizer.zero_grad()
    G_loss = generator_loss(discriminator((generated_image, labels)), real_target)
    G_loss_list.append(G_loss)

    G_loss.backward()
    G_optimizer.step()
    
    g_error += G_loss
    d_error += D_total_loss
    
    if index%15 ==0 :
        vutils.save_image(real_images, '%s/real_samples.png' % "./results_epochs_7500", normalize = True)
        fake = generator((noise_vector,labels))
        vutils.save_image(fake.data, '%s/fake_samples_epoch_%03d.png' % ("./results_epochs_7500", epoch), normalize = True)
print('Epoch {}: g_loss: {:.8f} d_loss: {:.8f}\r'.format(epoch, g_error/index, d_error/index))

img = generator((noise_vector,labels)).cpu().detach()
# if epoch%100==0:
#     for i in range(img.shape[0]):
#         vutils.save_image(img[i], '%s/fake_samples_epoch_%03d_img_%01d.png' % ("./results", epoch,i), normalize = True)
img = make_grid(img)
images.append(img)
</code></pre>
<p>print(‘Training Finished’)<br>
torch.save(generator.state_dict(), ‘Conditional-GAN.pth’)</p>
<p>frames = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for i in range(1,len(images)+1):<br>
image = imageio.v2.imread(f’%s/fake_samples_epoch_%03d.png’% (“./results_epochs_7500”, i))<br>
frames.append(image[:500])<br>
print(f’%s/fake_samples_epoch_%03d.png’% (“./results_epochs_7500”, i))</p>
<p>imageio.mimsave(‘./progress.gif’, # output gif<br>
frames,          # array of input frames<br>
fps = 7)         # optional: frames per second</p>
<p><strong>Real Sample Used for Training:</strong><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a283bb4e8228f3a45af06ed16e12a8c9ea2c544b.png" alt="real_samples" data-base62-sha1="nbFAcVdvbZmgxrxuFmYjgOxAHCX" width="262" height="132"></p>
<p><strong>Epoch 1 Noise Image :</strong><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c7467a7f368140f3e8d3adca8152d2bcda50cca1.png" alt="fake_samples_epoch_001" data-base62-sha1="sqS0ZX7x29owCa8z3ZYaqTyjCIV" width="262" height="132"></p>
<p><strong>Epoch 6000 Noise Image :</strong><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74490dd55a4bea4919099736cd724c51ba7b286e.png" alt="fake_samples_epoch_6630" data-base62-sha1="gAHU3s3MBwml046kB8tpDKouTeu" width="262" height="132"></p>
<hr>
<hr>
<p><strong>Below is Code for 512x512 Image feed :</strong><br>
train_transform = transforms.Compose([<br>
transforms.Resize((512,512)),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.5,0.5,0.5], [0.5,0.5,0.5])])<br>
train_dataset = datasets.ImageFolder(root=‘data/KB_Images/’, transform=train_transform)<br>
train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=1, shuffle=True)</p>
<p><strong>3. Optimizer and Losses :</strong><br>
def weights_init(m):</p>
<pre><code>classname = m.__class__.__name__
if classname.find('Conv') != -1:
    torch.nn.init.normal_(m.weight, 0.0, 0.02)
elif classname.find('BatchNorm') != -1:
    torch.nn.init.normal_(m.weight, 1.0, 0.02)
    torch.nn.init.zeros_(m.bias)
</code></pre>
<p>class Generator(nn.Module):</p>
<pre><code>def __init__(self):
    super(Generator, self).__init__()

    self.label_conditioned_generator = nn.Sequential(
        nn.Embedding(3, 100),
        nn.Linear(100, 256))  # Changed from 16 to 256

    # Initial generator from latent space
    self.latent = nn.Sequential(
        nn.Linear(100, 16*16*512),
        nn.LeakyReLU(0.2, inplace=True))

    # Generator model definition
    self.model = nn.Sequential(
        nn.ConvTranspose2d(513, 256, 4, 2, 1, bias=False),
        nn.BatchNorm2d(256),
        nn.ReLU(True),
        nn.ConvTranspose2d(256, 128, 4, 2, 1, bias=False),
        nn.BatchNorm2d(128),
        nn.ReLU(True), 
        nn.ConvTranspose2d(128, 64, 4, 2, 1, bias=False),
        nn.BatchNorm2d(64),
        nn.ReLU(True), 
        nn.ConvTranspose2d(64, 32, 4, 2, 1, bias=False),
        nn.BatchNorm2d(32),
        nn.ReLU(True),
        nn.ConvTranspose2d(32, 3, 4, 2, 1, bias=False),
        nn.Tanh())

def forward(self, inputs):
    noise_vector, label = inputs
    label_output = self.label_conditioned_generator(label)
    label_output = label_output.view(-1, 1, 16, 16)  # Correctly reshaped now
    latent_output = self.latent(noise_vector)
    latent_output = latent_output.view(-1, 512, 16, 16)
    concat = torch.cat((latent_output, label_output), dim=1)
    image = self.model(concat)
    return image
</code></pre>
<p>class Discriminator(nn.Module):</p>
<pre><code>def __init__(self):
    super(Discriminator, self).__init__()
     
    self.label_condition_disc = nn.Sequential(nn.Embedding(3, 100),
                  nn.Linear(100, 3*512*512))  # Adjusted for 512x512 resolution     
    
    self.model = nn.Sequential(
        nn.Conv2d(6, 64, 4, 2, 1, bias=False),
        nn.LeakyReLU(0.2, inplace=True),
        nn.Conv2d(64, 128, 4, 2, 1, bias=False),
        nn.BatchNorm2d(128),
        nn.LeakyReLU(0.2, inplace=True),
        nn.Conv2d(128, 256, 4, 2, 1, bias=False),
        nn.BatchNorm2d(256),
        nn.LeakyReLU(0.2, inplace=True),
        nn.Conv2d(256, 512, 4, 2, 1, bias=False),
        nn.BatchNorm2d(512),
        nn.LeakyReLU(0.2, inplace=True),
        nn.Flatten(),
        nn.Dropout(0.4),
        nn.Linear(524288, 1),  # Updated to match the correct number of features
        nn.Sigmoid()
    )

def forward(self, inputs):
    img, label = inputs
    label_output = self.label_condition_disc(label)
    label_output = label_output.view(-1, 3, 512, 512)
    concat = torch.cat((img, label_output), dim=1)
    output = self.model(concat)
    return output
</code></pre>
<p>def noise(n, n_features=512): # this used to be 128<br>
return Variable(torch.randn(n, n_features)).to(device)</p>
<p>device = ‘cuda:1’<br>
generator = Generator().to(device)<br>
generator.apply(weights_init)<br>
discriminator = Discriminator().to(device)<br>
discriminator.apply(weights_init)<br>
learning_rate = 0.0002<br>
G_optimizer = optim.Adam(generator.parameters(), lr = learning_rate, betas=(0.5, 0.999))<br>
D_optimizer = optim.Adam(discriminator.parameters(), lr = learning_rate, betas=(0.5, 0.999))<br>
images = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
test_noise = noise(512) # This used ti be 100<br>
adversarial_loss = nn.BCELoss()</p>
<p>def generator_loss(fake_output, label):</p>
<pre><code>gen_loss = adversarial_loss(fake_output, label)
#print(gen_loss)
return gen_loss
</code></pre>
<p>def discriminator_loss(output, label):</p>
<pre><code>disc_loss = adversarial_loss(output, label)
return disc_loss
</code></pre>
<p><strong>4. Training :</strong><br>
num_epochs = 10000<br>
for epoch in range(1, num_epochs + 1):</p>
<pre><code>D_loss_list, G_loss_list = [], []
g_error, d_error=0.0, 0.0
print("Epoch :",epoch)
for index, (real_images, labels) in enumerate(train_loader):
    D_optimizer.zero_grad()
    real_images = real_images.to(device)
    labels = labels.to(device)
    labels = labels.unsqueeze(1).long()

    real_target = Variable(torch.ones(real_images.size(0), 1).to(device))
    fake_target = Variable(torch.zeros(real_images.size(0), 1).to(device))
   
    D_real_loss = discriminator_loss(discriminator((real_images, labels)), real_target)
    # print(discriminator(real_images))
    #D_real_loss.backward()
 
    noise_vector = torch.randn(real_images.size(0), 100, device=device)  
    noise_vector = noise_vector.to(device)
     
    
    generated_image = generator((noise_vector, labels))
    output = discriminator((generated_image.detach(), labels))
    D_fake_loss = discriminator_loss(output,  fake_target)

 
    # train with fake
    #D_fake_loss.backward()
   
    D_total_loss = (D_real_loss + D_fake_loss) / 2
    D_loss_list.append(D_total_loss)
   
    D_total_loss.backward()
    D_optimizer.step()

    # Train generator with real labels
    G_optimizer.zero_grad()
    G_loss = generator_loss(discriminator((generated_image, labels)), real_target)
    G_loss_list.append(G_loss)

    G_loss.backward()
    G_optimizer.step()
    
    g_error += G_loss
    d_error += D_total_loss
    
    if index%15 ==0 :
        vutils.save_image(real_images, '%s/real_samples.png' % "./results_epochs_10000_512x512", normalize = True)
        fake = generator((noise_vector,labels))
        vutils.save_image(fake.data, '%s/fake_samples_epoch_%03d.png' % ("./results_epochs_10000_512x512", epoch), normalize = True)
print('Epoch {}: g_loss: {:.8f} d_loss: {:.8f}\r'.format(epoch, g_error/index, d_error/index))

img = generator((noise_vector,labels)).cpu().detach()
if epoch%100==0:
    for i in range(img.shape[0]):
        vutils.save_image(img[i], '%s/fake_samples_epoch_%03d_img_%01d.png' % ("./results_epochs_10000_512x512", epoch,i), normalize = True)
img = make_grid(img)
images.append(img)
</code></pre>
<p>print(‘Training Finished’)<br>
torch.save(generator.state_dict(), ‘Conditional-GAN.pth’)</p>
<p>frames = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for i in range(1,len(images)+1):</p>
<pre><code>image = imageio.v2.imread(f'%s/fake_samples_epoch_%03d.png'% ("./results_epochs_10000_512x512", i))
frames.append(image[:500])
print(f'%s/fake_samples_epoch_%03d.png'% ("./results_epochs_10000_512x512", i))
</code></pre>
<p>imageio.mimsave(‘./progress.gif’, # output gif<br>
frames,          # array of input frames<br>
fps = 7)         # optional: frames per second</p>
<p><strong>Real Sample Used for Training:</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/0/901eb7206f4784c2b6fd5a80e36f237a10fff23b.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/901eb7206f4784c2b6fd5a80e36f237a10fff23b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/0/901eb7206f4784c2b6fd5a80e36f237a10fff23b.jpeg" alt="image" data-base62-sha1="kyWyUnANJxkSYE5kCX7fQlhPoDh" width="500" height="500" data-dominant-color="EAC5D6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">516×516 91.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Epoch 1 Noise Image :</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/3/23036ff9366124638faeea757db2cf9aac59e24d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/23036ff9366124638faeea757db2cf9aac59e24d" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/3/23036ff9366124638faeea757db2cf9aac59e24d.jpeg" alt="image" data-base62-sha1="4ZK4IlHAcY7wd5OPrF0UU2m8MQl" width="499" height="500" data-dominant-color="93898E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">515×516 66.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Epoch 2000 Noise Image :</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/a/ca37c2428e4d9fd442ed1538fc788904deff6496.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ca37c2428e4d9fd442ed1538fc788904deff6496" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/a/ca37c2428e4d9fd442ed1538fc788904deff6496.jpeg" alt="image" data-base62-sha1="sQTUwQ3fRexiJxq1KjhGDAcoYPc" width="499" height="500" data-dominant-color="958B90"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">513×514 66.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Below is the loss output of both Generator and Discriminator<br>
Epoch : 1<br>
Epoch 1: g_loss: 1.32907557 d_loss: 42.10420227<br>
Epoch : 2<br>
Epoch 2: g_loss: 0.00000000 d_loss: 50.18867874<br>
Epoch : 3<br>
Epoch 3: g_loss: 0.00000000 d_loss: 50.18867874<br>
Epoch : 4<br>
Epoch 4: g_loss: 0.00000000 d_loss: 50.18867874<br>
Epoch : 5<br>
Epoch 5: g_loss: 0.00000000 d_loss: 50.18867874<br>
Epoch : 6<br>
Epoch 6: g_loss: 0.00000000 d_loss: 50.18867874<br>
Epoch : 7<br>
Epoch 7: g_loss: 0.00000000 d_loss: 50.18867874</p>
<p>Both G and D are being constant for all the epoch.</p>
<p>Can you please let me know where make the changes so the model can be trained properly on a 512x512 image</p>