CNN model does not update weights

SynthexX: <p>Hello,<br>
I implemented a CNN for CIFAR-10 image classification, but my model does not seem to update its weights. I already looked into several threads discussing the same symptoms, but my model seems to have a different problem.</p>
<p><strong>Model</strong></p>
<pre><code class="lang-auto">class CNNModel(ImageClassificationBase):
    def __init__(self):
        super().__init__()
        self.network = nn.Sequential(
            nn.Conv2d(3, 32, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2, 2), # output: 64 x 16 x 16

            nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2, 2), # output: 128 x 8 x 8

            nn.Conv2d(128, 256, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2, 2), # output: 256 x 4 x 4

            nn.Flatten(), 
            nn.Linear(256*4*4, 1024),
            nn.ReLU(),
            nn.Linear(1024, 512),
            nn.ReLU(),
            nn.Linear(512, 128),
            nn.ReLU(),
            nn.Linear(128, 10))
        
    def forward(self, x):
        return self.network(x)
</code></pre>
<p>The model inherits <strong>ImageClassificationBase</strong></p>
<pre><code class="lang-auto">class ImageClassificationBase(nn.Module):
    def training_step(self, batch, criterion):
        images, labels = batch
        images = images.to(device)
        labels = labels.to(device)
        out = self(images)              # Generate predictions
        loss = criterion(out, labels)   # Calculate loss
        return loss
    
    def validation_step(self, batch, criterion):
        images, labels = batch
        images = images.to(device)
        labels = labels.to(device)
        out = self(images)                    # Generate predictions
        loss = criterion(out, labels)         # Calculate loss
        acc = accuracy(out, labels)           # Calculate accuracy
        return {'val_loss': loss.detach(), 'val_acc': acc}
        
    def validation_epoch_end(self, outputs):
        batch_losses = [x['val_loss'] for x in outputs]
        epoch_loss = torch.stack(batch_losses).mean()   # Combine losses
        batch_accs = [x['val_acc'] for x in outputs]
        epoch_acc = torch.stack(batch_accs).mean()      # Combine accuracies
        return {'val_loss': epoch_loss.item(), 'val_acc': epoch_acc.item()}
    
    def epoch_end(self, epoch, result):
        print("Epoch [{}], train_loss: {:.4f}, val_loss: {:.4f}, val_acc: {:.4f}".format(
            epoch+1, result['train_loss'], result['val_loss'], result['val_acc']))
</code></pre>
<p><strong>Training Loop</strong></p>
<pre><code class="lang-auto">def fit(epochs, model, train_loader, val_loader, optimizer, criterion):
    history = []
    for epoch in range(epochs):
        # Training phase 
        model.train()
        train_losses = []
        for batch in train_loader:
            loss = model.training_step(batch, criterion)
            train_losses.append(loss)

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
        # Validation phase
        result = evaluate(model, val_loader, criterion)
        result['train_loss'] = torch.stack(train_losses).mean().item()
        model.epoch_end(epoch, result)
        history.append(result)
    return history
</code></pre>
<p>Further code used for training</p>
<pre><code class="lang-auto">def accuracy(output, labels):
    _, predictions = torch.max(output, dim=1)
    return torch.tensor(torch.sum(predictions == labels).item() / len(predictions))

def evaluate(model, val_loader, criterion):
    model.eval()
    outputs = [model.validation_step(batch, criterion) for batch in val_loader]
    return model.validation_epoch_end(outputs)
</code></pre>
<p>I use CrossEntropyLoss as criterion</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
</code></pre>
<p>and Adam as optimizer</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate, weight_decay=weight_decay, eps=eps)
</code></pre>
<p>In some other thread someone said to look at</p>
<pre><code class="lang-auto">list(model.parameters())[0].grad
</code></pre>
<p>which returns arrays completly filled with zeros.</p>
<p>Accuracy and loss stays the same over a couple of epochs (the tensor-list is the model prediction for individual images).<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d62f97c9df9f32dd809f44d7e235afac28dbf049.png" data-download-href="https://discuss.pytorch.org/uploads/default/d62f97c9df9f32dd809f44d7e235afac28dbf049" title="Screenshot 2023-01-11 at 16.15.23"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d62f97c9df9f32dd809f44d7e235afac28dbf049_2_690x140.png" alt="Screenshot 2023-01-11 at 16.15.23" data-base62-sha1="uyM93oMnUWTVMcRQAn9d2vQ0h2F" width="690" height="140" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d62f97c9df9f32dd809f44d7e235afac28dbf049_2_690x140.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d62f97c9df9f32dd809f44d7e235afac28dbf049_2_1035x210.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d62f97c9df9f32dd809f44d7e235afac28dbf049_2_1380x280.png 2x" data-dominant-color="494949"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-01-11 at 16.15.23</span><span class="informations">2790×568 114 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In some other threads the problem was a combination of Softmax and CrossEntropyLoss - which is not the problem in my case as i am using ReLU as last activation layer.<br>
I already tried SGD as well, both optimizers with different learning rates (0.001-0.0001). Nothing worked.</p>
<p>I really have no idea where the problem is…<br>
Thanks for the help!</p>

eqy: <p>Could you post how you are creating the optimizer and also the result of loss for each batch in the loader?</p>

SynthexX: <p>I initialize the optimizier as follows:</p>
<pre><code class="lang-auto">model = CNNModel().to(device)
optimizer = initialize_optimizer(model, "adam")

def initialize_optimizer(model, opt_func):
    if opt_func == 'adam':
        return torch.optim.Adam(model.parameters(), lr=learning_rate, weight_decay=weight_decay, eps=eps)
    elif opt_func == 'adagrad':  
        return torch.optim.Adagrad(model.parameters(), lr=learning_rate, lr_decay=lr_decay, weight_decay=weight_decay, eps=eps)
    elif opt_func == 'sgd':
        return torch.optim.SGD(model.parameters(), lr=learning_rate, momentum=momentum, weight_decay=weight_decay)
    else:
        print("Error! Specified optimizer has not been implemented.")
</code></pre>
<p>Here are the losses for each batch for the third epoch (I left some out since the loss is pretty much the same for every batch for every epoch.</p>
<pre><code class="lang-auto">Batch Loss: tensor(2.3031, device='cuda:0', grad_fn=&lt;NllLossBackward0&gt;)
Batch Loss: tensor(2.3029, device='cuda:0', grad_fn=&lt;NllLossBackward0&gt;)
Batch Loss: tensor(2.3038, device='cuda:0', grad_fn=&lt;NllLossBackward0&gt;)
Batch Loss: tensor(2.3028, device='cuda:0', grad_fn=&lt;NllLossBackward0&gt;)
....
Batch Loss: tensor(2.3024, device='cuda:0', grad_fn=&lt;NllLossBackward0&gt;)
Batch Loss: tensor(2.3024, device='cuda:0', grad_fn=&lt;NllLossBackward0&gt;)
Batch Loss: tensor(2.3040, device='cuda:0', grad_fn=&lt;NllLossBackward0&gt;)
Batch Loss: tensor(2.3011, device='cuda:0', grad_fn=&lt;NllLossBackward0&gt;)
Batch Loss: tensor(2.3009, device='cuda:0', grad_fn=&lt;NllLossBackward0&gt;)
</code></pre>

eqy: <p>I would sanity check something like <code>sum([p.sum() for p in model.parameters()]</code> after each iteration. If this isn’t changing, it suggests that the gradient updates aren’t being applied. If it is changing, then it could be that the model is being repeatedly re-initialized somehow.</p>

SynthexX: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a43eb115ae5614a6112f35911517e5cd83fe6bc0.png" data-download-href="https://discuss.pytorch.org/uploads/default/a43eb115ae5614a6112f35911517e5cd83fe6bc0" title="Screenshot 2023-01-12 at 09.47.12"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a43eb115ae5614a6112f35911517e5cd83fe6bc0.png" alt="Screenshot 2023-01-12 at 09.47.12" data-base62-sha1="nqYCxm7LK3rIAV8Oy2zzeZuLiI8" width="690" height="252" data-dominant-color="4F4F4F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-01-12 at 09.47.12</span><span class="informations">1254×458 67.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Model parameters seem to change; going towards zero. Validation accuracy and training loss stay the same though.</p>
<p>I don’t think that the model is repeatedly re-initialized though. This is exactly how I initiate training:</p>
<pre><code class="lang-auto"># Initialize dataloaders
train_dl = DataLoader(train_ds, batch_size_target, num_workers=2, shuffle=True, pin_memory=True)
val_dl = DataLoader(val_ds, batch_size_target, num_workers=2, pin_memory=True)

# Initialize model
model = CNNModel().to(device)

# Initialize optimizer
optimizer = initialize_optimizer(model, opt_func)
criterion = nn.CrossEntropyLoss()

_ = fit(num_epochs, model, train_dl, val_dl, optimizer, criterion)
</code></pre>
<p>I recoded <code>initialize_optimizer</code> to instanciate the optimizer first before returning the optimizer object, but it did not help as well.</p>

SynthexX: <p>I found the problem! The reason for it seemed to be the combination of settings for <code>weight_decay=0.005</code> and <code>learning_rate=0.0005</code>. When I set <code>weight_decay=0</code> the model started to learn.</p>
<p>Still, I have to say that I find it quite strange. I always thought weight_decay and learning_rate are independent of each other - accordingly I thought that weight_decay can also be greater than learning_rate. Strange.</p>
<p>Thanks for your help though!</p>