Test accuracy remains the same even though loss is changing

Faruk_Celik: <p>Hello as you can see below, my model always has the same test accuracy, any kind of help would be nice.</p>
<pre><code class="lang-auto">Epoch: 1 | train_loss: 233082867353.8990 | train_acc: 0.1761 | test_loss: 1.7912 | test_acc: 0.1790
Epoch: 2 | train_loss: 1.7914 | train_acc: 0.1781 | test_loss: 1.7912 | test_acc: 0.1790
Epoch: 3 | train_loss: 1.7917 | train_acc: 0.1763 | test_loss: 1.7909 | test_acc: 0.1790
Epoch: 4 | train_loss: 1.7915 | train_acc: 0.1760 | test_loss: 1.7913 | test_acc: 0.1618
Epoch: 5 | train_loss: 1.7912 | train_acc: 0.1756 | test_loss: 1.7912 | test_acc: 0.1790
Epoch: 6 | train_loss: 1.7915 | train_acc: 0.1752 | test_loss: 1.7910 | test_acc: 0.1790
Epoch: 7 | train_loss: 1.7915 | train_acc: 0.1756 | test_loss: 1.7910 | test_acc: 0.1790
Epoch: 8 | train_loss: 1.7915 | train_acc: 0.1769 | test_loss: 1.7911 | test_acc: 0.1790
Epoch: 9 | train_loss: 1.7916 | train_acc: 0.1747 | test_loss: 1.7909 | test_acc: 0.1790
Epoch: 10 | train_loss: 1.7914 | train_acc: 0.1748 | test_loss: 1.7909 | test_acc: 0.1790
</code></pre>
<pre><code class="lang-auto">class KAMYOnet(nn.Module):
    def __init__(self, input_shape: int, output_shape: int):
        super().__init__()
        self.hidden_units = 128
        num_blocks = 2
        self.blocks = nn.ModuleList()
        
        self.blocks.append(nn.Sequential(
            nn.Conv2d(in_channels=input_shape,
                      out_channels=self.hidden_units,
                      kernel_size=3,
                      stride=1,
                      padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2)
        ))
                
        for j in range(num_blocks):
            self.hidden_units += 128
            self.block1()
            self.hidden_units += 128
            self.block1()
            self.blocks.append(nn.Sequential(nn.MaxPool2d(kernel_size=2, stride=2)))
            

        self.blocks.append(nn.Sequential(
            nn.Conv2d(in_channels=self.hidden_units,
                      out_channels=output_shape,
                      kernel_size=3,
                      stride=1,
                      padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2)
        ))
                

        # Dynamically calculate the input size for the linear layer
        with torch.no_grad():
            x = torch.randn(1, input_shape, 64, 64)
            self.weightedAvgNum = 0
            
            for block_idx, block in enumerate(self.blocks):
                for layer_idx, layer in enumerate(block):
                        x = layer(x)
                    #print(f"Block {block_idx}, Layer {layer_idx}: Output shape: {x.shape}")
                
        # Flatten the output tensor before passing it to the linear layer
        x = torch.flatten(x, start_dim=1)
        
        self.classifier = nn.Linear(x.size(1), output_shape)
    
    def block1(self):
        self.blocks.append(nn.Sequential(
            nn.Conv2d(in_channels=(self.hidden_units-128),
                        out_channels=self.hidden_units,
                        kernel_size=3,
                        stride=1,
                        padding=1),
            nn.ReLU(),
            nn.Conv2d(in_channels=(self.hidden_units),
                        out_channels=self.hidden_units,
                        kernel_size=3,
                        stride=1,
                        padding=1),
            nn.ReLU(),
        ))

    def forward(self, x: torch.Tensor):
        for block in self.blocks:
            for layer in block:
                    x = layer(x)
                    
        x = self.classifier(x.view(x.size(0), -1))  # Flatten before passing to linear layer
        return x

model = KAMYOnet(input_shape=3, output_shape=6).to(device)

criterion = nn.CrossEntropyLoss()

optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

loss_fn = nn.CrossEntropyLoss()
</code></pre>

ptrblck: <p>The initial loss seems to be quite large. Did you check the first output(s) and target(s) to see why the loss seems to be to large?</p>

Faruk_Celik: <p>This is what I got. I am still not sure why it is too large</p>
<pre><code class="lang-auto">Outputs: tensor([[-0.1163, -0.0291, -0.0039,  0.0696, -0.0227,  0.0322]],
       device='cuda:0', grad_fn=&lt;AddmmBackward0&gt;)
Targets: tensor([5], device='cuda:0')
Epoch: 1 | train_loss: 1004291905552786522112.0000 | train_acc: 0.1732 | test_loss: 1.7910 | test_acc: 0.1790
</code></pre>

ptrblck: <p>I cannot reproduce the large loss from this batch as the outputs also contain values in the expected range:</p>
<pre data-code-wrap="python"><code class="lang-python">output = torch.tensor([[-0.1163, -0.0291, -0.0039,  0.0696, -0.0227,  0.0322]])
targets = torch.tensor([5])

criterion = nn.CrossEntropyLoss()
loss = criterion(output, targets)
print(loss)
# tensor(1.7495)
</code></pre>
<p>Are you using the latest PyTorch version? If not, could you update and rerun your code?<br>
In an older release some error checks were broken (E.g. <code>1.13</code> as described <a href="https://github.com/pytorch/pytorch/issues/106467#issuecomment-1677712857">here</a>) and your loss values might indeed be caused by accessing invalid values and thus uninitialized memory.</p>