Weird behavior when obtaining training accuracy

Rafid_Mahbub: <p>Hi,</p>
<p>I am new to PyTorch and have been working on the CIFAR10 dataset. When I try to plot the accuracy history over the training epochs, I see a weird behavior. The accuracy up to the penultimate epochs are all negligible, after which it shoots up to something that is reasonable. I get something like the following<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/3/73f3af91cabe189fd70e455d2f54982e4a11ba5b.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/73f3af91cabe189fd70e455d2f54982e4a11ba5b" title="Screenshot 2024-08-19 at 17.10.05"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/3/73f3af91cabe189fd70e455d2f54982e4a11ba5b_2_690x366.jpeg" alt="Screenshot 2024-08-19 at 17.10.05" data-base62-sha1="gxL0dGlRnovsnKlZsfe04fQfvoD" width="690" height="366" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/3/73f3af91cabe189fd70e455d2f54982e4a11ba5b_2_690x366.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/3/73f3af91cabe189fd70e455d2f54982e4a11ba5b_2_1035x549.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/3/73f3af91cabe189fd70e455d2f54982e4a11ba5b_2_1380x732.jpeg 2x" data-dominant-color="F8F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-08-19 at 17.10.05</span><span class="informations">1708×908 76.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The training loop that I have is</p>
<pre><code class="lang-auto">def train(dataloader, model, loss_fn, optimizer, num_epochs):
    size = len(dataloader.dataset)
    batch_size = len(dataloader)
    loss_train = np.zeros(num_epochs) # Loss history per epoch during training
    acc_train = np.zeros(num_epochs) # Accuracy history per epoch during training
    running_correct = 0 # Keeps track of the number of correct classifications
    running_total = 0
    model.train()
    for epoch in range(num_epochs):
        for i, (image, label) in enumerate(dataloader):
            image, label = image.to(device), label.to(device)
            
            '''  
            Calculate prediction and loss function per epoch
            '''
            label_pred = model(image)
            loss = loss_fn(label_pred, label)
            ''' 
            Backpropagating the loss function to re-adjust weights
            '''
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()

            loss_train[epoch] += loss.item()*image.size(0) # Accumulated loss per training batch
            _, predicted = torch.max(label_pred, 1)
            running_total += label.size(0)
            running_correct += (predicted == label).sum().item()
            acc_train[epoch] = running_correct # Number of correct predictions per training batch

            if (i + 1) % 2500 == 0:
                print (f'Epoch [{epoch+1}/{num_epochs}], Step [{i+1}/{batch_size}], Loss: {loss:.4f}')

        loss_train[epoch] = loss_train[epoch]/size
        acc_train = acc_train/running_total # Proportion of accurate predictions      
    
    return loss_train, acc_train


</code></pre>
<p>I am not sure why I am seeing this behavior.</p>
<p>Regards</p>

Soumya_Kundu: <p>Your <code>running_{variables}</code> should be within the loop so they are refreshed each epoch.</p>
<p>The <code>acc_train[epoch]</code> should be outside the epoch loop.</p>
<p>Also look into this: <a href="https://discuss.pytorch.org/t/order-of-backward-step-and-zero-grad/119023" class="inline-onebox">Order of backward(), step() and zero_grad()</a> for your ordering of backwards and zerograd.</p>
<p>Generally, it is more with the data or the hyper-parameters which cause weird trends like this and not a broken training logic. Is this happening every time or is this one off?</p>

Rafid_Mahbub: <p>Thanks for the answer. It solved the problem. I was also updating one of the arrays incorrectly, so it is all good now.</p>