Training loss keeps increasing by approx double after each iteration and accuracy goes beyond 100 after 5 epochs

username: Panashe1812
cooked: <p>Hey community,</p>
<p>I have hit a brick wall on a project I am working on for my master’s dissertation. The project is to compare different methods used for explainable AI in image recognition ( LIME SHAP and Grad CAM). Before doing that, I need to create a classifier for the images. The problem I am facing is that when training a vgg16 or Alexnet models on the, [Fairface Dataset]<br>
(<a href="https://github.com/joojs/fairface" class="inline-onebox" rel="noopener nofollow ugc">GitHub - joojs/fairface: FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age</a>), the loss keeps increasing by approx double after each iteration and the training accuracy goes beyond 100 after 5 epoch (148). I am sure something is wrong somewhere but I cant figure out what the problem is and I a kindly asking for assistance.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/4/243a307f02fdb11fa247f75a402d9e5132d95f69.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/243a307f02fdb11fa247f75a402d9e5132d95f69" title="first few iterations losses"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/243a307f02fdb11fa247f75a402d9e5132d95f69_2_666x500.jpeg" alt="first few iterations losses" data-base62-sha1="5atRgT5JsLz8a2saHpfCPEJuqut" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/243a307f02fdb11fa247f75a402d9e5132d95f69_2_666x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/243a307f02fdb11fa247f75a402d9e5132d95f69_2_999x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/243a307f02fdb11fa247f75a402d9e5132d95f69_2_1332x1000.jpeg 2x" data-dominant-color="878B92"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">first few iterations losses</span><span class="informations">1920×1440 408 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<strong>training loss for the first few iterations</strong></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f0acc8d281d23585edc64d158c9b4a2f8d452de0.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f0acc8d281d23585edc64d158c9b4a2f8d452de0" title="Training loss after the first epoch"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f0acc8d281d23585edc64d158c9b4a2f8d452de0_2_666x500.jpeg" alt="Training loss after the first epoch" data-base62-sha1="yl6MtFaaOAl9RNHxic5LjpB1vgs" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f0acc8d281d23585edc64d158c9b4a2f8d452de0_2_666x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f0acc8d281d23585edc64d158c9b4a2f8d452de0_2_999x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f0acc8d281d23585edc64d158c9b4a2f8d452de0_2_1332x1000.jpeg 2x" data-dominant-color="8F8C8E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Training loss after the first epoch</span><span class="informations">1920×1440 580 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Training loss after the first epoch</strong></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b33c42e63e26b075db19d827cbe00dfc18b094ab.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b33c42e63e26b075db19d827cbe00dfc18b094ab" title="training loss after 2 epochs"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b33c42e63e26b075db19d827cbe00dfc18b094ab_2_666x500.jpeg" alt="training loss after 2 epochs" data-base62-sha1="pzAzK5m3CC2KH0ODNOCEWshSufN" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b33c42e63e26b075db19d827cbe00dfc18b094ab_2_666x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b33c42e63e26b075db19d827cbe00dfc18b094ab_2_999x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b33c42e63e26b075db19d827cbe00dfc18b094ab_2_1332x1000.jpeg 2x" data-dominant-color="909395"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">training loss after 2 epochs</span><span class="informations">1920×1440 662 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>training loss after 2 epochs</strong></p>
<p>below is my code. I have only included the the custom dataset and training code but happy to provide further details.</p>
<pre><code class="lang-auto">#custom Dataset class to load the custom data
class CustomDataset(Dataset):
    def __init__(self, csv_file, root_dir, transform=None):
        self.annotations = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.annotations)

    def __getitem__(self, index):
        #if torch.is_tensor(index):
            #idx = idx.tolist()
            
        
        
        img_path = os.path.join(self.root_dir, self.annotations.iloc[index, 0])
        image = io.imread(img_path)
        
        #convert str labels to int labels
        class_map = {'Latino_Hispanic': 0,
             'East Asian': 1,
             'Indian': 2,
             'Middle Eastern': 3,
             'Black': 4,
             'Southeast Asian': 5,
             'White': 6
             }
        
        label = self.annotations.iloc[index, 3] # returns a tuple of strings
        label =class_map[label]
        #label= OneHotEncoder(label)
        
        #label = label.applylambda label: list(map(int, label)))
        
        if self.transform:
            image = self.transform(image)
        
        return image, label

</code></pre>
<pre><code class="lang-auto">vis_augs = transforms.Compose([
                    T.ToPILImage(),
                    T.Resize((224,224)),
                    T.ToTensor()
    ]
                    )
</code></pre>
<pre><code class="lang-auto">train_data = CustomDataset(
    csv_file="fairface_label_train.csv",
    root_dir="train/",
    transform=vis_augs,
)

# Load Data
test_data = CustomDataset(
    csv_file="fairface_label_val.csv",
    root_dir="/val",
    transform=vis_augs,
)
</code></pre>
<pre><code class="lang-auto">batch_size =256
train_loader = DataLoader(train_data, batch_size=batch_size, shuffle=True)
test_loader = DataLoader(test_data,batch_size=batch_size, shuffle=True)
</code></pre>
<pre><code class="lang-auto">model = model.to(device)

criterion = nn.CrossEntropyLoss()

 #Observe that all parameters are being optimized
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)

# Decay LR by a factor of 0.1 every 7 epochs
scheduler = lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)

</code></pre>
<p>The training loop is as below:</p>
<pre><code class="lang-auto">  # Put model in train mode

model.train()
running_loss = 0.0
running_corrects =0.0

counter = 0
for epoch in range(2): 
    # Iterate over data.
    for inputs, labels in train_loader:
        set_trace()
        counter +=1
        #inputs, labels =data
        
        inputs = inputs.to(device)
        labels = labels.to(device)
        #set_trace()
        # zero the parameter gradients
        optimizer.zero_grad()

        # forward      
        outputs = model(inputs)
        _, preds = torch.max(outputs, 1)
      # preds = torch.argmax(torch.softmax(pred, dim=1), dim=1)
        loss = criterion(outputs, labels)

        # backward + optimize only if in training phase
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
        print(f'running losss: {running_loss}', end= "  ")
        running_corrects += torch.sum(preds == labels)
        print(f'running corrects: {running_corrects}', end= "  ")
        #scheduler.step()
        print(f'iteration: {counter}')
    epoch_loss = running_loss / len(train_loader)
    epoch_acc = running_corrects.double() / len(train_loader)

    print(f'{epoch} train Loss: {epoch_loss:.4f} Acc: {epoch_acc:.4f}')
    #return epoch_loss, epoch_acc
</code></pre>

username: ptrblck
cooked: <p>You are accumulating the loss and accuracy into the <code>running_</code> variables for all epochs without resetting them at the beginning of a new epoch. Move the <code>running_</code> variable creation into the <code>for epoch</code> loop and check if this would fix the issue.</p>

username: Panashe1812
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="160217">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Move the <code>running_</code> variable creation into the <code>for epoch</code> loop</p>
</blockquote>
</aside>
<p>Thank you for response. I am now starting to get the expected results, although the training accuracy is too low compared to validation accuracy. ( I used a very small subset of the data and ran a few epochs and I hoping increasing these will yield better results.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/c/fc47da990c9f9b8b104f87e5f10491217924f3e7.png" data-download-href="https://discuss.pytorch.org/uploads/default/fc47da990c9f9b8b104f87e5f10491217924f3e7" title="accuracy"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/c/fc47da990c9f9b8b104f87e5f10491217924f3e7.png" alt="accuracy" data-base62-sha1="zZMgFlaolsyTD5UCaWsto54zy9F" width="524" height="500" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">accuracy</span><span class="informations">873×832 37.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>