Fluctuating Loss and Accuracy in CNN Classification Model

Sammmm: <p>Hi all,</p>
<p>I am attempting to learn how to classify participants from the ABIDE dateset using PyTorch (a CNN) and fMRI data.</p>
<p>I have been playing around with this <a href="https://github.com/Vishal232406/FMRI_CNN" rel="noopener nofollow ugc">model</a> that I found online. After a couple of weeks of troubleshooting I still can’t get it to work properly. I have tried changing all the hyper-parameters, different data, a different CNN model, and more (at one stage I re-coded everything but the data loader).</p>
<p>However; every-time I run the model it doesn’t appear to train and has fluctuating loss and accuracy:</p>
<p>See the following examples:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2055d53beac9632857136c5a4a8f415c89ad7e2.png" data-download-href="https://discuss.pytorch.org/uploads/default/a2055d53beac9632857136c5a4a8f415c89ad7e2" title="Figure 2022-10-12 175204"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a2055d53beac9632857136c5a4a8f415c89ad7e2_2_690x232.png" alt="Figure 2022-10-12 175204" data-base62-sha1="n7iQjhUuPs4YsZY0tVCtMNdpuZY" width="690" height="232" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a2055d53beac9632857136c5a4a8f415c89ad7e2_2_690x232.png, https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2055d53beac9632857136c5a4a8f415c89ad7e2.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2055d53beac9632857136c5a4a8f415c89ad7e2.png 2x" data-dominant-color="FCF7F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Figure 2022-10-12 175204</span><span class="informations">940×317 26.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Epoch [1/50],<br>
Loss: 54.1667, Accuracy: 50.0000<br>
Epoch [2/50],<br>
Loss: 53.1250, Accuracy: 66.6667<br>
Epoch [3/50],<br>
Loss: 52.7778, Accuracy: 50.0000<br>
Epoch [4/50],<br>
Loss: 52.6042, Accuracy: 33.3333<br>
Epoch [5/50],<br>
Loss: 51.6667, Accuracy: 50.0000<br>
Epoch [6/50],<br>
Loss: 51.3889, Accuracy: 50.0000<br>
Epoch [7/50],<br>
Loss: 51.1905, Accuracy: 16.6667<br>
Epoch [8/50],<br>
Loss: 51.3021, Accuracy: 33.3333<br>
Epoch [9/50],<br>
Loss: 51.3889, Accuracy: 66.6667<br>
Epoch [10/50],<br>
Loss: 51.4583, Accuracy: 50.0000</p>
<p>If anyone has some insight on what I could do I would really appreciate it!<br>
I am also a bit of an amateur (only been working in this area for a year) so I really appreciate any input you can give.</p>
<p>Best,<br>
Sam</p>

Sammmm: <p>Here is the code for the <a href="https://pastebin.com/pZLVW0vQ" rel="noopener nofollow ugc">dataloader</a>, <a href="https://pastebin.com/gGSHc5C9" rel="noopener nofollow ugc">model</a>, and the <a href="https://pastebin.com/77h91Kjh" rel="noopener nofollow ugc">training/testing procedure</a>.</p>
<p>I feel like the problem might be with the dataloader or the way the data is put through the model. I have tried rewriting the training/testing procedure but had no luck. I have not been able to rewrite the dataloader yet or troubleshoot it as loading/transforming nifti files is complex and I am still learning.</p>

Sammmm: <p>Maybe this could be an over-fitting problem as the dateset only contains ~500 participants?</p>

Sammmm: <p>Okay I’m now thinking that it is because the data-loader is randomly selecting data, thus the model cannot train properly.</p>
<p>This is the original get item code for the random data-loader:</p>
<pre><code class="lang-auto">def __getitem__(self, index: int) -&gt; Tensor:
        # just return a random patch
        global array_1
        path = np.random.choice(self.img_paths)
        img = nib.load(str(path))
        # going larger than max_idx will put us past the end of the array
        max_idx = np.array(img.shape) - np.array(self.patch_shape) + 1

        # Python has a `slice` object which you can use to index into things with the `[]` operator
        # we are going to build the slices we need to index appropriately into our niis with the
        # `.dataobj` trick
        slices = []
        for length, maximum in zip(self.patch_shape, max_idx):
            start = np.random.randint(0, maximum)
            slices.append(slice(start, start + length))
        array = img.dataobj[slices[0], slices[1], slices[2], slices[3]]

        if self.standardize:
            array_1 = np.copy(array)
            array_1 -= np.mean(array_1)
            array_1 /= np.std(array_1, ddof=1)
        return torch.Tensor(array_1)
</code></pre>
<p>I have tried altering it by changing <code>path = np.random.choice(self.img_paths)</code> to <code>path = self.img_paths[index]</code> and <code>start = np.random.randint(0, maximum)</code> to <code>start = np.array(0, maximum)</code>.</p>
<p>However, I am still having the same problems.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8dae1eeb31ae5cd352ebbe402b9c869f83c61dfb.png" alt="image" data-base62-sha1="kdmngWBMh3v3Sai9o4AZfNL2RFV" width="288" height="292"></p>
<p>Any suggestions for stopping the data-loader from being random are appreciated.</p>

ptrblck: <p>Using <code>np.random.choice</code> instead of the passed <code>index</code> sounds weird, but might still work assuming the target is still corresponding to the created image.<br>
However, in your code snippet only the input is returned without a target tensor, so could you explain your classification use case a bit more as it doesn’t seem to depend on ground truth information?</p>
<p>Your current loss is also negative, which is usually wrong, so check which loss function is used and make sure the values of the model output (and potentially a target) are in the valid range.</p>

Sammmm: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> thank you so much for the reply!</p>
<p>Firstly, thanks for pointing out the problem with the negative loss. I found that my labels were coded as 1 and 2 instead of 1 and 0 (I found the issue discussed <a href="https://github.com/keras-team/keras/issues/1917#issuecomment-451534575" rel="noopener nofollow ugc">here</a>). After changing my labels the loss values are now positive (see image below)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/0/20bb2ee3e2b1a5382455e33f48f28ed1c950418d.png" alt="image" data-base62-sha1="4FyjF4dF4wQY4BnUSewTPxUPiKp" width="261" height="287"></p>
<p>As for the ground truth and randomness, there should be a ground truth as I am trying to classify  participants (diagnose control or Autism) using preexisting diagnostic labels and imaging data. So the fact that the data-loader doesn’t output the target data is indeed something I missed. I have now added in <code>single_label = self.labels[index]</code> and <code>return (img_tensor, single_label)</code> to the data-loader. The results now look a bit better but the accuracy and loss are still rather random(see image below).<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/5/25c3a4b55abb592dce98fbbb410b7d2e25656140.png" alt="image" data-base62-sha1="5o4PudqkBZA5y2qjKBEEn6jlf68" width="278" height="291"></p>
<p>Correct me if I’m wrong, but I think I may need to rewrite parts of the training/testing file now that the model returns two values?</p>
<p>If so, I will work on that tomorrow.<br>
Once again thank you so much for your help!</p>

ptrblck: <aside class="quote no-group" data-username="Sammmm" data-post="6" data-topic="163336">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/8dc957/48.png" class="avatar"> Sammmm:</div>
<blockquote>
<p>Correct me if I’m wrong, but I think I may need to rewrite parts of the training/testing file now that the model returns two values?</p>
</blockquote>
</aside>
<p>I don’t know why the model should return two value or do you mean the <code>Dataset</code> and <code>DataLoader</code>?<br>
If so, then yes you should use the corresponding target tensors returned by your <code>DataLoader</code> for training which would need some code changes.</p>

Sammmm: <p>Yes my mistake, I meant to say data-loader.</p>
<p>I have rewritten my data-loader to output the image tensors and labels. I have also simplified it a lot. For reference, it now reads as follows:</p>
<pre><code class="lang-auto">import pandas as pd
import numpy as np
from torch.utils.data import Dataset
import torch
import nibabel as nib
from pathlib import Path
from typing import List

class SamDataloader(Dataset):
    def __init__(self):
        #loading the labels from a csv file where the lables are the second colum with a header
        self.labels_df = pd.read_csv("ABIDE_FMRI_cleaned.csv")
        self.labels =  np.asarray(self.labels_df)
        self.labels = np.asarray(self.labels[:, 1])
        
        #laoding the nii paths from a file directory (directory is set in Spyder IDE)
        self.image_path: List[Path] = sorted(Path(__file__).resolve().parent.rglob("*.nii"))
        #setting the length fir the dataloader, this is nessisary but idk why
        self.data_len = len(self.labels)
    
    def __getitem__(self, index):
        #loading a single image using indexing and nibload
        single_image_path = self.image_path[index]
        single_image_nii = nib.load(single_image_path)
        single_image_array = single_image_nii.get_fdata()
        single_image_array = single_image_array.astype(np.float32)
        #reordering the image as torch preferes it in a different order
        single_image = single_image_array.transpose((3,2,1,0)) #not sure if this is the right order but we will see
        #converting the image to tensor
        single_image_tensor = torch.FloatTensor(single_image)
        
        #loading a single  label using the index in get item and making it a tensor for torch
        single_label = self.labels[index]
        label_temp = np.array(single_label)
        #label_as_tensor = torch.from_numpy(label_temp)
        
        #making the loader output the final image and label tensors for on participant (torch loops it for all participants)
        return (single_image_tensor, label_temp)
    
    def __len__(self):
        #returning the length defined in init
        return self.data_len
</code></pre>
<p>I also changed parts of the training/testing file to take two inputs from the data-loader as we discussed. This file now appears as follows:</p>
<pre><code class="lang-auto">import warnings
import torch
from torch.autograd import Variable
import torchvision.transforms as transforms
from torch.utils.data import DataLoader
from sklearn.model_selection import train_test_split
from torch.utils.data import Subset
from numpy import vstack
from sklearn.metrics import accuracy_score
import numpy as np
from matplotlib import pyplot as plt
from torch.utils.data.sampler import SubsetRandomSampler
from datetime import datetime

from SamDataloader import SamDataloader
from cnn_model_V2 import CNN_model

"""
Use "torch.cuda.empty_cache()" if you have the CUDA out of memory RuntimeError
"""

if __name__ == '__main__':
    learning_rate = 1e-3
    batch_size = 32
    num_epochs = 200
    validation_split = .2
    shuffle_dataset = True
    random_seed= 42
    num_classes = 2
    compose = transforms.Compose([
        transforms.ToTensor(),
        transforms.RandomHorizontalFlip(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
    ])
    
    dataset = SamDataloader()
    

    device = torch.device("cuda")
    model = CNN_model().cuda()
    
    #I removed the part that added labels and image tensors together because my data loader already outputs them.

     # Creating data indices for training and validation splits:
    dataset_size = len(dataset)
    indices = list(range(dataset_size))
    split = int(np.floor(validation_split * dataset_size))
    if shuffle_dataset :
         np.random.seed(random_seed)
         np.random.shuffle(indices)
    train_indices, val_indices = indices[split:], indices[:split]
     
    # Creating PT data samplers and loaders:
    train_sampler = SubsetRandomSampler(train_indices)
    validation_sampler = SubsetRandomSampler(val_indices)
     
    train_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, 
                                                sampler=train_sampler)
    validation_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size,
                                                     sampler=validation_sampler)

    def train_model(train_dl, model):
        criterion = torch.nn.BCELoss(reduction='mean').to(device)
        optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)
        loss_array = []
        loss_values = []
        acc_values = []
        for epoch in range(num_epochs):
            for ind, (data, img_label) in enumerate(train_dl):
                inputs = data.permute(0, 1, 4, 2, 3), #reordered from original because my dataloader orders them differently

                inputsV, labelsV = Variable(inputs[0]), Variable(img_label)
                inputsV = inputsV.float() 
                labelsV = labelsV.float() 
                inputsV = inputsV.to(device) 
                labelsV = labelsV.to(device) 
                y_pred = model(inputsV)

                loss = criterion(y_pred.squeeze(), labelsV)
                running_loss = abs(loss.item())
                loss_array.append(running_loss)
                optimizer.zero_grad()
                loss.backward()
                optimizer.step()
            avg_loss = np.average(loss_array)
            loss_values.append(avg_loss)
            acc = evaluate_model(validation_loader, model)
            acc = acc * 100
            acc_values.append(acc)
            now = datetime.now()
            current_time = now.strftime("%H:%M:%S")
            print("Time:", current_time, '\n' f'Epoch [{epoch + 1}/{num_epochs}], \n Loss: {abs(avg_loss):.4f}, Accuracy: {acc:.4f}')
        plot(acc_values, loss_values)

    def evaluate_model(test_dl, model):
        predictions, actuals = list(), list()
        accuracy_array = []
        for ind, (data, img_label) in enumerate(test_dl):
            inputs = data.permute(0, 1, 4, 2, 3), #reordered from original because my dataloader orders them differently

            inputsV, labelsV = Variable(inputs[0]), Variable(img_label)
            inputsV = inputsV.float() 
            labelsV = labelsV.float() 
            inputsV = inputsV.to(device)
            labelsV = labelsV.to(device) 
            
            yhat = model(inputsV) #added cpu conversion to get numpy to work

            actual = labelsV.cpu().numpy() #added cpu conversion to get numpy to work
            
            actual = actual.reshape((len(actual), 1))
            
            yhat = yhat.cpu()
            yhat = yhat.detach().numpy()
            yhat = yhat.round()

            predictions.append(yhat)
            actuals.append(actual)
            predictions, actuals = vstack(predictions), vstack(actuals)

            acc = accuracy_score(actuals, predictions)
            accuracy_array.append(acc)
            print( 'Accuracy:', acc*100)

            return acc


    def plot(x, y):
        plt.figure(figsize=(16, 5))
        plt.xlabel('EPOCHS')
        plt.ylabel('LOSS/ACC')

        plt.plot(x, 'r', label='ACCURACY')
        plt.plot(y, 'b', label='LOSS')
        plt.legend()
        plt.show()

    train_model(train_loader, model)
    evaluate_model(validation_loader, model)
</code></pre>
<p>However, I am still having the same problems with the loss and accuracy as seen by the output below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/2/2248a5ede92ad3d6dadc64e719128e130d4a42ef.png" data-download-href="https://discuss.pytorch.org/uploads/default/2248a5ede92ad3d6dadc64e719128e130d4a42ef" title="loss and acc 10.11.22"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/2/2248a5ede92ad3d6dadc64e719128e130d4a42ef.png" alt="loss and acc 10.11.22" data-base62-sha1="4ThSGgzxc7zJB5rhND0IvUqB4Yv" width="219" height="500" data-dominant-color="2D363F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss and acc 10.11.22</span><span class="informations">250×569 5.16 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea51d28e15ef9c9c44372413be64996919db8ec8.png" data-download-href="https://discuss.pytorch.org/uploads/default/ea51d28e15ef9c9c44372413be64996919db8ec8" title="loss and acc plot 10.11.22"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea51d28e15ef9c9c44372413be64996919db8ec8_2_690x232.png" alt="loss and acc plot 10.11.22" data-base62-sha1="xqT1YMtxVtRbr7pV6DyNAVffiDS" width="690" height="232" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea51d28e15ef9c9c44372413be64996919db8ec8_2_690x232.png, https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea51d28e15ef9c9c44372413be64996919db8ec8.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea51d28e15ef9c9c44372413be64996919db8ec8.png 2x" data-dominant-color="FCF9FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss and acc plot 10.11.22</span><span class="informations">940×317 25.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m thinking that maybe I need to try things like giving the CNN more layers or rewriting the training/testing file but I’m really not sure . Do you have any suggestions on how to proceed?</p>

ptrblck: <p>I would recommend to try to overfit a small dataset (e.g. just 10 samples) by playing around with some hyperparameters of your model. Once this is done, you could try to scale up the use case again. If you cannot overfit on this tiny data, your training script might have more issues.</p>

Sammmm: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Thank you so much for your assistance!</p>
<p>I followed your recommendation and found that the model was busted and only outputting 0’s. After some review it seems that I fundamentally misunderstood the difference between creating a binary classification and multi-class classification model. Specifically, I was trying to use “One Hot Encoded” data with a softmax activation layer, and BCEwithlogits loss. I was able to run the model after setting the model to one class while using a sigmoid activation layer and BCE loss.</p>
<p>At the moment the training looks as follows:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f29104f387ba2152c01095d6e5545f1f879fca9c.png" data-download-href="https://discuss.pytorch.org/uploads/default/f29104f387ba2152c01095d6e5545f1f879fca9c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f29104f387ba2152c01095d6e5545f1f879fca9c_2_690x231.png" alt="image" data-base62-sha1="yBQfwmjng1dbdhpZhs5Lv7Jfqz2" width="690" height="231" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f29104f387ba2152c01095d6e5545f1f879fca9c_2_690x231.png, https://discuss.pytorch.org/uploads/default/original/3X/f/2/f29104f387ba2152c01095d6e5545f1f879fca9c.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/2/f29104f387ba2152c01095d6e5545f1f879fca9c.png 2x" data-dominant-color="FCFCFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">944×317 11.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3fc18614a343e9434abeb473885bb4cb5beb613c.png" data-download-href="https://discuss.pytorch.org/uploads/default/3fc18614a343e9434abeb473885bb4cb5beb613c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3fc18614a343e9434abeb473885bb4cb5beb613c_2_690x232.png" alt="image" data-base62-sha1="960HBJItWKfhXBn1jptEPbzz7f6" width="690" height="232" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3fc18614a343e9434abeb473885bb4cb5beb613c_2_690x232.png, https://discuss.pytorch.org/uploads/default/original/3X/3/f/3fc18614a343e9434abeb473885bb4cb5beb613c.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/f/3fc18614a343e9434abeb473885bb4cb5beb613c.png 2x" data-dominant-color="FDFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">940×317 11 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I just wanted to inquire about the accuracy. It is very blocky and gives poor results (chance-like accuracy). I have had success getting the loss to smoothly decline; however, I haven’t been able to get the accuracy to gradually and smoothly increase as I’ve seen in other’s results. Do you have any suggestions for how to approach this problem? Or is it more a matter of tweaking hyper-parameters?</p>
<p>At the moment I find that I can stop the accuracy from being so jumpy by increasing the batch size to 128 (see accuracy figure above) and it gradually increases with a learning rate of 0.001. If I change the learning rate the accuracy either decreases (0.01) or remains constant (0.0001). maybe I need a more complex model, momentum, or even data augmentation (I currently have 480 samples)? I am unsure.</p>
<p>I will reattach my code for my model and below for those observing this post so they can see the changes I’ve made.</p>
<p>Dataloader: <a href="https://pastebin.com/DFqrS8JS" class="inline-onebox" rel="noopener nofollow ugc">Dataloader_V2 - Pastebin.com</a></p>
<p>Model:</p>
<pre><code class="lang-auto">import torch
from torch import nn

import torchmetrics


class CNN_model(nn.Module):
    def __init__(self):
        super(CNN_model, self).__init__()

        self.convolutional_layer = nn.Sequential(
            nn.Conv3d(in_channels=8, out_channels=16, kernel_size=(8, 8, 8), dilation=(3, 3, 3), stride=(2, 2, 2),
                      padding=(0, 0, 0)),
            nn.BatchNorm3d(16),
            nn.ReLU(),
            nn.Conv3d(in_channels=16, out_channels=24, kernel_size=(8, 8, 8), dilation=(2, 2, 2), stride=(2, 2, 2),
                      padding=(0, 0, 0)),
            nn.BatchNorm3d(24),
            nn.ReLU(),
            nn.Conv3d(in_channels=24, out_channels=32, kernel_size=(3, 3, 6), dilation=(1, 1, 1), stride=(1, 1, 1),
                      padding=(0, 0, 0)),
            nn.BatchNorm3d(32),
            nn.ReLU()
        )

        self.linear_layer = nn.Sequential(
            nn.Linear(in_features=32, out_features=24),
            nn.BatchNorm1d(24),
            nn.ReLU(),
            nn.Linear(in_features=24, out_features=16),
            nn.BatchNorm1d(16),
            nn.ReLU(),
            nn.Linear(in_features=16, out_features=1))
             
        #self.accuracy = torchmetrics.Accuracy()

    def forward(self, x):
        x = self.convolutional_layer(x)
        x = torch.flatten(x, 1)
        x = self.linear_layer(x)

        x = torch.sigmoid(x) # must have dim =1 for softmax https://discuss.pytorch.org/t/bceloss-vs-bcewithlogitsloss/33586
        return x
</code></pre>
<p>Training/Testing file:</p>
<pre><code class="lang-auto">import torch
import torchvision.transforms as transforms
import numpy as np

from torch.autograd import Variable
from numpy import vstack
from sklearn.metrics import accuracy_score
from matplotlib import pyplot as plt
from torch.utils.data.sampler import SubsetRandomSampler

from dataloader import RandomFmriDataset
from model import CNN_model

"""
Use "torch.cuda.empty_cache()" if you have the CUDA out of memory RuntimeError
"""

if __name__ == '__main__':
    #Defining hyperparameters
    learning_rate = 1e-3  #1e-2 results in decreseaing accuracy and 1e-4 in constant accuracy.
    batch_size = 128 #higher batch size makes accuracy the same or smoothes it out. https://medium.com/mini-distill/effect-of-batch-size-on-training-dynamics-21c14f7a716e
    num_epochs = 200
    validation_split = .2
    shuffle_dataset = True
    random_seed= 42
    num_classes = 1
    compose = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
    ]) #took out transforms.RandomHorizontalFlip(),
    
    #Defining key dependencies
    dataset = RandomFmriDataset()
    device = torch.device("cuda")
    model = CNN_model().cuda()
    

    #Creating data indices for training and validation splits:
    dataset_size = len(dataset)
    indices = list(range(dataset_size))
    split = int(np.floor(validation_split * dataset_size))
    if shuffle_dataset :
         np.random.seed(random_seed)
         np.random.shuffle(indices)
    train_indices, val_indices = indices[split:], indices[:split]
     
    #Creating the data samplers and loaders:
    train_sampler = SubsetRandomSampler(train_indices)
    validation_sampler = SubsetRandomSampler(val_indices)
     
    train_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, 
                                                sampler=train_sampler)
    validation_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size,
                                                     sampler=validation_sampler)
    #Model training
    def train_model(train_dl, model):
        criterion = torch.nn.BCELoss(reduction='mean').to(device)
        optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)
        loss_array = []
        loss_values = []
        acc_values = []
        for epoch in range(num_epochs):
            for ind, (data, img_label) in enumerate(train_dl):
                inputs = data.permute(0, 1, 4, 2, 3),

                inputsV, labelsV = Variable(inputs[0]), Variable(img_label)
                inputsV = inputsV.float() 
                labelsV = labelsV.float() 
                inputsV = inputsV.to(device) 
                labelsV = labelsV.to(device) 
                y_pred = model(inputsV)

                loss = criterion(y_pred, labelsV)
                running_loss = abs(loss.item())
                loss_array.append(running_loss)
                optimizer.zero_grad()
                loss.backward()
                optimizer.step()
            avg_loss = np.average(loss_array)
            loss_values.append(avg_loss)
            acc = evaluate_model(validation_loader, model)
            acc = acc * 100
            acc_values.append(acc)
            print(f'Epoch [{epoch + 1}/{num_epochs}], \n Loss: {abs(avg_loss):.4f}, Accuracy: {acc:.4f}')
        acc_plot(acc_values) 
        loss_plot(loss_values)

    #Model testing 
    def evaluate_model(test_dl, model):
        predictions, actuals = list(), list()
        accuracy_array = []
        for ind, (data, img_label) in enumerate(test_dl):
            inputs = data.permute(0, 1, 4, 2, 3), 

            inputsV, labelsV = Variable(inputs[0]), Variable(img_label)
            inputsV = inputsV.float() 
            labelsV = labelsV.float() 
            inputsV = inputsV.to(device)
            labelsV = labelsV.to(device) 
            
            yhat = model(inputsV)

            actual = labelsV.cpu().numpy()
            
            actual = actual.reshape((len(actual), 1))
            
            yhat = yhat.cpu()
            yhat = yhat.detach().numpy()
            yhat = yhat.round()

            predictions.append(yhat)
            actuals.append(actual)
            predictions, actuals = vstack(predictions), vstack(actuals)
            acc = accuracy_score(actuals, predictions)
            accuracy_array.append(acc)
            
            #troubleshooting print commands for seeing model predictions and labels
            '''
            print(predictions)
            print(labelsV)
            '''
            
            print(acc)
            return acc

    #Accuracy plot over epochs
    def acc_plot(x):
        plt.figure(figsize=(16, 5))
        plt.xlabel('EPOCHS')
        plt.ylabel('ACCURACY')

        plt.plot(x, 'r', label='ACCURACY')
        plt.legend()
        plt.show()
    #Loss plot over epochs
    def loss_plot(x):
        plt.figure(figsize=(16, 5))
        plt.xlabel('EPOCHS')
        plt.ylabel('LOSS')

        plt.plot(x, 'b', label='LOSS')
        plt.legend()
        plt.show()
    
    #The command for training the model
    train_model(train_loader, model)
    
    #The command for testing the model
    evaluate_model(validation_loader, model)
    
</code></pre>

ptrblck: <p>The steps in the accuracy curve indicate that your sample size is small (thus a change in the classification of a single sample could already be visible), which is expected since you are trying to overfit the small dataset.<br>
I would recommend to not increase the complexity of the model yet, but to make sure your model is able to overfit the tiny dataset first by playing around with some hyperparameters.</p>

Sammmm: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>I followed your advice and over-fitted the model (first on 10 samples and then on the whole dataset). During this stage I found that the accuracy was not being computed correctly so I replaced the relevant code with torchmetrics Accuracy calculation. I also cleaned up the validation code and visualized it. Now the model works better and can easily be over fit (see the following images).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/6/36d8e25cb2380f7f5b9673b6fe6907b88717f009.png" data-download-href="https://discuss.pytorch.org/uploads/default/36d8e25cb2380f7f5b9673b6fe6907b88717f009" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36d8e25cb2380f7f5b9673b6fe6907b88717f009_2_690x232.png" alt="image" data-base62-sha1="7PcsfMdtmHXYSgtEZp7jVdhWm1j" width="690" height="232" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36d8e25cb2380f7f5b9673b6fe6907b88717f009_2_690x232.png, https://discuss.pytorch.org/uploads/default/original/3X/3/6/36d8e25cb2380f7f5b9673b6fe6907b88717f009.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/6/36d8e25cb2380f7f5b9673b6fe6907b88717f009.png 2x" data-dominant-color="FBFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">944×318 12.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/d/ddb551ab788b635abe9bffaa1002d06c35b4a1f9.png" data-download-href="https://discuss.pytorch.org/uploads/default/ddb551ab788b635abe9bffaa1002d06c35b4a1f9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/ddb551ab788b635abe9bffaa1002d06c35b4a1f9_2_690x230.png" alt="image" data-base62-sha1="vDjZHu971HTSEbpwRKL3w4gekWd" width="690" height="230" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/d/ddb551ab788b635abe9bffaa1002d06c35b4a1f9_2_690x230.png, https://discuss.pytorch.org/uploads/default/original/3X/d/d/ddb551ab788b635abe9bffaa1002d06c35b4a1f9.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/d/ddb551ab788b635abe9bffaa1002d06c35b4a1f9.png 2x" data-dominant-color="FBFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">947×317 13.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Should I address the small sample size now? I was thinking of using data augmentation to expand the sample size.</p>

ptrblck: <aside class="quote no-group" data-username="Sammmm" data-post="12" data-topic="163336">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/8dc957/48.png" class="avatar"> Sammmm:</div>
<blockquote>
<p>Should I address the small sample size now? I was thinking of using data augmentation to expand the sample size.</p>
</blockquote>
</aside>
<p>Do you mean the small subset for the initial overfitting testing or your general dataset?<br>
I couldn’t find the dataset size in this topic so unsure if you are still training with only 10 samples.</p>

Sammmm: <p>Sorry for the confusion.</p>
<p>I meant that I should address the small sample size of my general dataset (N = 480).</p>
<p>All results and graphs in this thread have been using the general, 480 participant, dataset.</p>