High train accuracy but zero validation accuracy for imbalanced dataset

Mahmoud_Gad: <p>I’m working on a binary classification neural network with severe class imbalance (33130 zeros and 415 ones). While training my network, i test the accuracy for each epoch and it starts with 40%, reaches to 85% after 30 epochs. I’m using F1 score, recall and precision to measure the accuracy. the results vary a lot when i change the batch size of the dataloader.</p>
<p>when i feed the validation dataloader to the model however, accuracy plummets to 0%.</p>
<p>I don’t have the slightest idea what would be causing this, so ill try to include only essential information in this post while giving as much details as possible.</p>
<p>I started by splitting the dataset into features and labels, I then imputed the features, scaled it and reduced it using my SimpleImputer, StandardScaler and PCA objects</p>
<p>I then split the data using train_test_split</p>
<pre><code class="lang-auto">x_train, x_val, y_train, y_val = train_test_split(reduced_features, 
                                                  labels, 
                                                  test_size = 0.33, 
                                                  random_state = 0)
</code></pre>
<p>I used WeightedRandomSampler to deal with class imbalance while following the guide in this article: <a href="https://towardsdatascience.com/pytorch-basics-sampling-samplers-2a0f29f0bf2a" rel="nofollow noopener">PyTorch [Basics] — Sampling</a></p>
<pre><code class="lang-auto">target_list = torch.tensor(y_train.values)
target_list = target_list[torch.randperm(len(target_list))]

class_count = y_train.value_counts()
class_weights = 1./torch.tensor(class_count, dtype=torch.float)

class_weights_all = class_weights[target_list]

weighted_sampler = WeightedRandomSampler(
    weights=class_weights_all,
    num_samples=len(class_weights_all),
    replacement=True
)
</code></pre>
<p>I created the following class to pass an object containing features and labels to the dataloader:</p>
<pre><code class="lang-auto">class Data_for_loader(Dataset):
    
    def __init__(self, X_data, y_data):
        self.X_data = X_data
        self.y_data = y_data
        
    def __getitem__(self, index):
        return self.X_data[index], self.y_data[index]
        
    def __len__ (self):
        return len(self.X_data)


train_data = Data_for_loader(torch.FloatTensor(x_train.values), 
                             torch.FloatTensor(y_train.values))

validation_data = Data_for_loader(torch.FloatTensor(x_val.values),
                            torch.FloatTensor(y_val.values))
</code></pre>
<p>I then created the dataloader objects:</p>
<pre><code class="lang-auto">train_loader = DataLoader(dataset = train_data, shuffle=False, batch_size = 150, sampler=weighted_sampler)
validation_loader = DataLoader(dataset = validation_data, batch_size= 1)
</code></pre>
<p>since this is a binary classification problem I’m using BCEWithLogitsLoss as my criterion</p>
<p>I’m using Adam as my optimizer</p>
<p>I created the following class for my model:</p>
<pre><code class="lang-auto">class BinaryClassifier(nn.Module):
    def __init__(self):
        super(BinaryClassifier, self).__init__()

        self.h1 = nn.Linear(140, 90) 
        self.h2 = nn.Linear(90, 50)
        self.output = nn.Linear(50, 1) 
        
        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(p=0.2)
        self.batchnorm1 = nn.BatchNorm1d(90)
        self.batchnorm2 = nn.BatchNorm1d(50)
        
    def forward(self, x):
        x = self.h1(x)
        x = self.relu(x)
        x = self.batchnorm1(x)
        
        x = self.h2(x)
        x = self.relu(x)
        x = self.batchnorm2(x)

        x = self.dropout(x)
        x = self.output(x)
        
        return x
</code></pre>
<p>I used the following code to train the model:</p>
<pre><code class="lang-auto">model.train()

for e in range(epochs):
    
    epoch_loss = 0   
    
    recall_total_score = 0  
    precision_total_score = 0
    f1_total_score = 0
    
    for inputs, labels in train_loader:
        inputs, labels = inputs.to(device), labels.to(device)
        optimizer.zero_grad()
        
        output = model.forward(inputs)
        loss = criterion(output, labels.unsqueeze(1))
        
        #rounding output for accuracy measure
        output = torch.round(torch.sigmoid(output))
        output = output.cpu().detach().numpy()
        labels = labels.cpu().unsqueeze(1)
        
        #measuring accuracy
        recall_total_score += recall_score(labels, output)
        f1_total_score += f1_score(labels, output)
        precision_total_score += precision_score(labels, output)
        
        
        loss.backward()
        optimizer.step()
        
        epoch_loss += loss.item()
</code></pre>
<p>finaly i used the following code to evaluate the model:</p>
<pre><code class="lang-auto">y_pred = []
y_true = []

model.eval()
with torch.no_grad():
    for inputs, labels in validation_loader:
        inputs = inputs.to(device)
        
        output = model.forward(inputs)
        output = torch.round(torch.sigmoid(output))

        y_pred.extend(output.cpu().numpy())
        y_true.extend(labels.cpu().numpy())

f1 = f1_score(y_true, y_pred)
</code></pre>
<p>I’m relatively new to Pytorch, my only guess is that the model is overfitting, even tho I’m using dropout in my model, or i’m somehow making a mistake while dealing with the validation set.</p>
<p>Thanks for your help!</p>

ptrblck: <p>It seems you are shuffling the targets here:</p>
<pre><code class="lang-python">target_list = target_list[torch.randperm(len(target_list))]
</code></pre>
<p>but I cannot see any code, which applies the same shuffling to the data.<br>
Are you sure you are not breaking the correspondence between each data sample and the target?</p>

Mahmoud_Gad: <p>you are right! i removed the shuffle on the targets. after doing that the model’s accuracy reaches 80% after 4 epochs only, unfortunately issue persists with the validation set tho, still about 0.05% at best. after training the model on a different range of epochs (10-50) i evaluated it on a new dataloader object that doesn’t include the WeightedRandomSampler, I inserted the training data into the dataloader to check if the model trained properly.</p>
<pre><code class="lang-auto">train_loader2 = DataLoader(dataset = train_data, shuffle=True, batch_size = 20)
</code></pre>
<p>it received good results on the training dataset, that means the model successfully trained, but since it has 0% accuracy on the validation set it is definitely overfitting. could it be the difference in weights is so large that its forcing the model to overfit very quickly? my learning rate is 0.005</p>
<p>the strange thing is i fit the data to a random forest classifier, which should be very resistant to overfitting, but the same issue happens, high training accuracy 0 validation accuracy.</p>

ptrblck: <p>A 0 validation accuracy seems a bit weird, as I would at least assume your model should get the “random accuracy”. I.e. even if it’s overfitting to a specific class, it should still get some samples right, no?</p>
<p>Could you check the labels of your validation set and compare the model predictions to it?<br>
Could these labels be somehow inverted?</p>

Mahmoud_Gad: <p>I double checked and they are not inverted, the true targets contain 137 ones and 10933 zeros, the predicted targets contain 41 ones and the rest are zeros. that is the reason i’m not getting the random accuracy, my recall and precision are so low. if i use accuracy_score as my metric i do get 98% accuracy but that’s because most of the targets are 0 anyway.</p>
<p>I plotted the loss and accuracy with and without WeightedRandomSampler.</p>
<p>the top two plots show the result without WRS, the bottom two show the result with WRS:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff2d96e03d11f8a66a4fa223427d50319852139a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ff2d96e03d11f8a66a4fa223427d50319852139a" title="imgonline-com-ua-twotoone-dq71ra5Npmb4bW"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff2d96e03d11f8a66a4fa223427d50319852139a_2_600x500.jpeg" alt="imgonline-com-ua-twotoone-dq71ra5Npmb4bW" data-base62-sha1="AppqDPkW4Ht7iR9TSlkcAG2ekVc" width="600" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff2d96e03d11f8a66a4fa223427d50319852139a_2_600x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff2d96e03d11f8a66a4fa223427d50319852139a_2_900x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff2d96e03d11f8a66a4fa223427d50319852139a_2_1200x1000.jpeg 2x" data-dominant-color="FCFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">imgonline-com-ua-twotoone-dq71ra5Npmb4bW</span><span class="informations">1356×1130 108 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Thanks for the update. Your model is apparently still overfitting, since the validation loss increases immediately after the training begins, so you might need to either increase the oversampling of the minority classes or add a class weighting in your criterion.</p>

Mahmoud_Gad: <p>adjusted pos_weight and tried using imblearn oversampling, didn’t seem to do the trick, However, i changed the scaler to MinMax scaler instead of standard scaler, and instead of imputing the data with the mean i just straight up removed all rows with any missing data, that seems to have raised the f1 score to 0.45 and my ROC AUC score to 0.67!! it seems i still need to work on the data cleaning somehow to get the accuracy even higher. this has been a very good learning experience tho.</p>
<p>ptrblck, Thank you so much for your time and your suggestions, you have been very kind. once i reach better accuracy ill put an update <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>