Low loss but undesirable F-1 score! Binary classifier

username: Wookie
cooked: <p>Hi Community,</p>
<p>Thanks to the posts within this community. The following code that takes numerical inputs that are 1 x 6156 (in the range of 0 to 1) and classifies them in 2 classes [0 or 1]. With a 10 layer network I was about to get to a low loss (0.000089) but the test data gives a 60% on the F-1 score. Previous architecture had a loss of 0.002 with an F-1 score of 68%. This is counter intuitive as to why the lower loss architecture did worse. Other posts say it could be something in the code — is it possible I am doing something wrong in the testing or training?</p>
<p>Hope this code can also help others that don’t deal with images !</p>
<p>Side note: for the forward, only having the sigmoid in the first layer gave the lowest loss. I tried all other variations and it worsen the training.</p>
<pre><code class="lang-auto">X_train,X_test,y_train,y_test = train_test_split(Concatenated_x,Concatenated_y,random_state = 28,test_size=0.3151)

from torch.utils.data import Dataset, DataLoader
class Data(Dataset):
    def __init__(self, X_train, y_train):
        self.X = torch.from_numpy(X_train.astype(np.float32))
        self.y = torch.from_numpy(y_train).type(torch.LongTensor)
        self.len = self.X.shape[0]
    
    def __getitem__(self, index):
        return self.X[index], self.y[index]
    
    def __len__(self):
        return self.len

traindata = Data(np.asarray(X_train), np.asarray(y_train))

trainloader = DataLoader(traindata, batch_size=batch_size,shuffle=True, num_workers=0)
</code></pre>
<pre><code class="lang-auto">import torch.nn as nn
input_dim = 6156 #
hidden_layers = 12312 
hidden_layers2 = 6156 
hidden_layers3 = 3078 
hidden_layers4 = 1539 
hidden_layers5 = 770 
hidden_layers6 = 385 
hidden_layers7 = 192 
hidden_layers8 = 96 
hidden_layers9 = 48 
hidden_layers10 = 24 
output_dim = 2

class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.linear1 = nn.Linear(input_dim, hidden_layers)
        self.linear2 = nn.Linear(hidden_layers, hidden_layers2)
        self.linear3 = nn.Linear(hidden_layers2, hidden_layers3)
        self.linear4 = nn.Linear(hidden_layers3, hidden_layers4)
        self.linear5 = nn.Linear(hidden_layers4, hidden_layers5)
        self.linear6 = nn.Linear(hidden_layers5, hidden_layers6)
        self.linear7 = nn.Linear(hidden_layers6, hidden_layers7)
        self.linear8 = nn.Linear(hidden_layers7, hidden_layers8)
        self.linear9 = nn.Linear(hidden_layers8, hidden_layers9)
        self.linear10 = nn.Linear(hidden_layers9, hidden_layers10)
        self.linear11 = nn.Linear(hidden_layers10, output_dim )

    def forward(self, x):
        x = torch.sigmoid(self.linear1(x)) #original
        x = (self.linear2(x))
        x = (self.linear3(x))
        x = (self.linear4(x))
        x = (self.linear5(x))
        x = (self.linear6(x))
        x = (self.linear7(x))
        x = (self.linear8(x))
        x = (self.linear9(x))
        x = (self.linear10(x))
        x = (self.linear11(x))
        return x
</code></pre>
<pre><code class="lang-auto">valid_loss_min = np.Inf
path = "Model_1.pth"
clf.train()


for epoch in range(epochs):
    print("Epoch-----"+str(epoch)+"------Running")
    running_loss = 0.0
    loss_values = []
    for i, data in enumerate(trainloader, 0):
        
        
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data[0].to(device), data[1].to(device)
#         inputs, labels = data
        
        # Clear the gradients
        optimizer.zero_grad() 
        
        # Forward Pass
        outputs = clf(inputs)
        _, preds = torch.max(outputs, 1)
        labels = labels.squeeze_()
        
        # Find the Loss
        loss = criterion(outputs, labels) 
        
        # Calculate Gradients
        loss.backward() 
        
        # Update Weights
        optimizer.step() 
        
        # Calculate Loss
        running_loss += loss.item() * inputs.size(0) # Loss of batch
        
#     print(f'[{epoch + 1}, {i + 1:5d}] loss: {running_loss / len(traindata):.5f}') # Original
    print('epoch {}, loss {:.6f}'.format(epoch, loss.item()))
    
    ## save the model if validation loss has decreased
    if loss &lt;= valid_loss_min:
            print('Validation loss decreased ({:.6f} --&gt; {:.6f}). Saving model..'.format(valid_loss_min, loss))
            torch.save(clf.state_dict(), path)
            print('Model Saved')
</code></pre>
<p>Now for the testing code, it’s as follows:</p>
<pre><code class="lang-auto">clf.load_state_dict(torch.load(path))
clf.eval() #  set the dropout and batch normalization layers to evaluation mode

testdata = Data(np.asarray(X_test), np.asarray(y_test))
testloader = DataLoader(testdata, batch_size=batch_size, 
                        shuffle=False, num_workers=0)

dataiter = iter(testloader)
inputs, labels = dataiter.next()
print(inputs)
print(labels)

outputs = clf(inputs) # The outputs are energies for the 2 classes, higher energy, the more the network thinks its a class
__, predicted = torch.max(outputs, 1) # get the index of the highest energy
print(predicted)

</code></pre>
<pre><code class="lang-auto">nb_classes = 2
from sklearn.metrics import precision_recall_fscore_support as score
confusion_matrix = torch.zeros(nb_classes, nb_classes)
with torch.no_grad():
    for i, (inputs, classes) in enumerate(testloader):
        inputs = inputs
        classes = classes
        outputs = clf(inputs)
        _, preds = torch.max(outputs, 1)
        for t, p in zip(classes.view(-1), preds.view(-1)):
            confusion_matrix[t.long(), p.long()] += 1

        cm = confusion_matrix.cpu().data.numpy()
        recall = np.diag(cm) / np.sum(cm, axis = 1)
        precision = np.diag(cm) / np.sum(cm, axis = 0)
print(confusion_matrix)
print(confusion_matrix.diag()/confusion_matrix.sum(1))
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/7/a7fcfb7111bb12970137f39778fbcf279e879b52.png" data-download-href="https://discuss.pytorch.org/uploads/default/a7fcfb7111bb12970137f39778fbcf279e879b52" title="Screenshot 2022-06-14 122033"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a7fcfb7111bb12970137f39778fbcf279e879b52_2_614x500.png" alt="Screenshot 2022-06-14 122033" data-base62-sha1="nY5KsKONwSpYTUewjMxHg6vCxX4" width="614" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a7fcfb7111bb12970137f39778fbcf279e879b52_2_614x500.png, https://discuss.pytorch.org/uploads/default/original/3X/a/7/a7fcfb7111bb12970137f39778fbcf279e879b52.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/7/a7fcfb7111bb12970137f39778fbcf279e879b52.png 2x" data-dominant-color="848089"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-06-14 122033</span><span class="informations">656×534 15 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Wookie" data-post="1" data-topic="154122">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/e36b37/48.png" class="avatar"> Wookie:</div>
<blockquote>
<p>This is counter intuitive as to why the lower loss architecture did worse.</p>
</blockquote>
</aside>
<p>It seems you are dealing with an imbalanced dataset, so the loss and accuracy don’t seem to reflect the metric you care about (<code>f1_score</code>).<br>
Here is a small demonstration showing that the <a href="https://en.wikipedia.org/wiki/Accuracy_paradox">accuracy paradox</a> might indeed be misleading:</p>
<pre data-code-wrap="python"><code class="lang-python"># setup using imbalanced use case
N = 1000
y_true = np.zeros(N)
y_true[-2:] = 1.
y_pred = np.zeros(N)
y_pred[-2:] = 1.

# perfet results
print(accuracy_score(y_true, y_pred))
# 1.0
print(f1_score(y_true, y_pred))
# 1.0

# 2/1000 mismatches
y_pred = np.zeros(N)
print(accuracy_score(y_true, y_pred))
# 0.998
print(f1_score(y_true, y_pred))
# 0.0
</code></pre>
<p>so you might want to either balance the samples using a <code>WeightedRandomSampler</code> or use loss weighting.</p>

username: Wookie
cooked: <p>Thank you for the reply ptrblck, makes sense that the loss is disproportional based on the imbalanced data. For some reason Matlab gives me a 88% F1 with the same setup and imbalance and I don’t understand why I can’t get the same results here. The test data will also have this imbalance…</p>
<p>But I believe there is also something else happening since after calculating F1-Score, which shouldn’t be affected by the imbalanced data, I get the following results:</p>
<p>Loss of 0.000376 gives 68% F1<br>
Loss of 0.000089 gives 60% F1, shouldn’t the confusion matrix still show better results (less # values for FN, FP)?</p>
<p>Is this still explained by the accuracy paradox?</p>
<p>Side question, is it okay to only have the sigmoid activation in the first layer? I can’t get a definite answer. In this code it seems to give the best results for some reason…</p>
<p>Thank you for always replying! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Wookie" data-post="3" data-topic="154122">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/e36b37/48.png" class="avatar"> Wookie:</div>
<blockquote>
<p>Is this still explained by the accuracy paradox?</p>
</blockquote>
</aside>
<p>Yes, I think that can still be the case and you could manually verify it by using the confusion matrix and comparing different metrics to the loss calculation.<br>
Depending on the used criterion the outputs would also represent how “sure” the model is about the prediction and would of course yield a larger loss for wrong predictions with a high confidence value while the accuracy could stay the same.</p>
<aside class="quote no-group" data-username="Wookie" data-post="3" data-topic="154122">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/e36b37/48.png" class="avatar"> Wookie:</div>
<blockquote>
<p>Side question, is it okay to only have the sigmoid activation in the first layer? I can’t get a definite answer. In this code it seems to give the best results for some reason…</p>
</blockquote>
</aside>
<p>I don’t know and haven’t seen a lot of <code>sigmoid</code> activations in common models, but if this activation function works fine for your use case it sounds like a reasonable choice.</p>

username: Wookie
cooked: <p>I balanced the data (1:1) and it gave me great results on training but actual data that is unbalanced, the results are 20% (F1-score).</p>
<p>I tried using the <code>WeightedRandomSampler</code>, is this the right implementation?</p>
<pre><code class="lang-auto">labels = np.array(traindata)[:,1] # turn to array and take all of column index 1 which are the labels

labels = labels.astype(int) # change to int

num_of_majority_class_training_examples = 121087 # Hard code number of 0's
num_of_minority_class_training_examples =2817 # Hard code number of 1's

majority_weight = 1/num_of_majority_class_training_examples

minority_weight = 1/num_of_minority_class_training_examples

sample_weights = np.array([majority_weight, minority_weight]) # This is assuming that your minority class is the integer 1 in the labels object. If not, switch places so it's minority_weight, majority_weight.

weights = sample_weights[labels] # this goes through each training example and uses the labels 0 and 1 as the index in sample_weights object which is the weight you want for that class.

sampler = WeightedRandomSampler(weights=weights, num_samples= len(traindata), replacement=True)

trainloader = DataLoader(traindata, batch_size = batch_size, sampler=sampler)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/4/44c7da7b013a99431b3b6e978f8e1704e0bf50ba.png" data-download-href="https://discuss.pytorch.org/uploads/default/44c7da7b013a99431b3b6e978f8e1704e0bf50ba" title="Training"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44c7da7b013a99431b3b6e978f8e1704e0bf50ba_2_517x500.png" alt="Training" data-base62-sha1="9OsEqkal731D4lxk29wDCxaXgjU" width="517" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44c7da7b013a99431b3b6e978f8e1704e0bf50ba_2_517x500.png, https://discuss.pytorch.org/uploads/default/original/3X/4/4/44c7da7b013a99431b3b6e978f8e1704e0bf50ba.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/4/44c7da7b013a99431b3b6e978f8e1704e0bf50ba.png 2x" data-dominant-color="ABA4A7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Training</span><span class="informations">535×517 14.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Test set is still not great. I may be coding wrong?</p>

username: ptrblck
cooked: <p>The balancing looks correct and the training results seem also to reflect it.<br>
Is the validation set still overfitting to the majority class or what does the confusion matrix show?</p>

username: Wookie
cooked: <p>The Validation results are odd, but the confusion matrix for the new dataset is okay, it would be ideal if I can improve on these results…</p>
<p>This is the training</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/1/91594751eda1d0896705ccf5acf101e7bc289336.png" data-download-href="https://discuss.pytorch.org/uploads/default/91594751eda1d0896705ccf5acf101e7bc289336" title="Training"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/91594751eda1d0896705ccf5acf101e7bc289336_2_527x500.png" alt="Training" data-base62-sha1="kJOvJ3hsjx7iackDyoiUqLciMDQ" width="527" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/91594751eda1d0896705ccf5acf101e7bc289336_2_527x500.png, https://discuss.pytorch.org/uploads/default/original/3X/9/1/91594751eda1d0896705ccf5acf101e7bc289336.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/1/91594751eda1d0896705ccf5acf101e7bc289336.png 2x" data-dominant-color="B1AAAC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Training</span><span class="informations">564×535 15.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is the test</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/1/31fd4071ef262a6dda39818a352b84efa2c09e4d.png" data-download-href="https://discuss.pytorch.org/uploads/default/31fd4071ef262a6dda39818a352b84efa2c09e4d" title="Test"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/31fd4071ef262a6dda39818a352b84efa2c09e4d_2_579x500.png" alt="Test" data-base62-sha1="78dZ5SfJs8X3vLSxkiTB2SvWZFb" width="579" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/31fd4071ef262a6dda39818a352b84efa2c09e4d_2_579x500.png, https://discuss.pytorch.org/uploads/default/original/3X/3/1/31fd4071ef262a6dda39818a352b84efa2c09e4d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/1/31fd4071ef262a6dda39818a352b84efa2c09e4d.png 2x" data-dominant-color="938F97"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Test</span><span class="informations">613×529 15 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Validation</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/2/e2ede5769f99c2d4a81cdc146a6d2b3b7a325253.png" data-download-href="https://discuss.pytorch.org/uploads/default/e2ede5769f99c2d4a81cdc146a6d2b3b7a325253" title="Validation"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2ede5769f99c2d4a81cdc146a6d2b3b7a325253_2_635x500.png" alt="Validation" data-base62-sha1="wnvB86mrQYLQYi9pIIOcc9kb36b" width="635" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2ede5769f99c2d4a81cdc146a6d2b3b7a325253_2_635x500.png, https://discuss.pytorch.org/uploads/default/original/3X/e/2/e2ede5769f99c2d4a81cdc146a6d2b3b7a325253.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/2/e2ede5769f99c2d4a81cdc146a6d2b3b7a325253.png 2x" data-dominant-color="A59BA5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Validation</span><span class="informations">677×533 13.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>New Data</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b29c757861652ad8aa2c3e757a2ed934267defbe.png" data-download-href="https://discuss.pytorch.org/uploads/default/b29c757861652ad8aa2c3e757a2ed934267defbe" title="NewData"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b29c757861652ad8aa2c3e757a2ed934267defbe_2_584x500.png" alt="NewData" data-base62-sha1="pu4ctRVnvFgrkDlv0QnA0xcPxdc" width="584" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b29c757861652ad8aa2c3e757a2ed934267defbe_2_584x500.png, https://discuss.pytorch.org/uploads/default/original/3X/b/2/b29c757861652ad8aa2c3e757a2ed934267defbe.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/2/b29c757861652ad8aa2c3e757a2ed934267defbe.png 2x" data-dominant-color="938F96"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">NewData</span><span class="informations">615×526 14.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>