Binary classification model average loss not changing in training

MalachiMassey: <p>I’m training a binary classification model that takes in a list of numerical values and then classifies them based on a binary label. It’s an unbalanced dataset, about 95% 0s and about 5% 1s. During training the average loss doesn’t change at all. I have played around with the weights and I have gotten the average loss to invert, but never actually change in the training process.</p>
<p>Model:</p>
<pre><code class="lang-auto">class Network(nn.Module):

    def __init__(self, inputSize):
        super(Network, self).__init__()
        self.inputSize = inputSize
        self.fc1 = nn.Linear(inputSize, 512)
        self.fc2 = nn.Linear(512, 512)
        self.fc3 = nn.Linear(512, 1)
        
    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        x = torch.sigmoid(self.fc3(x))
        return x



model = Network(48).double().to(device)
loss_fn = nn.BCELoss().to(device) 
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)

def train(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    model.train()
    for batch, X in enumerate(dataloader):
        X, y = torch.tensor(X['Text']).to(device), X['Class'].double().to(device)

        # Compute prediction error
        pred = model(X)
        loss = loss_fn(pred, y)
        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch % 500 == 0:
            loss, current = loss.item(), batch * len(X)
            print(f"loss: {loss:&gt;7f}  [{current:&gt;5d}/{size:&gt;5d}]")

def test(dataloader, model, loss_fn):
    size = len(dataloader.dataset)
    num_batches = len(dataloader)
    model.eval()
    test_loss, correct = 0, 0
    with torch.no_grad():
        for X in dataloader:
            X, y = torch.tensor(X['Text']).to(device), X['Class'].double().to(device)
            pred = model(X)
            test_loss += loss_fn(pred, y).item()
            correct += (pred == y).type(torch.float).sum().item()
    test_loss /= num_batches
    correct /= size
    print(f"Test Error: \n Accuracy: {(100*correct):&gt;0.1f}%, Avg loss: {test_loss:&gt;8f} \n")


</code></pre>
<p>I get that some of the size calculations are a bit off, but I don’t think that changes the outputed loss value. I’ve trained up to 20 epochs (the dataset has millions of entries) and there has been no change the average loss up to all five decimal places. The accuracy sits at the percentage of 0s in the dataset.</p>

KFrank: <p>Hi Malachi!</p>
<aside class="quote no-group" data-username="MalachiMassey" data-post="1" data-topic="160832" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/malachimassey/48/53016_2.png" class="avatar"> MalachiMassey:</div>
<blockquote>
<p>It’s an unbalanced dataset, about 95% 0s and about 5% 1s. During training the average loss doesn’t change at all.</p>
</blockquote>
</aside>
<p>I would suggest first making sure that you can overfit a subset of your<br>
data.  Take a <em>balanced</em> subset (equal number of 0s and 1s) of your<br>
dataset – perhaps something like 100 or 1000 samples – and run lots<br>
of epochs on it.  You should be able to train it to get essentially 100%<br>
accuracy because the model will be learning the specific samples that<br>
it keeps seeing again and again (rather than learning the general<br>
features that it would need to predict effectively on your test set).</p>
<p>If you get that working, try overfitting an unbalanced subset (both with<br>
and without weights).</p>
<p>If you can’t get overfitting to work, you almost certainly have a bug<br>
somewhere.</p>
<aside class="quote no-group">
<blockquote>
<p>I have played around with the weights</p>
</blockquote>
</aside>
<p>I would recommend using <code>BCEWithLogitsLoss</code> instead of <code>BCELoss</code>.<br>
(If you do, you need to get rid of the final <code>sigmoid()</code> activation in<br>
your model as <code>BCEWithLogitsLoss</code> has, in effect, <code>sigmoid()</code> built in.)</p>
<p><code>BCEWithLogitsLoss</code> has better numerical stability than <code>BCELoss</code> and<br>
it also supports the <code>pos_weight</code> constructor argument that will be the<br>
most convenient way to reweight your unbalanced dataset.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">            correct += (pred == y).type(torch.float).sum().item()
</code></pre>
</blockquote>
</aside>
<p><code>(pred == y)</code> performs an <em>exact</em> equality test on floating-point numbers<br>
which (for useful values of <code>pred</code>) will fail, because two floating-point<br>
numbers that “ought” to be equal will typically differ by floating-point<br>
round-off error.  You should threshold <code>pred</code> to turn it into a binary <code>0.0</code> vs<br>
<code>1.0</code> prediction.  (If you use <code>BCEWithLogitsLoss</code> – as you should – you<br>
should threshold against <code>0.0</code>.  If you insist upon using <code>BCELoss</code>, you<br>
should threshold against <code>0.5</code>.)</p>
<aside class="quote no-group">
<blockquote>
<p>The accuracy sits at the percentage of 0s in the dataset.</p>
</blockquote>
</aside>
<p>This suggests that all (or almost all) of your predictions are (exactly) <code>0.0</code>.<br>
This would be a little unexpected, although possible, because your<br>
<code>sigmoid()</code> could be underflowing (which is part to the reason to use<br>
<code>BCEWithLogitsLoss</code> without the <code>sigmoid()</code>).</p>
<p>But, as noted above, first start with the overfitting test and see if you can<br>
get that to work.</p>
<p>Best.</p>
<p>K. Frank</p>

MalachiMassey: <p>Thanks for the help.<br>
I took a balanced sample of my dataset, switched to the BCEWithLogitsLoss and used this function:</p>
<pre><code class="lang-auto">def binary_acc(y_pred, y_test):
    y_pred_tag = torch.round(torch.sigmoid(y_pred))

    correct_results_sum = (y_pred_tag == y_test).sum().float()
    acc = correct_results_sum/y_test.shape[0]
    acc = torch.round(acc * 100)
    
    return acc
</code></pre>
<p>to calculate the accuracy.<br>
The average loss value is now changing. It fluctuates up and down and doesn’t seem to follow a trend, and the accuracy is equal to 50.0% with the balanced dataset which would seem to mean that the model is still picking just one answer.</p>

KFrank: <p>Hi Malachi!</p>
<aside class="quote no-group" data-username="MalachiMassey" data-post="3" data-topic="160832" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/malachimassey/48/53016_2.png" class="avatar"> MalachiMassey:</div>
<blockquote>
<p>switched to the BCEWithLogitsLoss</p>
</blockquote>
</aside>
<p>Just to confirm:  Did you remove the <code>sigmoid()</code> from <code>Network</code> when<br>
you switched to <code>BCEWithLogitsLoss</code>?</p>
<aside class="quote no-group">
<blockquote>
<p>The average loss value is now changing. It fluctuates up and down and doesn’t seem to follow a trend,</p>
</blockquote>
</aside>
<p>Did you try to overfit a small subset of your data?  If so, how many<br>
individual data samples did you use, how many epochs did you run,<br>
and what batch size did you use?</p>
<p>You might try increasing your learning rate (<code>lr</code>) to see if you can get<br>
your model to train faster.  (You might also try decreasing your learning<br>
rate in case your training has become unstable, but that seems unlikely.)</p>
<aside class="quote no-group">
<blockquote>
<p>the accuracy is equal to 50.0% with the balanced dataset which would seem to mean that the model is still picking just one answer.</p>
</blockquote>
</aside>
<p>Have you looked at the predictions?  Are they all just one answer?<br>
Even if your “rounded” binary predictions are all the same, do you<br>
get any significant variation in the unrounded prediction that come<br>
directly from <code>Network</code>?</p>
<p>Best.</p>
<p>K. Frank</p>

MalachiMassey: <aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="160832">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Just to confirm: Did you remove the <code>sigmoid()</code> from <code>Network</code> when<br>
you switched to <code>BCEWithLogitsLoss</code>?</p>
</blockquote>
</aside>
<p>Yes, I did remove it.</p>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="160832">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Did you try to overfit a small subset of your data? If so, how many<br>
individual data samples did you use, how many epochs did you run,<br>
and what batch size did you use?</p>
</blockquote>
</aside>
<p>I did an overfit with about 500,000 samples, fed in by batches of 1 over 100 epoches. I recorded the predictions and at the end the model was just returning vales between -2 and -4 before rounding. There did seem to be any consistency between the values, and they all round to 0.<br>
I tried increasing the learning rate and it didn’t change anything.</p>