Binary classification on MNIST: loss and accuracies remain costant

CasellaJr: <p>I am trying to do binary classification on MNIST dataset. Class 0 for even numbers and class 1 for odd numbers. I am using a simplified version of VGG.<br>
My NN has a loss and an accuracy that remain costant.<br>
I want to say that my model, reached to over 90% of accuracy before of changing targets into binary targets, so probably there is something wrong.<br>
Here I change the targets into binary:</p>
<pre><code class="lang-auto">for i in range(10):
  idx = (train_set.targets==i)
  if (i == 0) or ((i % 2) == 0): train_set.targets[idx] = 0

  else: train_set.targets[idx] = 1

for i in range(10):
  idx = (test_set.targets==i)
  if (i == 0) or ((i % 2) == 0): test_set.targets[idx] = 0

  else: test_set.targets[idx] = 1
</code></pre>
<p>This is my net:</p>
<pre><code class="lang-auto">class VGG16(nn.Module):

    def __init__(self, num_classes):
        super(VGG16, self).__init__()

        # calculate same padding:
        # (w - k + 2*p)/s + 1 = o
        # =&gt; p = (s(o-1) - w + k)/2

        self.block_1 = nn.Sequential(
            nn.Conv2d(in_channels=1,
                      out_channels=64,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      # (1(32-1)- 32 + 3)/2 = 1
                      padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.Conv2d(in_channels=64,
                      out_channels=64,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=(2, 2),
                         stride=(2, 2))
        )

        self.block_2 = nn.Sequential(
            nn.Conv2d(in_channels=64,
                      out_channels=128,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.Conv2d(in_channels=128,
                      out_channels=128,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=(2, 2),
                         stride=(2, 2))
        )
        
        self.block_3 = nn.Sequential(
            nn.Conv2d(in_channels=128,
                      out_channels=256,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(),
            nn.Conv2d(in_channels=256,
                      out_channels=256,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(),
            nn.Conv2d(in_channels=256,
                      out_channels=256,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=(2, 2),
                         stride=(2, 2))
        )

        self.block_4 = nn.Sequential(
            nn.Conv2d(in_channels=256,
                      out_channels=512,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(),
            nn.Conv2d(in_channels=512,
                      out_channels=512,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(),
            nn.Conv2d(in_channels=512,
                      out_channels=512,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=(2, 2),
                         stride=(2, 2))
        )            

        self.classifier = nn.Sequential(
            nn.Linear(2048, 4096),
            nn.ReLU(True),
            nn.Dropout(p=0.65),
            nn.Linear(4096, 4096),
            nn.ReLU(True),
            nn.Dropout(p=0.65),
            nn.Linear(4096, num_classes),
            nn.Sigmoid() 
        )

        for m in self.modules():
            if isinstance(m, torch.nn.Conv2d) or isinstance(m, torch.nn.Linear):
                nn.init.kaiming_uniform_(m.weight, mode='fan_in', nonlinearity='leaky_relu')
#                 nn.init.xavier_normal_(m.weight)
                if m.bias is not None:
                    m.bias.detach().zero_()

        # self.avgpool = nn.AdaptiveAvgPool2d((7, 7))

    def forward(self, x):

        x = self.block_1(x)
        x = self.block_2(x)
        x = self.block_3(x)
        x = self.block_4(x)
        # x = self.avgpool(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x
        #logits = self.classifier(x)
        #probas = F.softmax(logits, dim=1)
        # probas = nn.Softmax(logits)
        #return probas
        # return logits
</code></pre>
<pre><code class="lang-auto"># Define an optimizier
import torch.optim as optim
optimizer = optim.SGD(model.parameters(), lr = 0.01)
# Define a loss 
criterion = nn.BCELoss()


def train(net, loaders, optimizer, criterion, epochs=20, dev=dev, save_param = False, model_name="valerio"):
    try:
        net = net.to(dev)
        #print(net)
        # Initialize history
        history_loss = {"train": [], "val": [], "test": []}
        history_accuracy = {"train": [], "val": [], "test": []}
        # Store the best val accuracy
        best_val_accuracy = 0

        # Process each epoch
        for epoch in range(epochs):
            # Initialize epoch variables
            sum_loss = {"train": 0, "val": 0, "test": 0}
            sum_accuracy = {"train": 0, "val": 0, "test": 0}
            # Process each split
            for split in ["train", "val", "test"]:
                if split == "train":
                  net.train()
                else:
                  net.eval()
                # Process each batch
                for (input, labels) in loaders[split]:
                    # Move to CUDA
                    input = input.to(dev)
                    labels = labels.to(dev)
                    # Reset gradients
                    optimizer.zero_grad()
                    # Compute output
                    pred = net(input)
                    labels = labels.unsqueeze(1)
                    labels = labels.float()
                    loss = criterion(pred, labels)
                    # Update loss
                    sum_loss[split] += loss.item()
                    # Check parameter update
                    if split == "train":
                        # Compute gradients
                        loss.backward()
                        # Optimize
                        optimizer.step()
                    # Compute accuracy
                    _,pred_labels = pred.max(1)
                    batch_accuracy = (pred_labels == labels).sum().item()/input.size(0)
                    # Update accuracy
                    sum_accuracy[split] += batch_accuracy
            # Compute epoch loss/accuracy
            epoch_loss = {split: sum_loss[split]/len(loaders[split]) for split in ["train", "val", "test"]}
            epoch_accuracy = {split: sum_accuracy[split]/len(loaders[split]) for split in ["train", "val", "test"]}

            # Store params at the best validation accuracy
            if save_param and epoch_accuracy["val"] &gt; best_val_accuracy:
              #torch.save(net.state_dict(), f"{net.__class__.__name__}_best_val.pth")
              torch.save(net.state_dict(), f"{model_name}_best_val.pth")
              best_val_accuracy = epoch_accuracy["val"]

            # Update history
            for split in ["train", "val", "test"]:
                history_loss[split].append(epoch_loss[split])
                history_accuracy[split].append(epoch_accuracy[split])
            # Print info
            print(f"Epoch {epoch+1}:",
                  f"TrL={epoch_loss['train']:.4f},",
                  f"TrA={epoch_accuracy['train']:.4f},",
                  f"VL={epoch_loss['val']:.4f},",
                  f"VA={epoch_accuracy['val']:.4f},",
                  f"TeL={epoch_loss['test']:.4f},",
                  f"TeA={epoch_accuracy['test']:.4f},")
    except KeyboardInterrupt:
        print("Interrupted")
    finally:
        # Plot loss
        plt.title("Loss")
        for split in ["train", "val", "test"]:
            plt.plot(history_loss[split], label=split)
        plt.legend()
        plt.show()
        # Plot accuracy
        plt.title("Accuracy")
        for split in ["train", "val", "test"]:
            plt.plot(history_accuracy[split], label=split)
        plt.legend()
        plt.show()
</code></pre>
<p>From the previous model of digit recognition i changed only the targets, and the final layer of classifier from 10 classes to 1 class + Sigmoid. And i changed also cross entropy to BCELoss. What I am doing wrong?</p>
<p>These are loss and accuracy values:</p>
<pre><code class="lang-auto">Epoch 1: TrL=49.0955, TrA=31.4211, VL=49.7285, VA=31.7340, TeL=49.2635, TeA=31.3758,
Epoch 2: TrL=49.0992, TrA=31.4235, VL=49.7285, VA=31.7340, TeL=49.2635, TeA=31.3758,
Epoch 3: TrL=49.0899, TrA=31.4176, VL=49.7285, VA=31.7340, TeL=49.2635, TeA=31.3758,
Epoch 4: TrL=49.0936, TrA=31.4199, VL=49.7285, VA=31.7340, TeL=49.2635, TeA=31.3758,
Epoch 5: TrL=49.0936, TrA=31.4199, VL=49.7285, VA=31.7340, TeL=49.2635, TeA=31.3758,
Epoch 6: TrL=49.0825, TrA=31.4128, VL=49.7285, VA=31.7340, TeL=49.2635, TeA=31.3758,
</code></pre>
<p>What’s wrong? How is it possible that with 10 classes I reached over 90% accuracy, and with a simplified version, only 2 classes, I reach 30% of accuracy?</p>
<p>Edit: increasing batch size from 64 to 128, accuracy reaches to 60% and remains constant…</p>

KFrank: <p>Hi Bruno!</p>
<p>I do see what appears to be an error in your conversion from a<br>
multi-class to a binary model (although I can’t explain the entirety<br>
of your results).</p>
<p>Specifically, I believe your use of <code>argmax()</code> to convert your probabilistic<br>
predictions into “hard” yes-no binary predictions is incorrect, and will<br>
always yield <code>0</code>.</p>
<aside class="quote no-group quote-modified" data-username="CasellaJr" data-post="1" data-topic="119370" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/50afbb/48.png" class="avatar"> CasellaJr:</div>
<blockquote>
<p>This is my net:</p>
<pre><code class="lang-auto">class VGG16(nn.Module):

    def __init__(self, num_classes):
        ...
        self.classifier = nn.Sequential(
            ...
            nn.Linear(4096, num_classes),
            nn.Sigmoid() 
        )
</code></pre>
</blockquote>
</aside>
<p>When you create your <code>net</code>, presumably by instantiating your <code>VGG16</code>,<br>
you don’t show us what value you use for <code>num_classes</code>.  I assume<br>
that you use <code>num_classes = 1</code>.  (Otherwise I would expect that<br>
<code>BCELoss</code> would throw an error when you call it.)</p>
<p>Therefore I assume that the output of your model has shape<br>
<code>[nBatch, 1]</code>.  Note, even though you are performing a binary<br>
classification, I assume that you do <em>not</em> use <code>num_classes = 2</code><br>
and that your output does <em>not</em> have shape <code>[nBatch, 2]</code>.</p>
<p>(In the case of a ten-class classifier, your model output would have<br>
shape <code>[nBatch, 10]</code>.)</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">                    pred = net(input)
                    labels = labels.unsqueeze(1)
                    labels = labels.float()
                    loss = criterion(pred, labels)
</code></pre>
</blockquote>
</aside>
<p>Here I believe that <code>labels</code> starts out having shape <code>[nBatch]</code>, with no<br>
class dimension, as would be appropriate for a multi-class problem<br>
using <code>CrossEntropyLoss</code>.  However, <code>BCELoss</code> requires that <code>pred</code><br>
and <code>labels</code> have the same shape, and I believe that <code>pred</code> has<br>
shape <code>[nBatch, 1]</code> (where the singleton dimension is not necessary,<br>
but is also not incorrect).</p>
<p>That is, <code>pred</code> has shape <code>[nBatch, num_classes]</code>, which is appropriate<br>
for both a multi-class problem with <code>CrossEntropyLoss</code> and for a plain<br>
binary problem (or a multi-label, multi-class problem) with <code>BCELoss</code>,<br>
with, in this case, <code>num_classes = 1</code>.</p>
<p>Therefore you <code>unsqueeze()</code> <code>labels</code> in order that <code>labels</code> have the<br>
same shape as <code>pred</code>, namely, <code>[nBatch, 1]</code>.</p>
<p>So far, so good.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">                    _,pred_labels = pred.max(1)
                    batch_accuracy = (pred_labels == labels).sum().item()/input.size(0)
</code></pre>
</blockquote>
</aside>
<p>Just for clarity, note that <code>_,pred_labels = pred.max(1)</code> is the same<br>
as <code>pred_labels = pred.argmax(1)</code>.</p>
<p>So your predicted class label is the index of the largest predicted<br>
value along the class dimension.  <code>pred_labels</code> takes on integer<br>
values in <code>[0, num_classes - 1]</code> (inclusive).  This is correct for<br>
the multi-class case, but in the binary case (implemented with<br>
<code>num_classes = 1</code>), <code>pred_labels</code> will always have value <code>0</code>.</p>
<p>So the predictions you use for computing your accuracy are always<br>
predicting “0”.  (Note that this issue does not affect the predictions<br>
you use for computing your loss.  As far as I can tell, your loss should<br>
be correct and your training should work.)</p>
<p>What’s odd here is that your computed accuracy of about 31% suggests<br>
that your dataset is rather unbalanced.  If my theory here is right, this<br>
means that your dataset consists of about 31% even-number digits,<br>
and about 69% odd-number digits.  This isn’t impossible, but it does<br>
seem a bit peculiar.</p>
<aside class="quote no-group">
<blockquote>
<p>Edit: increasing batch size from 64 to 128, accuracy reaches to 60% and remains constant…</p>
</blockquote>
</aside>
<p>That your computed accuracy depends on your batch size is very<br>
strange.  I don’t see any batch-size-dependent error in your accuracy<br>
calculation.  You don’t show us your <code>loaders</code>, so one could speculate<br>
that something weird is going on there.</p>
<p>A couple of minor comments:</p>
<p>First, when you get this otherwise working, you should consider<br>
using <code>BCEWithLogitsLoss</code> and getting rid of the final <code>Sigmoid</code><br>
layer in your <code>classifier</code> block.  <code>BCELoss</code> with the <code>Sigmoid</code> and<br>
<code>BCEWithLogitsLoss</code> without are mathematically equivalent, but<br>
<code>BCEWithLogitsLoss</code> is numerically more stable.</p>
<p>Second, this following code is not incorrect, but it’s unnecessarily<br>
complicated:</p>
<aside class="quote no-group">
<blockquote>
<p>Here I change the targets into binary:</p>
<pre><code class="lang-auto">for i in range(10):
  idx = (train_set.targets==i)
  if (i == 0) or ((i % 2) == 0): train_set.targets[idx] = 0

  else: train_set.targets[idx] = 1
</code></pre>
</blockquote>
</aside>
<p>This all can be simplified down to:</p>
<pre data-code-wrap="python"><code class="lang-python">train_set.targets = train_set.targets % 2
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

CasellaJr: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="119370">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Hi Bruno!</p>
<p>I do see what appears to be an error in your conversion from a<br>
multi-class to a binary model (although I can’t explain the entirety<br>
of your results).</p>
<p>Specifically, I believe your use of <code>argmax()</code> to convert your probabilistic<br>
predictions into “hard” yes-no binary predictions is incorrect, and will<br>
always yield <code>0</code>.</p>
</blockquote>
</aside>
<p>Thank you for the answer, I think you are right.<br>
Obviously when I create the model the number of classes is =1, as you said. I changed the loss from BCELoss to BCEWithLogitsLoss, and I removed Sigmoid, following your advice.<br>
My loaders are these:</p>
<pre><code class="lang-auto"># Define dictionary of loaders
loaders = {"train": train_loader,
           "val": val_loader,
           "test": test_loader}
</code></pre>
<p>I don’t know that change in batch_size changed the results…<br>
However, what do you suggest?<br>
Do you think is correct writing this?</p>
<pre><code class="lang-auto">pred_labels = pred.argmax(1)+1
</code></pre>
<p>Edit: no, this solution I proposed is wrong, in this way I will obtain always 1. I don’t know how can I fix…<br>
Can I just remove argmax/max and use simply the output?</p>

CasellaJr: <p>When you said:</p>
<blockquote>
<p>Blockquote<br>
Here I believe that <code>labels</code> starts out having shape <code>[nBatch]</code>, with no<br>
class dimension, as would be appropriate for a multi-class problem<br>
using <code>CrossEntropyLoss</code>. However, <code>BCELoss</code> requires that <code>pred</code><br>
and <code>labels</code> have the same shape, and I believe that <code>pred</code> has<br>
shape <code>[nBatch, 1]</code> (where the singleton dimension is not necessary,<br>
but is also not incorrect).</p>
</blockquote>
<p>I tried to change this, using:</p>
<pre><code class="lang-auto">pred = pred.squeeze(dim=1) # Output shape is [Batch size, 1], but we want [Batch size]
</code></pre>
<p>But it returned an error that the shape must be the same, so I removed, avoiding other problems <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<p>Now, I did this:</p>
<pre><code class="lang-auto"># Compute accuracy
#pred_labels = pred.argmax(1) + 1
pred_labels = (pred &gt;= 0.5).long() # Binarize predictions to 0 and 1
batch_accuracy = (pred_labels == labels).sum().item()/input.size(0)
</code></pre>
<p>And now, accuracy is over 97% <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_three_hearts.png?v=12" title=":smiling_face_with_three_hearts:" class="emoji" alt=":smiling_face_with_three_hearts:" loading="lazy" width="20" height="20"><br>
Do you think is a good solution?</p>

KFrank: <p>Hi Bruno!</p>
<aside class="quote no-group quote-modified" data-username="CasellaJr" data-post="4" data-topic="119370" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/50afbb/48.png" class="avatar"> CasellaJr:</div>
<blockquote>
<pre><code class="lang-auto"># Compute accuracy
pred_labels = (pred &gt;= 0.5).long() # Binarize predictions to 0 and 1
batch_accuracy = (pred_labels == labels).sum().item()/input.size(0)
</code></pre>
<p>And now, accuracy is over 97% <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_three_hearts.png?v=12" title=":smiling_face_with_three_hearts:" class="emoji" alt=":smiling_face_with_three_hearts:" loading="lazy" width="20" height="20"><br>
Do you think is a good solution?</p>
</blockquote>
</aside>
<p>This potentially makes sense.  Let me offer a few words of explanation:</p>
<p>You have a binary classifier that has a <em>single</em> output (per batch<br>
sample).  If this value is the output of your final <code>Linear</code> layer,<br>
without a <code>Sigmoid</code>, it should be understood as a raw-score<br>
<em>logit</em> that runs from <code>-inf</code> to <code>inf</code> and you should feed it into<br>
<code>BCEWithLogitsLoss</code> (preferred).  If you have a <code>Sigmoid</code> then<br>
you are converting the logit to a probability in <code>[0.0, 1.0]</code> and<br>
you should feed it into <code>BCELoss</code> (not preferred).</p>
<p>This probability (after the <code>Sigmoid</code>) is your predicted probability of<br>
your sample being in the “1” or “yes” class.  The probability of your<br>
sample being in the “0” or “no” class is one minus the “1”-class<br>
probability.</p>
<p>How do you covert such a probabilistic prediction into a hard yes-no<br>
prediction?  A reasonable and straightforward approach is to say that<br>
if your predicted probability (for your “1” class) is greater than <code>1/2</code>,<br>
then you will make a hard prediction of your sample being in the<br>
“1” class.  Conversely, if your predicted probability is less than <code>1/2</code>,<br>
you make the “0”-class prediction.</p>
<p>So your <code>(pred &gt;= 0.5).long()</code> is thresholding  your probabilistic<br>
prediction, <code>pred</code>, against <code>1/2</code> to get a hard prediction.  This is correct<br>
if <code>pred</code> is a probability (that is, if you have a <code>Sigmoid</code> after your final<br>
<code>Linear</code> layer), but is imperfect if <code>pred</code> is a logit (that is, no <code>Sigmoid</code>).</p>
<p>What should you do if <code>pred</code> is a logit?  Note that <code>Sigmoid</code> maps logits<br>
that are negative to probabilities that are less than <code>1/2</code> (and positive<br>
logits to probabilities that are greater than <code>1/2</code>).  So thresholding the<br>
logit against <code>0</code> is equivalent to thresholding the corresponding<br>
probability against <code>1/2</code>.</p>
<p>Therefore if your <code>pred</code> is a logit you should tweak your code to read<br>
<code>(pred &gt;= 0.0).long()</code>.</p>
<p>As noted before, <code>BCEWithLogitsLoss</code> is numerically more stable<br>
than <code>BCELoss</code>, so you are better of using <code>BCEWithLogitsLoss</code><br>
without <code>Sigmoid</code>, having your predictions be logits, and therefore<br>
thresholding your predictions against <code>0</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

CasellaJr: <p>Thank you very much for all these exhaustive answers!!</p>