The accuracy of the convolutional neural network stays the same when the criterion is selected as CrossEntropyLoss

talhak: <p>As the title clearly describes, the  <code>accuracy</code>  of my  <code>CNN</code>  stays the same when the  <code>criterion</code>  is selected as  <code>CrossEntropyLoss</code>. I especially selected  <code>CrossEntropyLoss</code>  since only it achieves the  <code>test loss</code>  close to the  <code>training loss</code>. No issues at all for the other loss functions.</p>
<p>Here is the overview of the constructed  <code>CNN</code>  model:</p>
<pre><code class="lang-auto">MyNet(
  (activation_fn): ReLU(inplace)
  (conv1): Sequential(
    (0): Conv2d(3, 16, kernel_size=(5, 5), stride=(1, 1))
    (1): ReLU(inplace)
    (2): Dropout2d(p=0.5)
  )
  (conv2): Sequential(
    (0): Conv2d(16, 32, kernel_size=(5, 5), stride=(1, 1))
    (1): ReLU(inplace)
    (2): Dropout2d(p=0.5)
    (3): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (conv3): Sequential(
    (0): Conv2d(32, 64, kernel_size=(5, 5), stride=(1, 1))
    (1): ReLU(inplace)
    (2): Dropout2d(p=0.5)
  )
  (conv4): Sequential(
    (0): Conv2d(64, 128, kernel_size=(5, 5), stride=(1, 1))
    (1): ReLU(inplace)
    (2): Dropout2d(p=0.5)
  )
  (conv5): Sequential(
    (0): Conv2d(128, 256, kernel_size=(5, 5), stride=(1, 1))
    (1): ReLU(inplace)
    (2): Dropout2d(p=0.5)
    (3): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (fc1): Sequential(
    (0): Linear(in_features=4096, out_features=1600, bias=True)
    (1): Dropout2d(p=0.5)
  )
  (fc2): Sequential(
    (0): Linear(in_features=1600, out_features=400, bias=True)
    (1): Dropout2d(p=0.5)
  )
  (fc3): Sequential(
    (0): Linear(in_features=400, out_features=100, bias=True)
    (1): Dropout2d(p=0.5)
  )
  (fc4): Sequential(
    (0): Linear(in_features=100, out_features=8, bias=True)
  )
)
</code></pre>
<p>Here is my  <code>test</code>  function:</p>
<pre><code class="lang-auto">def test():
    model.eval()
    test_loss = 0
    correct = 0
    with torch.no_grad():
        for data, target in test_loader:
            data, target = data.to(device), target.to(device)

            output = model(data)
            output = output.to(device)
            test_loss += criterion(output, target).item()
            _, predicted = torch.max(output.data, 1)
            correct += (predicted == target).sum().item()

    test_loss /= math.ceil((len(test_loader.dataset) / test_batch_size))
    test_losses.append(test_loss)

    acc = 100. * correct / len(test_loader.dataset)
    print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.2f}%)\n'.format(
        test_loss, correct, len(test_loader.dataset), acc))
</code></pre>

ptrblck: <p>Which other loss functions did you use?<br>
Was the accuracy better or worse?<br>
Is your current loss changing, while the accuracy stays the same from the beginning?</p>

talhak: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>  for your care. Well, I used <code>CrossEntropyLoss</code> as I have stated in the OP. The <code>accuracy</code> stays at <code>50.22%</code> which is worse than <code>NLLLoss</code> but the test loss is a lot better than it. When it comes to current <code>loss</code>, yes, it is changing.</p>

ptrblck: <p>Are you applying <code>F.log_softmax</code> inside your <code>forward</code> method? <code>nn.CrossEntropyLoss</code> expects logits, as the log softmax will be applied internally. If you pass raw logits directly to <code>nn,NLLLoss</code>, I would assume your model performs worse.<br>
Have you played around with some hyperparameters, e.g. the learning rate?</p>
<p>Is your dataset balanced, i.e. is the class distribution approx. equal?</p>

talhak: <p>Yes, I am applying <code>F.log_softmax</code> inside the <code>forward</code> function. When I change <code>learning rate</code> from 0.001 to 0.01, now the <code>accuracy</code> changes a little bit but still it is way lower than <code>nn.NLLLoss</code>. So when using <code>nn.CrossEntropyLoss</code>, should I directly return the output inside the <code>forward</code> function?</p>
<p>Regarding my dataset, it is a publicly available facial expression <a href="https://github.com/muxspace/facial_expressions" rel="nofollow noopener">dataset hosted on GitHub</a>, it is not well balanced due to the nature of the context.</p>

ptrblck: <p>Thanks for the information!<br>
Since you are applying <code>F.log_softmax</code>, you should use <code>nn.NLLLoss</code> as the criterion.</p>

talhak: <p>Not at all, thanks for the feedback. Fair enough, but actually I am testing various loss functions in order to reveal if I might improve the accuracy. So, when using  <code>nn.CrossEntropyLoss</code>, should I directly return the output of the FC layers as the last command of the <code>forward</code> function?</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> expects logits, so you should return the output of the last layer without any non-linearity.<br>
However, internally <code>nn.CrossEntropyLoss</code> will apply <code>F.log_softmax</code> and <code>nn.NLLLoss</code> as seen <a href="https://github.com/pytorch/pytorch/blob/d3f6d5885d68d47d38a475d4fc9cf1611c91920c/torch/nn/functional.py#L1994" rel="nofollow noopener">here</a>, so you would get the same results.</p>

talhak: <p>Last question, what could be the reason for achieving a high <code>loss</code> despite a good <code>accuracy</code> when using <code>nn.NLLLoss</code>? For the same dataset, the average <code>loss</code> is <code>46.84</code> despite the <code>accuracy</code> is <code>84.19%</code>. And the result does not improve through the epochs - even decreases. I have played around the hyper-parameters and tried different optimizers through the recommendations available on the Internet, but nothing was significantly changed.</p>

ptrblck: <p>Your model might be overfitting to the majority class and thus not learning anything useful.<br>
Could you check your class distribution and see if the majority class occurs approx. In 84% of all cases?<br>
Also check the prediction for unique values.</p>
<p>If you see that the class imbalance might be the reason for this issue, you could use a <code>WeightedRandomSampler</code> or pass class weights to your loss function to counter this effect.</p>

talhak: <p>I see, just integrated the proposed method on <a href="https://discuss.pytorch.org/t/balanced-sampling-between-classes-with-torchvision-dataloader/2703/3">another topic here</a> to use <code>WeightedRandomSampler</code> and will post the result here when the training and test phases are completed. Between, could you recommend any utility function to check the class distribution and the prediction for unique values?</p>

ptrblck: <p>I would store the targets in a tensor and call <code>.unique(return_counts=True)</code> on it. The same would work for the predictions (use <code>torch.argmax(output, 1)</code> to get the class predictions).</p>

talhak: <p>When I use <code>WeightedRandomSampler</code> or the one published on GitHub named <strong><a href="https://github.com/ufoym/imbalanced-dataset-sampler" rel="nofollow noopener">imbalanced-dataset-sampler</a></strong>, the average <code>loss</code> was decreased a lot (now it is ~2), but the <code>accuracy</code> was decreased dramatically as well, which was ~50%, and now is ~2.50%. So, is the only solution simply skipping the used dataset since it is imbalanced, and samplers do not work well?</p>

ptrblck: <p>Balancing the dataset should trace the accuracy of the majority class for an increase in the accuracy of the minority classes.<br>
I’ve created a tutorial a while ago <a href="https://github.com/ptrblck/tutorials/blob/imbalanced_tutorial/intermediate_source/imbalanced_data_tutorial.py" rel="nofollow noopener">here</a>, which is quite old by now, but might give you some insight.</p>