Is it ok to use nn.CrossEntropyLoss() even for binary classification?

rkmp: <p>I am training a binary classifier, however I have a softmax layer as the last layer, thus is it ok if I use nn.CrossEntropyLoss() as objective function instead of Binary Cross entropy loss? are there any difference?</p>

Shani_Gamrian: <p>I tried using <code>nn.CrossEntropyLoss()</code> for binary classification and it didn’t work. You should use <code>BCELoss</code>.</p>

Happy_NewYear: <p>What are the other loss functions which work for multi-class classification(one is Cross entropy.)?</p>

Shani_Gamrian: <p>MultiLabelMarginLoss, MultiLabelSoftMarginLoss, MultiMarginLoss</p>

tjoseph: <p>Shouldn’t <code>nn.CrossEntropyLoss()</code> be equivalent to <code>nn.Sigmoid() --&gt; nn.BCELoss()</code>? (With two / one output respectivly).</p>

rkmp: <p>But BCE Loss does Sigmoid, I have a softmax layer in the end, any workaround?</p>

Latope2-150: <p>BCE Loss does not do sigmoid, you can look at the doc <a href="http://pytorch.org/docs/master/nn.html#torch.nn.BCELoss" rel="nofollow noopener">http://pytorch.org/docs/master/nn.html#torch.nn.BCELoss</a><br>
If you want BCE to do sigmoid, use BCE with logits <a href="http://pytorch.org/docs/master/nn.html#torch.nn.BCEWithLogitsLoss" rel="nofollow noopener">http://pytorch.org/docs/master/nn.html#torch.nn.BCEWithLogitsLoss</a></p>

Sayak_Paul: <p>Yes exactly. You might want to follow this little thread on SO to get this clearer: <a href="https://stackoverflow.com/questions/53628622/loss-function-its-inputs-for-binary-classification-pytorch" rel="nofollow noopener">https://stackoverflow.com/questions/53628622/loss-function-its-inputs-for-binary-classification-pytorch</a></p>

dsprahul: <p><code>nn.CrossEntropyLoss</code> for binary classification didn’t work for me too! In fact, it did the opposite of learning. Why didn’t it work for you? Can you please explain the behavior I am observing?<br>
Note: The same model with <code>nn.MSELoss</code> worked fine (passed overfitting test). And I am supplying the unnormalized FC layer output to the CE loss function.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/8/683827a211b0cc84b8f78f30f19bbda09585432e.png" data-download-href="https://discuss.pytorch.org/uploads/default/683827a211b0cc84b8f78f30f19bbda09585432e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/683827a211b0cc84b8f78f30f19bbda09585432e_2_690x334.png" alt="image" data-base62-sha1="eRXY1foqvP5O0PNRRFU8kOuDtlY" width="690" height="334" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/683827a211b0cc84b8f78f30f19bbda09585432e_2_690x334.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/683827a211b0cc84b8f78f30f19bbda09585432e_2_1035x501.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/683827a211b0cc84b8f78f30f19bbda09585432e_2_1380x668.png 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1450×702 57.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Pytorch-Noob: <p>Let me summarize for everyone:</p>
<ol>
<li>Use Cross Entropy as loss when dealing with multi-class problems. Use Softmax as last layer in the model.</li>
<li>Use Binary-Cross Entropy as loss when dealing with binary-class problems. Use Sigmoid as last layer in the model.</li>
</ol>
<p>Correct this Noob if wrong!</p>
<p>CORRECTION EDIT:</p>
<ol>
<li>
<p>When using Cross Entropy loss aka <code>nn.CrossEntropyLoss</code>, you do not need to have a Softmax as last layer applied in the model, since the Cross Entropy loss already applies it for you!</p>
</li>
<li>
<p>Replace <code>nn.BCELoss</code> with <code>nn.BCEWithLogitsLoss</code> and remove the <code>sigmoid</code> for better numerical stability.</p>
</li>
</ol>

ptrblck: <aside class="quote no-group" data-username="Pytorch-Noob" data-post="10" data-topic="14660">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pytorch-noob/48/78054_2.png" class="avatar"> Pytorch-Noob:</div>
<blockquote>
<p>Use Cross Entropy as loss when dealing with multi-class problems. Use Softmax as last layer in the model.</p>
</blockquote>
</aside>
<p>Don’t use a softmax in combination with <code>nn.CrossEntropyLoss</code> since the criterion expects raw logits as <code>F.log_softmax</code> will be applied internally.</p>
<aside class="quote no-group" data-username="Pytorch-Noob" data-post="10" data-topic="14660">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pytorch-noob/48/78054_2.png" class="avatar"> Pytorch-Noob:</div>
<blockquote>
<p>Use Binary-Cross Entropy as loss when dealing with binary-class problems. Use Sigmoid as last layer in the model.</p>
</blockquote>
</aside>
<p>Replace <code>nn.BCELoss</code> with <code>nn.BCEWithLogitsLoss</code> and remove the <code>sigmoid</code> for better numerical stability.</p>

vdw: <p>For a binary classification, you can actually do both, depending on how you choose the output of your model.</p>
<p>Nobody is stopping you to treat your binary classification task as a multiclass classification task. This means, your network has two outs, one for each class, and you use Softmax and the (general) cross entropy loss.</p>
<p>Or you use only one output capturing both classes. Then you use the Sigmoid and the (binary) cross entropy loss.</p>