BCE Loss vs Cross Entropy

username: kliu
cooked: <p>Hi all,</p>
<p>I am wondering what loss to use for a specific application.</p>
<p>I am trying to predict some binary image. For example, given some inputs a simple two layer neural net with ReLU activations after each layer outputs some 2x2 matrix [[0.01, 0.9], [0.1, 0.2]]. This prediction is compared to a ground truth 2x2 image like [[0, 1], [1, 1]] and the networks task is to get as close as possible.</p>
<p>I am currently using BCE loss with logits over the prediction and the ground truth, however I was wondering if I should be using the plain BCE loss instead (no logits)? If neither of these are suitable, should I be using CrossEntropyLoss? Not quite sure what the difference between these is in this context.</p>
<p>Any help is appreciated - thanks in advance</p>

username: KFrank
cooked: <p>Hi kliu!</p>
<aside class="quote no-group" data-username="kliu" data-post="1" data-topic="97437" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kliu/48/33714_2.png" class="avatar"> kliu:</div>
<blockquote>
<p>I am trying to predict some binary image.</p>
</blockquote>
</aside>
<p>If I understand your use case, you should start with<br>
<code>BCEWithLogitsLoss</code> as your loss function (and only change<br>
to something else if you have a good reason and testing shows<br>
that the change is for the better).</p>
<aside class="quote no-group">
<blockquote>
<p>a simple two layer neural net with ReLU activations after each layer</p>
</blockquote>
</aside>
<p>Just to be clear, for <code>BCEWithLogitsLoss</code>, the last layer of your<br>
network should be a <code>Linear</code>, and it should not be followed by<br>
an activation function (e.g., neither <code>ReLU</code> nor <code>Sigmoid</code>).</p>
<aside class="quote no-group">
<blockquote>
<p>outputs some 2x2 matrix [[0.01, 0.9], [0.1, 0.2]].</p>
</blockquote>
</aside>
<p>Note, these values – between 0.0 and 1.0 – are not values you<br>
would typically get from a final <code>Linear</code> layer.  They could occur,<br>
but in general, you would expect values running from <code>-inf</code> to<br>
<code>inf</code> (that would be interpreted as raw-score <em>logits</em>).</p>
<aside class="quote no-group">
<blockquote>
<p>I am currently using BCE loss with logits over the prediction and the ground truth, however I was wondering if I should be using the plain BCE loss instead (no logits)? If neither of these are suitable, should I be using CrossEntropyLoss? Not quite sure what the difference between these is in this context.</p>
</blockquote>
</aside>
<p>Yes, you should be using <code>BCEWithLogitsLoss</code>.</p>
<p><code>Sigmoid</code> followed by <code>BCELoss</code> is mathematically equivalent to<br>
<code>BCEWithLogitsLoss</code>, but numerically less stable.</p>
<p><code>CrossEntropyLoss</code> (which would better be called<br>
“CategoricalCrossEntropyWithLogitsLoss”) is essentially the same as<br>
<code>BCEWithLogitsLoss</code>, but requires making some small modifications<br>
to your network and your ground-truth labels that add a small amount<br>
of unnecessary redundancy to your network.</p>
<p>Best.</p>
<p>K. Frank</p>

username: kliu
cooked: <p>Hey KFrank!</p>
<p>Thanks for the quick detailed response. I was indeed having doubts about stacking the ReLU and Sigmoid - I will give this a shot today and I’ll follow up if things workout (fingers crossed!). In the meantime, do you happen to know why BCEWithLogitsLoss is numerically more stable than applying Sigmoid and BCELoss separately? I’m assuming this might be an implementation detail? Thanks for all your help.</p>
<p>Best,<br>
K. Liu</p>

username: KFrank
cooked: <p>Hi kliu!</p>
<aside class="quote no-group" data-username="kliu" data-post="3" data-topic="97437" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kliu/48/33714_2.png" class="avatar"> kliu:</div>
<blockquote>
<p>do you happen to know why BCEWithLogitsLoss is numerically more stable than applying Sigmoid and BCELoss separately? I’m assuming this might be an implementation detail?</p>
</blockquote>
</aside>
<p>Yes, it is an implementation detail.  Mathematically, the two are the<br>
same.</p>
<p>The issue is that <code>Sigmoid</code> (in particular, the so-call <em>logistic function</em>)<br>
uses exponential to map <code>(-inf, inf)</code> to (0.0, 1.0).  But then <code>BCELoss</code><br>
turns around and uses <code>log()</code> to map <code>(0.0, 1.0)</code> back to <code>(-inf, inf)</code>.<br>
Mathematically, this is fine, but numerically, using floating-point<br>
arithmetic, the exponentials can start to saturate, leading to loss<br>
of precision, and can underflow to <code>0.0</code> and overflow to <code>inf</code>, leading<br>
to <code>inf</code>s and <code>nan</code>s in your loss function and backpropagation.</p>
<p><code>BCEWithLogitsLoss</code> avoides this internally by rearranging the<br>
computation.  (Note that pytorch provides a <code>LogSigmoid</code> function<br>
that does the analogous  computation internally.)</p>
<p>A similar issue arises when feeding the results of <code>Softmax</code> to a<br>
plain cross-entropy loss.  Pytorch doesn’t even offer a plain<br>
cross-entropy function.  Instead, pytorch’s <code>CrossEntropyLoss</code><br>
requires logits as its inputs, and, in effect, applies the <code>Softmax</code><br>
internally.</p>
<p>Best.</p>
<p>K. Frank</p>

username: kliu
cooked: <p>Hey K. Frank,</p>
<p>I see, that makes sense. I’ll be sure to checkout the source code at some point. In other good news - your answer worked out for me. Thanks for all of your help! Have a great weekend.</p>
<p>Best,<br>
K. Liu</p>