V1.0.1, nn.BCEWithLogitsLoss returns negative loss, Sigmoid layer not deployed

ruian1: <p>Hi Pytorch Community! I was using BCEWithLogintsLoss to train a multi-label network and getting negative loss as shown below. As mentioned in the class documentation, this loss function combines sigmoid and BCELoss…But actually as it shows. I also checked its definition in pytorch/torch/nn/functional.py and I was not able to find the sigmoid operation in this loss. Maybe I used it wrong ?</p>
<p>I’m using version 1.0.1</p>
<pre><code class="lang-auto">criterion = nn.BCEWithLogitsLoss()
a = torch.tensor([[1., 1., 1., 0., 0.]])
b = torch.tensor([[0., 0.0011122, 8.9638, 0., 0.]])
c = nn.Sigmoid()(b)
print(criterion(a,b))
print(criterion(a,c))

"tensor(-0.7278)"
"tensor(0.6652)"
</code></pre>
<p>Should line 598 of pytorch/torch/nn/moduels/loss.py be changed as below  ?</p>
<pre><code class="lang-auto">        return F.binary_cross_entropy_with_logits(nn.Sigmoid()(input), target,</code></pre>

izdeby: <p>This might be related to this <a href="https://github.com/pytorch/pytorch/issues/24933" rel="nofollow noopener">issue</a>. I will take a look into it</p>

KFrank: <p>Hello Rui An!</p>
<aside class="quote no-group quote-modified" data-username="ruian1" data-post="1" data-topic="57409" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/2bfe46/48.png" class="avatar"> ruian1:</div>
<blockquote>
<p>I was using BCEWithLogintsLoss to train a multi-label network and getting negative loss as shown below.</p>
<p>…</p>
<pre><code class="lang-auto">criterion = nn.BCEWithLogitsLoss()
a = torch.tensor([[1., 1., 1., 0., 0.]])
b = torch.tensor([[0., 0.0011122, 8.9638, 0., 0.]])
c = nn.Sigmoid()(b)
print(criterion(a,b))
print(criterion(a,c))

"tensor(-0.7278)"
"tensor(0.6652)"
</code></pre>
</blockquote>
</aside>
<p>It appears that you have switched the order of your inputs to<br>
<code>BCEWithLogitsLoss</code>.</p>
<p><code>BCEWithLogitsLoss</code> (like<br>
<code>binary_cross_entropy_with_logits()</code>) expects to be<br>
called with predictions that are logits (-infinity to infinity) and<br>
targets that are probabilities (0 to 1), in that order.</p>
<p>Your  <code>a</code> are legitimate probabilities, so they are your targets, and<br>
your <code>b</code> are legitimate logits, so they are your predictions.  Your<br>
call should therefore be:</p>
<pre data-code-wrap="python"><code class="lang-python">print(criterion(b,a))
</code></pre>
<p>(In your version of the call your second argument, <code>b</code>, is out of<br>
range (not 0 to 1), so the call returns the invalid negative result.<br>
<code>c</code>, however, being the result of <code>Sigmoid</code>, is in (0, 1), so the<br>
result of the call is indeed positive.)</p>
<aside class="quote no-group">
<blockquote>
<p>Should line 598 of pytorch/torch/nn/moduels/loss.py be changed as below  ?</p>
<pre><code class="lang-auto">        return F.binary_cross_entropy_with_logits(nn.Sigmoid()(input), target,
</code></pre>
</blockquote>
</aside>
<p>No.  <code>binary_cross_entropy_with_logits()</code> has built<br>
into it (implicitly, in effect) the sigmoid function (just like<br>
<code>BCEWithLogitsLoss</code>), so you want to pass logits as the first<br>
argument (<code>input</code>), not probabilities (<code>nn.Sigmoid()(input)</code>).</p>
<p>You shouldn’t expect to see an explicit call to <code>Sigmoid</code> in the<br>
source code for <code>BCEWithLogitsLoss</code>.  In effect <code>Sigmoid</code> is<br>
applied to <code>input</code>, but it’s implicit (using the “log-sum-exp trick”).</p>
<p>Good luck.</p>
<p>K. Frank</p>

ruian1: <p>Hello KFrank, thanks a lot for a very detailed explanation! Could you also point me to the block where “log-sum-exp” is applied to ? I traced it back to functional.py but still can’t figure out how it is done…</p>

KFrank: <p>Hi Rui An!</p>
<aside class="quote no-group" data-username="ruian1" data-post="4" data-topic="57409" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/2bfe46/48.png" class="avatar"> ruian1:</div>
<blockquote>
<p>Could you also point me to the block where “log-sum-exp” is applied to ? I traced it back to functional.py but still can’t figure out how it is done…</p>
</blockquote>
</aside>
<p>I don’t have the <code>binary_cross_entropy_with_logits</code> code<br>
in front of me, so I can’t give you the specifics.</p>
<p>The issue is that floating-point error can get amplified when you<br>
compute <code>log</code> of an expression containing <code>exp</code>.  <code>sigmoid</code> has<br>
the <code>exp</code>, and cross-entropy has the <code>log</code>, so you can run into<br>
this problem when using <code>sigmoid</code> as input to cross-entropy.<br>
Dealing with this issue is the main reason that<br>
<code>binary_cross_entropy_with_logits</code> exists.</p>
<p>See, for example, the comments about “log1p” in the Wikipedia<br>
article about <a href="https://en.wikipedia.org/wiki/Natural_logarithm" rel="noopener nofollow ugc">logarithm</a>.</p>
<p>(I was speaking loosely when I mentioned the related<br>
“log-sum-exp-trick.”  This is more directly relevant to computing<br>
<code>softmax</code>, but is basically another facet of the same issue.  For<br>
more on this, see, for example, Wikipedia’s <a href="https://en.wikipedia.org/wiki/LogSumExp" rel="noopener nofollow ugc">LogSumExp</a>)</p>
<p>Best.</p>
<p>K. Frank</p>