Manual Calculation of Binary Cross Entropy with logits

username: arif_saeed
cooked: <p>In order to ensure that I understood how BCE with logits loss works in pytorch, I tried to manually calculate the loss, however I cannot reconcile my manual calculation with the loss generated by the pytorch function F.binary_cross_entropy_with_logits.</p>
<p>can somebody please explain what i am doing wrong.  My code is below:</p>
<p>import torch<br>
import torch.nn.functional as F<br>
import math</p>
<p>def sigmoid(x):<br>
return 1/(1+math.exp(-x))</p>
<p>def manual_cross_entropy_with_logits(inp, target):</p>
<pre><code>y=target[0]
x=inp[0]
p1=y*(math.log(sigmoid(x),2))
p0=(1-y)*math.log(1-sigmoid(x),2)
return -1 * (p1 + p0)
</code></pre>
<p>inp = torch.randn(1, requires_grad=True)<br>
target = torch.empty(1).random_(2)</p>
<p>loss = F.binary_cross_entropy_with_logits(inp, target)<br>
lossman=manual_cross_entropy_with_logits(inp,target)<br>
print(‘bce loss’)<br>
print(loss)<br>
print(‘manual loss’)<br>
print(lossman)</p>

username: KFrank
cooked: <p>Hi Arif!</p>
<aside class="quote no-group quote-modified" data-username="arif_saeed" data-post="1" data-topic="61602" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arif_saeed/48/14172_2.png" class="avatar"> arif_saeed:</div>
<blockquote>
<p>In order to ensure that I understood how BCE with logits loss works in pytorch, I tried to manually calculate the loss, however I cannot reconcile my manual calculation with the loss generated by the pytorch function F.binary_cross_entropy_with_logits.</p>
<pre><code>p1=y*(math.log(sigmoid(x),2))
p0=(1-y)*math.log(1-sigmoid(x),2)
</code></pre>
</blockquote>
</aside>
<p>You need to use the <em>natural logarithm</em> (log-base-e) in your<br>
cross-entropy formula.  You are using log-base-2.  Remove<br>
the second argument in your calls to python’s <code>math.log()</code>.<br>
Thus:</p>
<pre data-code-wrap="python"><code class="lang-python">    p1=y*(math.log(sigmoid(x)))
    p0=(1-y)*math.log(1-sigmoid(x))
</code></pre>
<p>As a side note, calculating <code>log()</code> and <code>sigmoid()</code> separately<br>
can amplify numerical errors.  You can avoid this by using<br>
something like pytorch’s <a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.logsigmoid" rel="noopener nofollow ugc">logsigmoid()</a> that uses a better<br>
algorithm internally.</p>
<p>Good luck!</p>
<p>K. Frank</p>

username: eliphatfs
cooked: <p>Sorry to bump this thread, but how do I implement log(1 - sigmoid(x)) with logsigmoid?</p>

username: KFrank
cooked: <p>Hi Eliphat!</p>
<aside class="quote no-group" data-username="eliphatfs" data-post="3" data-topic="61602" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eliphatfs/48/57712_2.png" class="avatar"> eliphatfs:</div>
<blockquote>
<p>how do I implement log(1 - sigmoid(x)) with logsigmoid?</p>
</blockquote>
</aside>
<p><code>sigmoid (-x) = 1 - sigmoid (x)</code>, so you can compute <code>logsigmoid (-x)</code>.</p>
<p>This has the same numerical benefits when <code>x</code> becomes large and positive<br>
as <code>logsigmoid (x)</code> has when <code>x</code> becomes large and negative.</p>
<p>Best.</p>
<p>K. Frank</p>