Manual Mean != BCEWithLogitsLoss Mean

username: PiXAI
cooked: <p>I’m doing some experiments with MultiLabelClassification and I was trying to calculate the mean of the loss manually and found out, that there is a big difference to the BCEWithLogitsLoss-Mean and I don’t get why that’s the case.</p>
<p>Here is my code:</p>
<pre><code class="lang-python">import math, torch, torch.nn as nn
pre = torch.tensor([0.133, 0.145, 0.692, 0.030])
des = torch.tensor([0., 0., 1., 0.])
torch.sigmoid(pre)
--&gt; tensor([0.5332, 0.5362, 0.6664, 0.5075])

l1 = -(1*(math.log(0.5332))+(1-0)*math.log(1-0.5332))
--&gt; 1.3907130692473197

l2 = -(1*(math.log(0.5362))+(1-0)*math.log(1-0.5362))
--&gt; 1.391549907340984

l3 = -(1*(math.log(0.6664))+(1-1)*math.log(1-0.6664))
--&gt; 0.4058651881295041

l4 = -(1*(math.log(0.5075))+(1-0)*math.log(1-0.5075))
--&gt; 1.3865193864361882

1/4*(l1+l2+l3+l4)
--&gt; 1.143661887788499
</code></pre>
<p><br><br></p>
<p>And this is the pytorch-implementation:</p>
<pre><code class="lang-python">import math, torch, torch.nn as nn
criterion = nn.BCEWithLogitsLoss(reduction='mean')
pre = torch.tensor([0.133, 0.145, 0.692, 0.030])
des = torch.tensor([0., 0., 1., 0.])
criterion(pre, des)
--&gt; tensor(0.6611)
</code></pre>
<p>If I set the reduction to ‘none’ only l3 (Loss 3) hast the same value:</p>
<pre><code class="lang-python">tensor([0.7619, 0.7683, 0.4058, 0.7083])
</code></pre>
<p>Could somebody please explain to me, why that’s the case? <img src="https://discuss.pytorch.org/images/emoji/apple/upside_down_face.png?v=9" title=":upside_down_face:" class="emoji" alt=":upside_down_face:"><br>
Thanks!</p>

username: alexgo
cooked: <p>you have a mistake in the labels in your manual implementation.<br>
should be:</p>
<p>l1 = -(0*(math.log(0.5332))+(1-0)*math.log(1-0.5332))</p>
<p>l2 = -(0*(math.log(0.5362))+(1-0)*math.log(1-0.5362))</p>
<p>l3 = -(1*(math.log(0.6664))+(1-1)*math.log(1-0.6664))</p>
<p>l4 = -(0*(math.log(0.5075))+(1-0)*math.log(1-0.5075))</p>

username: PiXAI
cooked: <p><img src="https://discuss.pytorch.org/images/emoji/apple/man_facepalming/3.png?v=9" title=":man_facepalming:t3:" class="emoji only-emoji" alt=":man_facepalming:t3:"><br>
I really wonder sometimes where I left my brain.</p>
<p>Thank you! That was the mistake I made.</p>
<p>And now I’m getting a loss which I understand <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>BCEWLL:</p>
<pre><code class="lang-python">criterion = nn.BCEWithLogitsLoss(reduction='none', pos_weight=samples_weight)
pre = torch.tensor([-8., 0., 4., -2.])
des = torch.tensor([0., 0., 1., 0.])

criterion(pre, des)
--&gt; tensor([3.3569e-04, 6.9315e-01, 1.8150e-02, 1.2693e-01])

torch.mean(criterion(pre,des))
--&gt; tensor(0.2096)
</code></pre>
<p>Manual:</p>
<pre><code class="lang-python">pre = torch.tensor([-8., 0., 4., -2.])
des = torch.tensor([0., 0., 1., 0.])
import math, torch, torch.nn as nn
import numpy as np
def bll(pre, des):
  bitloss = 0
  for p, d in zip(pre, des):
    sp = torch.sigmoid(p)
    loss = -(d*(math.log(sp))+(1-d)*math.log(1-sp))
    bitloss += loss
    print(loss)
  return bitloss

ll = bll(pre, des)
BCEWLL = (1/len(pre))*ll
print(f"MEAN:{BCEWLL}")

--&gt; tensor(0.0003)
tensor(0.6931)
tensor(0.0181)
tensor(0.1269)
MEAN:0.20964014530181885
</code></pre>
<p>The only thing I wonder is how -8 can have such a low loss value when it’s the farthest away?<br>
According to <a href="https://www.wolframalpha.com/input/?i=log%281+-+1%2F%281+%2B+e%5E8%29%29&amp;assumption=%22ClashPrefs%22+-%3E+%7B%22Math%22%7D" rel="noopener nofollow ugc">log(1 - 1/(1 + e^8))</a> it should also be a negative value.</p>
<p>EDIT:<br>
Well I answer my question myself. Lowest loss because it’s the farthest away and thus “very wrong” -&gt; Not dangerous to interfere the true prediction. The 0 prediction is the nearest so the biggest loss there.</p>

username: alexgo
cooked: <p>Sigmoid of -8 is a very small number so it means the prediction is correct, it is close to the label 0. So it makes sense that the loss is low.</p>
<p>The expression you wrote is close to log(1) which is 0.</p>