Sigmoid and BCELoss

username: snip3r77
cooked: <p>Got this from the documentation</p>
<blockquote>
<p>m = nn.Sigmoid()<br>
loss = nn.BCELoss()</p>
<p><span class="hashtag-raw">#input</span> is of size N x C = 1 x 3<br>
input = torch.randn(3, requires_grad=True)</p>
<p>print(input)</p>
<p><span class="hashtag-raw">#each</span> element in target has to have 0 &lt;= value &lt; C<br>
target = torch.empty(3).random_(2)</p>
<p>print(m(input))<br>
print(target)</p>
<p>output = loss(m(input), target)</p>
</blockquote>
<p>print(output)</p>
<p>OUTPUT</p>
<blockquote>
<p>tensor([-0.8840,  0.7303, -0.5842], requires_grad=True)<br>
tensor([0.2923, 0.6749, 0.3580], grad_fn=)<br>
tensor([0., 1., 0.])<br>
tensor(0.3940, grad_fn=)</p>
</blockquote>
<p>Questions<br>
This is the values after sigmoid which is btw 0,1<br>
[0.2923, 0.6749, 0.3580] ← is this 3 y-predictions ?</p>
<p>this is y-actual<br>
tensor([0., 1., 0.])</p>
<p>In this case since the 2nd element will round to 1 we predict correctly for all the 3 elements</p>
<p>How does loss compare both and obtain 0.394?</p>
<p>Thanks</p>

username: KFrank
cooked: <p>Hi Snip!</p>
<aside class="quote no-group quote-modified" data-username="snip3r77" data-post="1" data-topic="74468" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/65b543/48.png" class="avatar"> snip3r77:</div>
<blockquote>
<p>Questions<br>
This is the values after sigmoid which is btw 0,1<br>
[0.2923, 0.6749, 0.3580] ← is this 3 y-predictions ?</p>
</blockquote>
</aside>
<p>Yes.  But these should be understood as <em>probabilistic</em> predictions.<br>
That is, you are predicting a 29% chance of being in class “1” (and<br>
hence a 71% chance of being in class “0”), a 67% chance of being<br>
in class “1”, and a 36% chance of being in class “1”, respectively.</p>
<aside class="quote no-group">
<blockquote>
<p>this is y-actual<br>
tensor([0., 1., 0.])</p>
</blockquote>
</aside>
<p>Yes.*</p>
<aside class="quote no-group">
<blockquote>
<p>In this case since the 2nd element will round to 1 we predict correctly for all the 3 elements</p>
</blockquote>
</aside>
<p>Yes, but you have note that you are converting from probabilistic to<br>
deterministic yes-no predictions.  Rounding (that is, testing that <code>p &gt; 1/2</code>)<br>
yields <code>[0, 1, 0]</code>.  This means class “0”, class “1”, class “0”, with zero<br>
predicted probability of being in the other class.  This is the right thing<br>
to do when you want to measure the <em>accuracy</em> of your classifier, but<br>
doesn’t really work as a loss function used for training.</p>
<aside class="quote no-group">
<blockquote>
<p>How does loss compare both and obtain 0.394?</p>
</blockquote>
</aside>
<p>In the code you posted <code>BCELoss</code> is comparing what you call “y-actual”<br>
with the probabilistic prediction, and is not rounding anything to obtain<br>
yes-no predictions.</p>
<p>If your probabilistic prediction were <code>[0.0, 1.0, 0.0]</code>, then it would be<br>
a perfect prediction (You would be predicting y-actual with 100%<br>
certainty.), and the loss would be 0.0.</p>
<p>But, for example, the first entry of your prediction is <code>0.29</code>.  So you<br>
are predicting the correct y-actual – class “0” – with 71% certainty.<br>
That’s pretty good, but not “perfect.”  And you are incorrectly predicting<br>
class “1” – not the y-actual – with a 29% chance, still pretty good, but<br>
partially wrong.</p>
<p>So your loss is greater than 0, reflecting your imperfect (but pretty<br>
good) prediction.</p>
<p>*)  When training binary classifiers with <code>BCELoss</code> our targets / labels<br>
are usually <code>0.0</code> and <code>1.0</code> and can be interpreted as class “0” and<br>
class “1” class labels.  But they can also be understood as the<br>
probability of being in class “1”, and are only required by <code>BCELoss</code><br>
to be in the range <code>0.0 &lt;= p &lt;= 1.0</code>.  Technically, <code>BCELoss</code> is<br>
calculating the cross-entropy between two probability distributions,<br>
the predicted probability, and the target (y-actual) probability.  The<br>
details can be found in the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BCELoss" rel="noopener nofollow ugc">BCELoss</a> documentation.</p>
<p>(As a side note, in practice you are better off leaving out the <code>Sigmoid</code>,<br>
and using <code>BCEWithLogitsLoss</code>, that has, in effect, <code>Sigmoid</code> built in.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: snip3r77
cooked: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a><br>
thanks for your prompt reply</p>
<p>I think I got it already<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/8/08082e44bdd4f28de2a5ec5a4be6ea68f01e7ff2.png" data-download-href="https://discuss.pytorch.org/uploads/default/08082e44bdd4f28de2a5ec5a4be6ea68f01e7ff2" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08082e44bdd4f28de2a5ec5a4be6ea68f01e7ff2_2_419x500.png" alt="image" data-base62-sha1="193lvwuqSAkSFAywzELWI9aM3wC" width="419" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08082e44bdd4f28de2a5ec5a4be6ea68f01e7ff2_2_419x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08082e44bdd4f28de2a5ec5a4be6ea68f01e7ff2_2_628x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/8/08082e44bdd4f28de2a5ec5a4be6ea68f01e7ff2.png 2x" data-dominant-color="F4F4F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">725×865 42.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>But I have some confusion for crossentropy.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3faa3e74d0104168d615529999fefcba4c95323a.png" alt="image" data-base62-sha1="95cPfGCnGIgU5LFyqXUZ0HecD8e" width="615" height="437"></p>
<p>I kinda understand regarding the dim=1 which came from the screenshot below.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/0/605b17e5bcdc7a20e9f493a90e93ca2fb25a5f57.png" alt="image" data-base62-sha1="dKp0et1tIY24JoCodswkza1A0J1" width="644" height="306"><br>
We need to sum up the denominator for both but why we do not require dim for CrossEntropyLoss</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b47a9267121acaa73a9e7d3fa33ce7655c78e96.png" alt="image" data-base62-sha1="hAAgcqMKly5130deLvLZCBmQcu2" width="650" height="498"><br>
Is my understanding correct when I try so simulate a bad prediction and a good one?<br>
For the target(y_actual),we need to do one hot encoding too.</p>
<p>Also, how can we simulate the NLL?</p><aside class="quote" data-post="1" data-topic="23702">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pawandeep_singh/48/6288_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/understanding-nllloss-function/23702">Understanding NLLLoss function</a> 
  </div>
  <blockquote>
    m = nn.LogSoftmax(dim=1)
loss = nn.NLLLoss()
# input is of size N x C = 1 x 3
input = torch.tensor([[0,0,0]], requires_grad=True, dtype=torch.float)
# each element in target has to have 0 &lt;= value &lt; C
target = torch.tensor([1])
output = loss(m(input), target)
print(output)

_Output: tensor(1.0986, grad_fn=&lt;NllLossBackward&gt;)_

I read the <a href="https://pytorch.org/docs/stable/nn.html#nllloss" rel="noopener nofollow ugc">Documentation</a> but it’s not clear. Can someone explain the math behind this example?
  </blockquote>
</aside>
<p>
we need to exp and then multiply by -1</p>
<p>Sidetrack, in DL if we’re saying logits does it mean raw form as in our input in the codes?<br>
after we apply sigmoid or softmax, it will be probability?</p>
<p>Thanks.</p>