Question about BCEWithLogitsLoss

username: chaslie
cooked: <p>Morning,</p>
<p>Silly question, but when using:</p>
<pre><code class="lang-auto">&gt; criterion = torch.nn.BCEWithLogitsLoss(size_average=True,reduce=False, reduction=None)
BCE=criterion(out,data)
</code></pre>
<p>which gives the output of:</p>
<pre><code class="lang-auto">BCE
tensor([[[[-6.5510e-01,  3.3377e+00,  3.8248e+00,  ...,  3.8797e+00,
            7.2267e-01, -5.6749e-01],
          [ 4.6354e+00,  3.2439e+00,  2.5523e+00,  ...,  2.2153e+00,
            3.5900e+00,  2.6402e-01],
          [-9.9316e-01,  9.2173e-01, -1.6896e+00,  ...,  3.5983e+00,
            8.9532e-01, -6.6330e-01],
          ...,
          [ 2.1913e+00,  2.4635e+00, -3.7076e-01,  ...,  2.8466e-01,
            1.7843e+00,  1.4812e+00],
          [-7.6295e-02,  3.0910e+00,  1.1081e+00,  ...,  3.8564e-02,
            1.6032e+00,  2.9802e-01],
          [-1.1393e+00,  2.7563e+00, -1.7027e+00,  ..., -4.7357e-01,
            2.0175e+00, -2.9758e-01]],

         [[-2.9676e+00, -9.7755e-01, -1.7805e+00,  ...,  1.2246e+00,
           -1.1476e+00,  7.8534e-01],
          [-1.6817e-01,  2.9306e+00,  5.1950e-01,  ...,  6.9879e-01,
            6.4816e-01,  2.9168e+00],
          [-1.2073e+00,  1.3979e-01, -4.0174e+00,  ...,  1.4663e+00,
           -1.5716e+00,  2.3580e+00],
</code></pre>
<p>If I understand the command BCEWithLogitsLoss, it uses a sigmoid function, so the $64k is why do i get values in the outut of BCE which are outside +/-1?</p>
<p>I realise its something i have down but not sure what?</p>
<p>Cheers,</p>
<p>chaslie</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>It seems you misunderstood the <code>BCEWithLogitsLoss</code>. It uses <code>sigmoid</code> function on its inputs not on outputs. Here is pipeline: <code>x-&gt;BCEWithLogitsLoss</code> = <code>x-&gt; sigmoid -&gt; BCELoss</code> (Note that <code>BCELoss</code> is a standalone function in PyTorch too.)</p>
<p>If you look at the documentation of <a href="https://pytorch.org/docs/master/generated/torch.nn.BCEWithLogitsLoss.html" rel="noopener nofollow ugc">torch.nn.BCEWithLogitsLoss</a>, it says “<em>This loss combines a Sigmoid layer and the BCELoss in one single class. This version is more numerically stable than using a plain Sigmoid followed by a BCELoss as…</em>”.</p>
<p>Also, this post may help you too.</p><aside class="quote quote-modified" data-post="5" data-topic="33586">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vmirly1/48/3309_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/bceloss-vs-bcewithlogitsloss/33586/5">BCELoss vs BCEWithLogitsLoss</a> 
  </div>
  <blockquote>
    <a class="mention" href="/u/ptrblck">@ptrblck</a>  Isn’t it the other way around? I thought BCELoss needs to receive the outputs of Sigmoid activation as its input, but the other-one BCEWithLogitsLoss will need the logits as inputs instead of outputs of Sigmoid, since it will apply sigmoid internally. 
Although, the example in the docs do not apply Sigmoid function prior to BCELoss: 
### Example from pytorch-docs:
&gt;&gt;&gt; m = nn.Sigmoid()
&gt;&gt;&gt; loss = nn.BCELoss()
&gt;&gt;&gt; input = torch.randn(3, requires_grad=True)
&gt;&gt;&gt; target = torch.empty(3).ran…
  </blockquote>
</aside>

<p>By the way, <code>sigmoid</code> maps input to <code>[0,1]</code> by default.</p>
<p>bests</p>

username: chaslie
cooked: <p>doh! thanks for that…</p>

username: KFrank
cooked: <p>Hi chaslie!</p>
<aside class="quote no-group" data-username="chaslie" data-post="1" data-topic="81182" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/9fc29f/48.png" class="avatar"> chaslie:</div>
<blockquote>
<p>If I understand the command BCEWithLogitsLoss, it uses a sigmoid function, so the $64k is why do i get values in the outut of BCE which are outside +/-1?</p>
</blockquote>
</aside>
<p>As Doosti explained, <code>BCEWitLogitsLoss</code> (in effect) passes your <code>out</code><br>
tensor through a <code>sigmoid()</code>, not its results, so it won’t be forcing its<br>
results to be in the range <code>[0.0, 1.0]</code>.</p>
<p>Note, however, that <em>logically</em> your <code>data</code> tensor – the labels you pass<br>
to <code>BCEWithLogitsLoss</code> – should be in the range <code>[0.0, 1.0]</code>.  These<br>
labels are normally either the discrete values <code>0.0</code> or <code>1.0</code> (and thought<br>
of as binary class labels) or they are continuous values in <code>[0.0, 1.0]</code><br>
(and thought of as the probability of your sample being in class-“1”).</p>
<p>If you respect this restriction, <code>BCEWithLogitsLoss</code> will not return<br>
negative loss values – its loss values will range from <code>0.0</code> to <code>inf</code>.<br>
(If you do not respect this restriction, you will be calculating a loss<br>
that doesn’t make sense, your gradients won’t make sense, and<br>
your model won’t train properly.)</p>
<p>The fact that the output you get from <code>BCEWithLogitsLoss</code> contains<br>
negative values means that you are breaking this rule and passing<br>
<code>data</code> values outside of <code>[0.0, 1.0]</code>.  (But this is your fault, not the<br>
fault of <code>BCEWithLogitsLoss</code>.)  On the other hand, the fact that the<br>
output contains values greater than 1 is fine, and, in fact<br>
<code>BCEWithLogitsLoss</code> diverges to <code>inf</code> for predictions that are<br>
“completely wrong.”</p>
<p>Best.</p>
<p>K. Frank</p>

username: chaslie
cooked: <p>Hi KFrank,</p>
<p>you hit the nail on the head, the inut was being normalised to (-1,1), changing  this solved the problem…</p>
<p>chaslie</p>