BCELoss vs BCEWithLogitsLoss

Shani_Gamrian: <p>What is the advantage of using <code>binary_cross_entropy_with_logits</code> (aka BCE with sigmoid) over the regular <code>binary_cross_entropy</code>? I have a multi-binary classification problem and I’m trying to decide which one to choose.</p>

ptrblck: <p>As you described the only difference is the included sigmoid activation in <code>nn.BCEWithLogitsLoss</code>.<br>
It’s comparable to <code>nn.CrossEntropyLoss</code> and <code>nn.NLLLoss</code>. While the former uses a <code>nn.LogSoftmax</code> activation function internally, you would have to add it in the latter criterion.</p>

Shani_Gamrian: <p>I understand the differences in the implementation, I don’t understand the theoretical advantages of using BCE with sigmoid vs without sigmoid.</p>

ptrblck: <p>Sorry for not being not clear enough.<br>
The sigmoid activation should be applied in both cases.<br>
While <code>nn.BCEWithLogitsLoss</code> will apply it internally for you, you should add it manually if you are using <code>nn.BCELoss</code>.</p>

vmirly1: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Isn’t it the other way around? I thought <code>BCELoss</code> needs to receive the outputs of <code>Sigmoid</code> activation as its input, but the other-one <code>BCEWithLogitsLoss</code> will need the logits as inputs instead of outputs of <code>Sigmoid</code>, since it will apply sigmoid internally.</p>
<p>Although, the example in the docs do not apply Sigmoid function prior to BCELoss:</p>
<pre><code class="lang-auto">### Example from pytorch-docs:
&gt;&gt;&gt; m = nn.Sigmoid()
&gt;&gt;&gt; loss = nn.BCELoss()
&gt;&gt;&gt; input = torch.randn(3, requires_grad=True)
&gt;&gt;&gt; target = torch.empty(3).random_(2)
&gt;&gt;&gt; output = loss(m(input), target)
&gt;&gt;&gt; output.backward()
</code></pre>
<p>So, I suppose the loss should be computed as</p>
<pre><code class="lang-auto">logits = m(input)
output = loss(torch.sigmoid(logits), target)
</code></pre>
<p>Is that right?</p>

ptrblck: <p>Yes, you are completely right and I’ve mixed up both names. I’ll edit my post to get it right.</p>
<p>In the example the <code>nn.Sigmoid</code> will be applied by <code>m(input)</code>, so it should be right.<br>
Your code should apply the sigmoid function twice (once in <code>m()</code> and the second time using <code>torch.sigmoid</code>).</p>

vmirly1: <p>Yes, that’s right. I somehow over-looked the definition of <code>m</code>.</p>
<p>Thanks</p>

v1vekkumar: <p><a class="mention" href="/u/shani_gamrian">@Shani_Gamrian</a> Use BCEWithLogitsLoss -  it’s stable than using a plain <code>Sigmoid followed by a</code>BCELoss` (uses log-sum-exp trick for numerical stability)</p>
<p>see <a href="https://github.com/pytorch/pytorch/issues/751" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/751</a></p>

sid-ls: <aside class="quote no-group quote-modified" data-username="ptrblck" data-post="2" data-topic="33586">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>As you described the only difference is the included sigmoid activation in <code>nn.BCELoss</code> .</p>
</blockquote>
</aside>
<p>Just to clarify, if using <code>nn.BCEWithLogitsLoss(target, output)</code>, <code>output</code> should be passed through a sigmoid and only then to <code>BCEWithLogitsLoss</code>? I don’t understand why one would pass it through a sigmoid twice because x is already a probability after passing through one sigmoid.</p>

ptrblck: <p>No, that was a typo which <a class="mention" href="/u/vmirly1">@vmirly1</a> already corrected.<br>
You should pass logits to <code>nn.BCEwithLogitsLoss</code> and probabilities (using sigmoid) to <code>nn.BCELoss</code>.</p>
<p>Also, make sure to pass the model output first and then the target to the criterion.</p>

shahriarshayesteh: <p>Sorry for asking my question here, I’m doing wod2vec with negative sampling and I had problem using nn.NLLLoss to train my network and I was reading pytorch loss functions, then I found out `binary_cross_entropy_with_logits, it says that This loss combines a Sigmoid layer and the BCELoss in one single class and This is used for measuring the error of a reconstruction in for example an auto-encoder. Note that the targets yy should be numbers between 0 and 1. So do you think I made a right choice for my loss function?<br>
Thanks</p>

ptrblck: <p>Negative sampling might work with <code>nn.BCE(WithLogits)Loss</code>, but might be inefficient, as you would probably calculate the non-reduced loss for all classes and mask them afterwards.<br>
Some implementations sample the negative classes beforehand and calculate the bce loss manually, e.g. as described <a href="https://adoni.github.io/2017/11/08/word2vec-pytorch/" rel="nofollow noopener">here</a>.</p>

n0obcoder: <p>but what exactly is the difference between the two? i mean how is BCEWithLogitsLoss more stable? what  stability means here. It would be great if you could explain it to me in little more detail. Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

Santosh-Gupta: <p>Just to be super duper clear, do <strong>Not</strong> pass inputs through a sigmoid if using <code>BCEWithLogitsLoss</code> ?</p>
<p>And for the outputs, just use ‘1’ for True classes, or ‘0’ for the false classes?</p>

ptrblck: <aside class="quote no-group" data-username="Santosh-Gupta" data-post="14" data-topic="33586">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/santosh-gupta/48/30716_2.png" class="avatar"> Santosh-Gupta:</div>
<blockquote>
<p>Just to be super duper clear, do <strong>Not</strong> pass inputs through a sigmoid if using <code>BCEWithLogitsLoss</code> ?</p>
</blockquote>
</aside>
<p>Yes.</p>
<ul>
<li>logits → <code>nn.BCEWithLogitsLoss</code></li>
<li>logits → sigmoid → <code>nn.BCELoss</code></li>
</ul>
<aside class="quote no-group" data-username="Santosh-Gupta" data-post="14" data-topic="33586">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/santosh-gupta/48/30716_2.png" class="avatar"> Santosh-Gupta:</div>
<blockquote>
<p>And for the outputs, just use ‘1’ for True classes, or ‘0’ for the false classes?</p>
</blockquote>
</aside>
<p>That would be the standard definition, but you can basically define True/False, Positive/Negative as you wish.</p>

bigyeet: <p>Thanks everyone for your input.</p>
<p>If the network outputs logits and is trained with BCEWithLogitsLoss, during inference, should I use the logits directly as a probability or should I apply sigmoid first?</p>
<p>edit:<br>
Now that I think about it, the logit could totally be negative or larger than 1, making it unsuitable as a probability. So I should apply a sigmoid during inference. Please confirm</p>

G.M: <p><code>y*ln(sig(x)) + (1 - y)*ln(1 - sig(x))</code> = <code>y*ln(1 / (1 + e^-x))+(1 - y)*ln(1 - 1 / (1 + e^-x))</code> = <code>ln(e^yx / (e^x +  1))</code> If I’m correct, it works like this.</p>

ptrblck: <p>The values of the logits might be harder to interpret, so you might want to apply a sigmoid to get the probabilities.<br>
Note that a logit of 0 will map to <code>p=0.5</code>, so you could still easily get the prediction for this simple threshold with logits.</p>

Dmitry_Rybakov: <p>sigmoid( ln(x/(1-x) ) ) = x</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://www.wolframalpha.com/static/favicon.ico" class="site-icon" width="64" height="64">
      <a href="https://www.wolframalpha.com/input/?i=sigmoid%28ln%28x%2F%281-x%29%29%29" target="_blank" rel="nofollow noopener">wolframalpha.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://www.wolframalpha.com/static/share_3G6HuGr6.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://www.wolframalpha.com/input/?i=sigmoid%28ln%28x%2F%281-x%29%29%29" target="_blank" rel="nofollow noopener">sigmoid(ln(x/(1-x))) - Wolfram|Alpha</a></h3>

<p>Wolfram|Alpha brings expert-level knowledge and capabilities to the broadest possible range of people—spanning all professions and education levels.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


snip3r77: <p>for both BCEWithLogitsLoss and CrossEntropyLoss ( 1 step )</p>
<p>we will need to do this when doing inferencing?</p>
<pre><code class="lang-auto">logps = model(img)
ps = torch.exp(logps)
</code></pre>
<p>Also, even if it’s 2steps(i.e logsoftmax + nlllosss) the above still applies right?</p>
<p>Thanks</p>