Binary Cross Entropy with logits does not work as expected

smonsays: <p>While tinkering with the <a href="https://github.com/pytorch/examples/blob/master/vae/main.py" rel="nofollow noopener">official code example for Variational Autoencoders</a>, I experienced some unexpected behaviour with regard to the Binary Cross-Entropy loss. When I use <code>F.binary_cross_entropy</code> in combination with the <code>sigmoid</code> function, the model trains as expected on MNIST.  However, when changing to the <code>F.binary_cross_entropy_with_logits</code> function, the loss suddenly becomes arbitrarily small during training and the model no longer produces meaningful results.</p>
<pre><code class="lang-auto"># For this loss function the loss becomes arbitrarily small 
BCE = F.binary_cross_entropy_with_logits(recon_x, x.view(-1, 784), reduction='sum') / x.shape[0]

# For this loss function, the training works as expected
BCE = F.binary_cross_entropy(torch.sigmoid(recon_x), x.view(-1, 784), reduction='sum') / x.shape[0]
</code></pre>
<p>To my understanding, the only difference between the two approaches should be numerical stability. Am I missing something?</p>

KFrank: <p>Hello Simon!</p>
<aside class="quote no-group" data-username="smonsays" data-post="1" data-topic="55930" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e47774/48.png" class="avatar"> smonsays:</div>
<blockquote>
<p>While tinkering with the <a href="https://github.com/pytorch/examples/blob/master/vae/main.py" rel="noopener nofollow ugc">official code example for Variational Autoencoders</a>, I experienced some unexpected behaviour with regard to the Binary Cross-Entropy loss. When I use <code>F.binary_cross_entropy</code> in combination with the <code>sigmoid</code> function, the model trains as expected on MNIST.  However, when changing to the <code>F.binary_cross_entropy_with_logits</code> function, the loss suddenly becomes arbitrarily small during training and the model no longer produces meaningful results.</p>
<pre><code class="lang-auto"># For this loss function the loss becomes arbitrarily small 
BCE = F.binary_cross_entropy_with_logits(recon_x, x.view(-1, 784), reduction='sum') / x.shape[0]

# For this loss function, the training works as expected
BCE = F.binary_cross_entropy(torch.sigmoid(recon_x), x.view(-1, 784), reduction='sum') / x.shape[0]
</code></pre>
<p>To my understanding, the only difference between the two approaches should be numerical stability. Am I missing something?</p>
</blockquote>
</aside>
<p>I agree with your understanding, and looking at the two lines of<br>
code you posted, I don’t see anything suspicious (although I miss<br>
things all the time …).</p>
<p>If I had to guess, I would guess that you have a typo somewhere<br>
else in your code that causes the two runs to differ.</p>
<p>However, we can start by checking out some basics:</p>
<p>Here I run the example given in the documentation for<br>
<code>torch.nn.functional.binary_cross_entropy</code>.   (Please note<br>
that I am running this test, for whatever reason, with pytorch 0.3.0.)</p>
<p>Here is the script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)
torch.manual_seed (2019)
input =  torch.autograd.Variable (torch.randn ((3, 2)))
print (input)
target = torch.autograd.Variable (torch.rand ((3, 2)))
print (target)
loss_plain = torch.nn.functional.binary_cross_entropy (torch.sigmoid (input), target)
print (loss_plain)
loss_logits = torch.nn.functional.binary_cross_entropy_with_logits (input, target)
print (loss_logits)
print (loss_plain - loss_logits)
</code></pre>
<p>And here is the output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
0.3.0b0+591e73e
&gt;&gt;&gt; torch.manual_seed (2019)
&lt;torch._C.Generator object at 0x00000207759A60F0&gt;
&gt;&gt;&gt; input =  torch.autograd.Variable (torch.randn ((3, 2)))
&gt;&gt;&gt; print (input)
Variable containing:
-0.1187  0.2110
 0.7463 -0.6136
-0.1186  1.5565
[torch.FloatTensor of size 3x2]

&gt;&gt;&gt; target = torch.autograd.Variable (torch.rand ((3, 2)))
&gt;&gt;&gt; print (target)
Variable containing:
 0.7628  0.0721
 0.2208  0.3979
 0.6338  0.1922
[torch.FloatTensor of size 3x2]

&gt;&gt;&gt; loss_plain = torch.nn.functional.binary_cross_entropy (torch.sigmoid (input), target)
&gt;&gt;&gt; print (loss_plain)
Variable containing:
 0.8868
[torch.FloatTensor of size 1]

&gt;&gt;&gt; loss_logits = torch.nn.functional.binary_cross_entropy_with_logits (input, target)
&gt;&gt;&gt; print (loss_logits)
Variable containing:
 0.8868
[torch.FloatTensor of size 1]

&gt;&gt;&gt; print (loss_plain - loss_logits)
Variable containing:
1.00000e-08 *
 -5.9605
[torch.FloatTensor of size 1]
</code></pre>
<p>And, indeed, the two expressions are the same (up to floating-point<br>
precision).</p>
<p>Is there any way you can capture a specific instance of your<br>
<code>recon_x</code> and <code>x.view()</code> so that you can pump what you know<br>
are the same values into your two cross-entropy expressions?</p>
<p>Is there any way you track your loss (your <code>BCE</code>) on a step-by-step<br>
basis so you can see when they first diverge?</p>
<p>The thing that is noteworthy to me is that you say that the less<br>
numerically stable version (regular bce) works, while the more<br>
stable version (bce_with_logits) quits working at some point.<br>
This is backwards of what one might expect.</p>
<p>Perhaps there is something about your model that puts it on<br>
the edge of being poorly behaved.  In such a case it could be<br>
plausible that, by happenstance, the bce version stays in a<br>
well-behaved region, but small differences due to using the<br>
bce_with_logits version cause it to drift into a poorly-behaved<br>
region.</p>
<p>If this were the case I would expect that other perturbations<br>
such as starting with different initial weights, or using a different<br>
optimization algorithm or learning rate could cause the training<br>
to “randomly” end up being well behaved or poorly behaved.</p>
<p>So:</p>
<p>I would</p>
<ol>
<li>proofread the code to make sure there isn’t some outright error</li>
<li>check that your model and data are reasonably well behaved<br>
and stable with respect to perturbations of the details of your<br>
training</li>
<li>track your bce and bce_logits step by step to find out where<br>
they first diverge, and drill down with the values that immediately<br>
precede the divergence.</li>
</ol>
<p>Have fun!</p>
<p>K. Frank</p>

smonsays: <p>Hello KFrank,</p>
<p>thank you for your elaborate response. I have uploaded an instance of tensors for which the two losses diverge <a href="https://send.firefox.com/download/bad4a176e394c37e/#ossrWpRTkA4GE-9aScGDzg" rel="nofollow noopener">here</a>. This is the resulting output for it:</p>
<pre><code class="lang-python">import torch

x_view  = torch.load("x_view.pt")
x_recon = torch.load("x_recon.pt")

BCE1 = torch.nn.functional.binary_cross_entropy_with_logits(x_recon, x_view, reduction='sum')
BCE2 = torch.nn.functional.binary_cross_entropy(torch.sigmoid(x_recon), x_view, reduction='sum')

print("BCE loss with logits: ", BCE1)  # -2.3662e+08
print("BCE loss with sigmoid: ", BCE2) # -379848.2500
print("Loss difference: ", BCE1-BCE2)  # -2.3624e+08
</code></pre>
<p>The <code>x_recon</code> values are all very small (around -1.0e+05). However, I am not sure why the version without logits behaves so differently in this regime. Do you have any insights in this regard?</p>
<p>In the mean time I’ll try to figure out the minimal changes to the official example that are necessary to reproduce the odd behaviour and post the code here.</p>

KFrank: <p>Hi Simon!</p>
<aside class="quote no-group quote-modified" data-username="smonsays" data-post="3" data-topic="55930" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e47774/48.png" class="avatar"> smonsays:</div>
<blockquote>
<p>I have uploaded an instance of tensors for which the two losses diverge</p>
<pre data-code-wrap="python"><code class="lang-python"># ...
print("BCE loss with logits: ", BCE1)  # -2.3662e+08
print("BCE loss with sigmoid: ", BCE2) # -379848.2500
print("Loss difference: ", BCE1-BCE2)  # -2.3624e+08
</code></pre>
<p>The <code>x_recon</code> values are all very small (around -1.0e+05). However, I am not sure why the version without logits behaves so differently in this regime. Do you have any insights in this regard?</p>
</blockquote>
</aside>
<p>First, unfortunately, I’m not able to load your sample data with<br>
my creaking, wheezing, 0.3.0-version of pytorch.  If they’re not<br>
too large, could you post them as text files?</p>
<p>You say, “The <code>x_recon</code> values are all very small (around -1.0e+05).”<br>
You give a numerical value of -10,000.  I would call this “a rather<br>
large negative number.”  (“Small,” to me, has the connotation of<br>
“close to zero.”)</p>
<p>Anyway, let’s go with -10,000.  That is, your logits (inputs)<br>
are rather large negative numbers.  So your probabilities<br>
(sigmoid (logit)) are all (positive) numbers quite close to zero.<br>
But (with 32-bit floating-point) they underflow and become<br>
exactly zero.  (32-bit sigmoid underflows to zero somewhere<br>
around <code>sigmoid (-90.0)</code>.)</p>
<p>Given this, I’m surprised you’re not getting <code>NaN</code>s (from the<br>
<code>log (0.0)</code> inside of <code>binary_cross_entropy()</code>).</p>
<p>Anyway, could you tell us the shape of <code>x_recon</code> and <code>x_view</code>,<br>
as well as the (algebraic) minima and maxima of <code>x_recon</code><br>
and <code>x_view</code>?</p>
<p>Assuming that your <code>x_recon</code> really have become something<br>
like -10,000, it’s not surprising that you’re getting weird results<br>
(at least for plain bce) – you’ve long since passed into the<br>
range where <code>sigmoid()</code> underflows.  (This still doesn’t explain<br>
why you’re getting seeming good results with plain bce, but<br>
things break down with bce_with_logits.)</p>
<p>Best.</p>
<p>K. Frank</p>

smonsays: <aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="55930">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>You say, “The <code>x_recon</code> values are all very small (around -1.0e+05).”<br>
You give a numerical value of -10,000. I would call this “a rather<br>
large negative number.” (“Small,” to me, has the connotation of<br>
“close to zero.”)</p>
</blockquote>
</aside>
<p>Ah sorry about that, you are of course right, the <code>x_recon</code> values are large negative numbers. Only the sigmoid is very small.</p>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="55930">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Anyway, could you tell us the shape of <code>x_recon</code> and <code>x_view</code> ,<br>
as well as the (algebraic) minima and maxima of <code>x_recon</code><br>
and <code>x_view</code> ?</p>
</blockquote>
</aside>
<p>Both have dimensions <code>torch.Size([100, 784])</code>. The first dimension is the batch size, the second is the number of pixels in a MNIST training image. The extrema are:</p>
<pre><code class="lang-auto">torch.min(x_recon) # -16971.4434
torch.max(x_recon) # 15807.5469
torch.min(x_view) # -0.4242
torch.max(x_view) # 2.8215)
</code></pre>
<p>So it turns out there are also very big entries in <code>x_recon</code> that saturate the sigmoid towards 1.</p>
<aside class="quote no-group" data-username="smonsays" data-post="3" data-topic="55930">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e47774/48.png" class="avatar"> smonsays:</div>
<blockquote>
<p>In the mean time I’ll try to figure out the minimal changes to the official example that are necessary to reproduce the odd behaviour and post the code here.</p>
</blockquote>
</aside>
<p>I created a demo with minimal changes to the original example that reproduces the odd behaviour <a href="https://gist.github.com/smonsays/dfdde77a54d5f287d32f0a66b725f59b" rel="noopener nofollow ugc">here</a>. The relevant change was the normalization of the data to zero mean and unit variance (i.e. <code>transform = transforms.Normalize((0.1307,), (0.3081,))</code>).</p>

KFrank: <p>Hi Simon!</p>
<aside class="quote no-group" data-username="smonsays" data-post="5" data-topic="55930" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e47774/48.png" class="avatar"> smonsays:</div>
<blockquote>
<p>Both have dimensions <code>torch.Size([100, 784])</code>. The first dimension is the batch size, the second is the number of pixels in a MNIST training image. The extrema are:</p>
<pre><code class="lang-auto">torch.min(x_recon) # -16971.4434
torch.max(x_recon) # 15807.5469
torch.min(x_view) # -0.4242
torch.max(x_view) # 2.8215)
</code></pre>
<p>So it turns out there are also very big entries in <code>x_recon</code> that saturate the sigmoid towards 1.</p>
</blockquote>
</aside>
<p>Okay, this makes more sense.</p>
<p>The <code>input</code> and <code>target</code> passed to <code>binary_cross_entropy()</code><br>
are both supposed to be probabilities, that is, numbers between<br>
0 and 1 (with singularities occurring at 0 and 1).</p>
<p>Your <code>target</code> contains negative numbers, which are not valid<br>
probabilities.</p>
<p>(Because you pass your <code>x_recon</code> through <code>sigmoid()</code>, your <code>input</code><br>
will always contain valid probabilities, although they will sometimes<br>
saturate at the singular 0 and 1.)</p>
<p>So my guess is that your bogus <code>target</code> values are causing your<br>
training to drive your inputs to the large values that saturate<br>
<code>sigmoid()</code>.  (Why you’re not getting <code>NaN</code>s, I don’t know.)</p>
<p>Given your bogus <code>target</code>, I’m not surprised that you’re getting<br>
weird results.  (I can come up with plausible speculations about<br>
why plain bce and bce_with_logits differ, but that’s not really the<br>
point.)</p>
<p>Figure out how to fix your inputs to <code>binary_cross_entropy()</code>,<br>
specifically <code>x_view</code>, and see if that cleans things up, or at<br>
least improves the situation.</p>
<aside class="quote no-group">
<blockquote>
<p>I created a demo with minimal changes to the original example that reproduces the odd behaviour <a href="https://gist.github.com/smonsays/dfdde77a54d5f287d32f0a66b725f59b" rel="noopener nofollow ugc">here</a>.</p>
</blockquote>
</aside>
<p>(I haven’t tried to run your demo, because, among other reasons,<br>
it likely won’t run with my decrepit pytorch 0.3.0.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

smonsays: <p>Okay, so normalization to zero mean, unit variance should be removed in this case as it violates the probabilistic interpretation of the images. That solves my issue, thank you!</p>