Torch.nn.functional.kl_div doesn't work as expected

daehwa001210: <pre><code class="lang-auto">with torch.no_grad():
        original_outputs = self.classification_model(original_images.to(device))
        changed_outputs = self.classification_model(changed_images.to(device))

# first(pytorch function)
rewards = torch.sum(
        torch.nn.functional.kl_div(
            changed_outputs.log(), original_outputs, size_average = None, reduction="none"
        ),
        dim=1,
)
print((rewards &lt; 0).sum())

# second(mine)
k = 0
for i in range(batch_size):
        if((original_outputs[i]*(original_outputs[i]/changed_outputs[i]).log()).sum() &lt; 0):
            k += 1
print(k)
</code></pre>
<p>And my classification model has SIGMOID at the end.</p>
<p>Both outputs(original, changed) add up to 1 in 1 dimension… However, when I f.kl_div the two results (original_outputs, changed_ouputs), negative numbers often came out, and it was the same when I made a function.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/d/7d744f8c40335f3cd57b8f8aeb8e4a2886be90d0.png" alt="image" data-base62-sha1="hTOSsAhf0JtZyfEK98qzYFtyKkw" width="388" height="70"></p>

KFrank: <p>Hi Daehwa!</p>
<aside class="quote no-group" data-username="daehwa001210" data-post="1" data-topic="166383" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/daehwa001210/48/55003_2.png" class="avatar"> daehwa001210:</div>
<blockquote>
<p>And my classification model has SIGMOID at the end.</p>
<p>Both outputs(original, changed) add up to 1 in 1 dimension… However, when I f.kl_div the two results (original_outputs, changed_ouputs), negative numbers often came out</p>
</blockquote>
</aside>
<p>This looks a bit odd.  It would be difficult to pass a tensor through <code>sigmoid()</code><br>
and still have a given dimension sum to one.</p>
<p>When given “legal” arguments, <code>kl_div()</code> should, indeed, only return tensors<br>
that sum along the probability-distribution dimension to non-negative values.</p>
<p>Could you verify that both <code>original_outputs</code> and <code>changed_outputs</code> are<br>
valid probability distributions along <code>dim = 1</code>?  All of the values should be<br>
between zero and one and, as you’ve noted, they should sum to one along<br>
<code>dim = 1</code>.</p>
<p>If you are still having your issue, please save a copy of the <code>original_outputs</code><br>
and <code>changed_outputs</code> that lead to negative values for the sums of <code>kl_div()</code><br>
and post a fully-self-contained, runnable script that reproduces your issue<br>
(with hard-coded values for those two tensors), together with its output.</p>
<p>Best.</p>
<p>K. Frank</p>

daehwa001210: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="166383">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>reproduces</p>
</blockquote>
</aside>
<pre><code class="lang-auto">print(torch.sum(original_outputs,dim=1))    # first code
print((torch.sum(original_outputs, dim=1)!= 1).sum())    # second code
</code></pre>
<p>My first code prints 1.000 or 1. by batch_size.</p>
<p>However, the second code is often False when compared to 1.</p>
<p>It’s probably because of the floating point that the error is caused by not adding to 1 perfectly.</p>
<p>It seems that kl_div is also confused with these floating points.</p>
<pre><code class="lang-auto">tensor([4.6966e-10, 7.7465e-25, 2.4687e-21, 1.1027e-20, 6.1598e-15, 3.7311e-14, 1.0000e+00, 1.0739e-20, 1.8067e-15, 7.9641e-18], device='cuda:0')    # original output
tensor([4.2339e-10, 7.5789e-25, 7.2592e-21, 6.8198e-21, 5.8756e-15, 2.1480e-14, 1.0000e+00, 3.6700e-21, 9.5929e-16, 3.5948e-18], device='cuda:0')    # changed output
</code></pre>
<p>here is my hard-coded two tensor.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/0/3064a789384d5d7706c834af8d43c020262f6056.png" data-download-href="https://discuss.pytorch.org/uploads/default/3064a789384d5d7706c834af8d43c020262f6056" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/3064a789384d5d7706c834af8d43c020262f6056_2_690x117.png" alt="image" data-base62-sha1="6U6zk065dbJaa150tjHtdtJiWJE" width="690" height="117" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/3064a789384d5d7706c834af8d43c020262f6056_2_690x117.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/3064a789384d5d7706c834af8d43c020262f6056_2_1035x175.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/3064a789384d5d7706c834af8d43c020262f6056_2_1380x234.png 2x" data-dominant-color="2F3141"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2059×352 48.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I always want kl_div to be output as a positive number. But I think this result is due to the error of sigmoid’s floating point. How should I solve it?</p>
<p>In addition, it’s not related to my problem, but won’t it unintentionally interfere with machine learning if kl_div comes out as a negative number?</p>
<p>Thank you for your help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

daehwa001210: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9ae755eef35f372be9042cd6d431da6ef4b4532e.png" data-download-href="https://discuss.pytorch.org/uploads/default/9ae755eef35f372be9042cd6d431da6ef4b4532e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9ae755eef35f372be9042cd6d431da6ef4b4532e_2_678x500.png" alt="image" data-base62-sha1="m6ladGQZIxNBSGpa8LWLl5mmUHk" width="678" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9ae755eef35f372be9042cd6d431da6ef4b4532e_2_678x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9ae755eef35f372be9042cd6d431da6ef4b4532e_2_1017x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9ae755eef35f372be9042cd6d431da6ef4b4532e_2_1356x1000.png 2x" data-dominant-color="2D3041"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1372×1011 83.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
my CNN model!</p>
<p>sorry. I misspelled softmax as sigmoid in the post above.<br>
Still, the problem remains.</p>

KFrank: <p>Hi Daehwa!</p>
<aside class="quote no-group" data-username="daehwa001210" data-post="3" data-topic="166383" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/daehwa001210/48/55003_2.png" class="avatar"> daehwa001210:</div>
<blockquote>
<p>It seems that kl_div is also confused with these floating points.</p>
<pre><code class="lang-auto">tensor([4.6966e-10, 7.7465e-25, 2.4687e-21, 1.1027e-20, 6.1598e-15, 3.7311e-14, 1.0000e+00, 1.0739e-20, 1.8067e-15, 7.9641e-18], device='cuda:0')    # original output
tensor([4.2339e-10, 7.5789e-25, 7.2592e-21, 6.8198e-21, 5.8756e-15, 2.1480e-14, 1.0000e+00, 3.6700e-21, 9.5929e-16, 3.5948e-18], device='cuda:0')    # changed output
</code></pre>
</blockquote>
</aside>
<p>Yes, I can reproduce your issue using these tensors.</p>
<p>I do believe that this is an issue of (expected) floating-point error.  Because<br>
of round-off error, your probability-distributions tensors don’t sum to exactly<br>
one (although you have to perform the sum in double precision to see this).<br>
Therefore they are not exactly “legal” probability distributions and the<br>
property that <code>kl_div()</code> will always be non-negative won’t necessarily hold<br>
exactly.</p>
<p>Note, although we get a result that is negative, it is quite close to zero, so<br>
the violation of non-negativity is very small – and, indeed, on the order of<br>
round-off error.</p>
<p>Consider this illustration that uses the tensors you posted:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.13.0
&gt;&gt;&gt;
&gt;&gt;&gt; original = torch.tensor([4.6966e-10, 7.7465e-25, 2.4687e-21, 1.1027e-20, 6.1598e-15, 3.7311e-14, 1.0000e+00, 1.0739e-20, 1.8067e-15, 7.9641e-18])
&gt;&gt;&gt; changed = torch.tensor([4.2339e-10, 7.5789e-25, 7.2592e-21, 6.8198e-21, 5.8756e-15, 2.1480e-14, 1.0000e+00, 3.6700e-21, 9.5929e-16, 3.5948e-18])
&gt;&gt;&gt;
&gt;&gt;&gt; changed.sum() - 1            # sums to one in single precision
tensor(0.)
&gt;&gt;&gt; changed.double().sum() - 1   # but not quite exactly one, so not quite a "legal" probability distribution
tensor(4.2342e-10, dtype=torch.float64)
&gt;&gt;&gt;
&gt;&gt;&gt; torch.equal (changed, original)      # not strictly equal
False
&gt;&gt;&gt; torch.allclose (changed, original)   # equal to within some sort of numerical error, so kl_div should be about zero
True
&gt;&gt;&gt;
&gt;&gt;&gt; kld = torch.nn.functional.kl_div (original.log(), changed, reduction = 'none').sum()
&gt;&gt;&gt; kld                                        # negative, but not by much
tensor(-4.3925e-11)
&gt;&gt;&gt; torch.allclose (kld, torch.tensor (0.0))   # zero to within some sort of numerical error
True
</code></pre>
<aside class="quote no-group">
<blockquote>
<p>here is my hard-coded two tensor.</p>
</blockquote>
</aside>
<p>As an aside, it isn’t really useful posting screen-shots of textual information.<br>
If someone wanted to use this in a test or example, they wouldn’t be able<br>
to copy-paste it.</p>
<aside class="quote no-group">
<blockquote>
<p>I always want kl_div to be output as a positive number. But I think this result is due to the error of sigmoid’s floating point. How should I solve it?</p>
</blockquote>
</aside>
<p>As discussed above, yes, I think this is a result of round-off error.  If you<br>
really need the result of <code>kl_div()</code> to be strictly non-negative – and I doubt<br>
that you do – you could clamp the result to be greater than or equal to zero.</p>
<aside class="quote no-group">
<blockquote>
<p>won’t it unintentionally interfere with machine learning if kl_div comes out as a negative number?</p>
</blockquote>
</aside>
<p>No, a negative value for a loss will cause no problem.</p>
<p>It is true that many loss functions are non-negative and only become zero<br>
when the predictions are “perfect” (including <code>kl-div()</code> in the absence of<br>
round-off error), but the optimization and training don’t care.</p>
<p>Consider training a regression with <code>MSELoss()</code> – strictly non-negative and<br>
only zero for perfect predictions.  Then consider the same training with<br>
<code>MSELoss() - 17.0</code>, which can now become negative (and is equal to<br>
<code>-17.0</code> for perfect predictions).  The gradients are the same and the training<br>
is the same.  The optimizer seeks to push the loss function to algebraically<br>
smaller numbers – including more negative numbers that are larger in<br>
absolute value – rather than pushing the loss specifically toward zero.</p>
<p>Best.</p>
<p>K. Frank</p>