Negative values in unreduced kl_div result

CharlesLu: <p>Hi there!</p>
<p>When I use torch.nn.functional.kl_div(), I notice that while the reduced mean of result is positive, some values in the unreduced result are negative. I was wondering if it is the correct behavior, or I made some mistake in my codes?</p>
<p>Thank you very much!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4bc5f6faadd767de0cc682105c413018e0bdaa59.png" data-download-href="https://discuss.pytorch.org/uploads/default/4bc5f6faadd767de0cc682105c413018e0bdaa59" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4bc5f6faadd767de0cc682105c413018e0bdaa59_2_518x342.png" alt="image" data-base62-sha1="aOjWLsKEmuRufDedDdiNV7TigOZ" width="518" height="342" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4bc5f6faadd767de0cc682105c413018e0bdaa59_2_518x342.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4bc5f6faadd767de0cc682105c413018e0bdaa59_2_777x513.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4bc5f6faadd767de0cc682105c413018e0bdaa59_2_1036x684.png 2x" data-dominant-color="F0F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1445×956 87.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

CharlesLu: <p>I just tried a toy example and got the same result, so I guess it’s the correct behavior of kl_div…<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4d23728274f26424a3812798523c77078e935266.png" data-download-href="https://discuss.pytorch.org/uploads/default/4d23728274f26424a3812798523c77078e935266" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d23728274f26424a3812798523c77078e935266_2_345x232.png" alt="image" data-base62-sha1="b0oI40LQtRWdhITyWaxCqZyaGt8" width="345" height="232" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d23728274f26424a3812798523c77078e935266_2_345x232.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d23728274f26424a3812798523c77078e935266_2_517x348.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d23728274f26424a3812798523c77078e935266_2_690x464.png 2x" data-dominant-color="EFF1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">962×646 48.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m still a little confused about how the mean of unreduced result is guaranteed to be non-negative, though I know KL divergence should be non-negative. Any explanation is appreciated!</p>

KFrank: <p>Hi Charles!</p>
<aside class="quote no-group" data-username="CharlesLu" data-post="2" data-topic="93102" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/charleslu/48/59531_2.png" class="avatar"> CharlesLu:</div>
<blockquote>
<p>I’m still a little confused about how the mean of unreduced result is guaranteed to be non-negative, though I know KL divergence should be non-negative.</p>
</blockquote>
</aside>
<p>You are correct that the KL divergence is non-negative, and you are<br>
also correct that the individual terms returned by pytorch’s “unreduced”<br>
<code>kl_div()</code> can be negative.</p>
<p>The reason is that, by definition, the KL divergence is taken between<br>
two probability distributions, not between two arbitrary sets of numbers.<br>
If you apply the KL-divergence formula to arbitrary numbers, the result<br>
is not guaranteed to be non-negative.</p>
<p>Your <code>a</code> and <code>b</code> contain three elements.  Let’s therefore say that you are<br>
working with a three-class problem.</p>
<p>After applying <code>log_softmax()</code>, your <code>a</code> is a log-probability distribution<br>
over the three classes, and satisfies <code>a.exp().sum() = 1.0</code>, that is,<br>
the three probabilities sum to one, as is required of a probability<br>
distribution.  Similarly (after applying <code>softmax()</code>) your <code>b</code> is a probability<br>
distribution over the three classes and sums to one.</p>
<p>The result of the unreduced <code>kl_div()</code> is simply the three terms in<br>
the KL-divergence formula before they are summed over the three<br>
classes.  Not being summed yet, the individual terms “can’t know”<br>
that they come from proper probability distributions that sum to one,<br>
and need not be non-negative.  Only when summed together to form<br>
the full KL divergence do they produce a result that is guaranteed to<br>
be non-negative.</p>
<p>It is a theorem – and, in the discrete case, the result of a not particularly<br>
difficult calculation – that the KL divergence is non-negative, but the<br>
theorem (and calculation) rely on the KL-divergence formula being<br>
applied to legitimate probability distributions.</p>
<p>Best.</p>
<p>K. Frank</p>

CharlesLu: <p>Thank you! Your answer perfectly resolves my confusion.</p>