KL Divergence produces negative values

WERush: <p>When I use the nn.KLDivLoss(), the KL gives the negative values. For example,</p>
<pre><code class="lang-python">a1 = Variable(torch.FloatTensor([0.1,0.2]))
a2 = Variable(torch.FloatTensor([0.3, 0.6]))
a3 = Variable(torch.FloatTensor([0.3, 0.6]))
a4 = Variable(torch.FloatTensor([-0.3, -0.6]))
a5 = Variable(torch.FloatTensor([-0.3, -0.6]))

c1 = nn.KLDivLoss()(a1,a2) #==&gt; -0.4088
c2 = nn.KLDivLoss()(a2,a3) #==&gt; -0.5588
c3 = nn.KLDivLoss()(a4,a5) #==&gt; 0
c4 = nn.KLDivLoss()(a3,a4) #==&gt; 0
c5 = nn.KLDivLoss()(a1,a4) #==&gt; 0

</code></pre>
<p>In theory, the KLDloss should not be negative. Can anyone explain that?</p>
<p>Thanks.</p>

agadetsky: <p>your variables a1, a2, a3, a4, a5 are not proper distributions, i.e. they don’t sum up to 1.</p>

WERush: <pre><code class="lang-python">a2 = Variable(torch.FloatTensor([0.3, 0.6, 0.1]))
a1 = Variable(torch.FloatTensor([0.4, 0.5, 0.1]))

c1 = nn.KLDivLoss()(a1,a2) #==&gt; -0.4426
</code></pre>

tom: <p>They still aren’t distributions. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Keep in mind that the loss functions take batches. So you’d want to unsqueeze(0).</p>
<p>Best regards</p>
<p>Thomas</p>

WERush: <pre><code class="lang-python">a1.unsqueeze_(0)
a2.unsqueeze_(0)

c1 = nn.KLDivLoss()(a1,a2) #==&gt; -0.4426 

</code></pre>

tom: <p>…and x (but not y) should be logprobs, say the doc. My apologies for missing that at first.</p>
<p>Best regards</p>
<p>Thomas</p>
<p><strong>Edit</strong>: so back at the (pytorch 0.4) computer</p>
<pre><code class="lang-auto">import torch
a2 = torch.FloatTensor([0.3, 0.6, 0.1])
a1 = torch.FloatTensor([0.4, 0.5, 0.1])
c1 = torch.nn.KLDivLoss(size_average=False)(a1.log(),a2)
print(c1)
</code></pre>
<p>gives <code>2.3088</code>, the same as explicitly computing</p>
<pre><code class="lang-auto">(a2*(a2.log()-a1.log())).sum()
</code></pre>
<p>Note that PyTorch <a href="https://github.com/pytorch/pytorch/issues/2259" rel="nofollow noopener">erroneously averages</a> the terms by default rather than summing them.</p>

WERush: <p>OK, I get it. Thanks very much.</p>

Pei_Wang: <p>I still get negative output</p>

RylanSchaeffer: <p>Same. I created a post over here since my negative KL divergence values might be caused by the output of torch.nn.functional.softmax producing vectors that don’t sum to 1.</p>
<aside class="quote quote-modified" data-post="1" data-topic="59516">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rylanschaeffer/48/17130_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/torch-nn-functionals-softmax-does-not-sum-to-1/59516">Torch.nn.functional's softmax does not sum to 1</a> 
  </div>
  <blockquote>
    I started receiving negative KL divergences between a target Dirichlet distribution and my model’s output Dirichlet distribution. Someone online suggested that this might be indicative that the parameters of the Dirichlet distribution don’t sum to 1. I thought this was ridiculous since the output of the model is passed through 
output = F.softmax(self.weights(x), dim=1) 
But after looking into it more closely, I found that  torch.all(torch.sum(output, dim=1) == 1.) returns False! Looking at the …
  </blockquote>
</aside>


MFajcik1: <p>Did you normalized values with log_softmax?</p>
<pre><code class="lang-auto">torch.nn.KLDivLoss(size_average=False)(F.log_softmax(scores, -1), targets)
</code></pre>

William_Correa: <p>Hi Tom. May I ask why those values are not stll distributions ? Thanks</p>

tom: <p>Because they’re a batch of three things with values  0.3, 0.6, 0.1 individually for a2 rather than a single batch of three values.<br>
Also, as noted in the post below (which has the correct use except for not using <code>torch.tensor</code> as one would now), a1 should be passed as log probabilities.</p>
<p>Best regards</p>
<p>Thomas</p>

Lin_Jia: <p>I was calculating KL Divergence loss, and it is negative, that leads me to here. Thanks for all your previous answers.</p>
<p>This is the mathematical proof of why KLDLoss should be above zero:<br>
</p><aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stats.stackexchange.com/questions/335197/why-kl-divergence-is-non-negative" target="_blank" rel="noopener nofollow ugc">stats.stackexchange.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stats.stackexchange.com/users/198896/metchaikovsky" target="_blank" rel="noopener nofollow ugc">
    <img alt="meTchaikovsky" src="https://i.stack.imgur.com/e92Ty.jpg?s=128&amp;g=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stats.stackexchange.com/questions/335197/why-kl-divergence-is-non-negative" target="_blank" rel="noopener nofollow ugc">Why KL divergence is non-negative?</a>
</h4>

<div class="tags">
  <strong>information-theory, kullback-leibler</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stats.stackexchange.com/users/198896/metchaikovsky" target="_blank" rel="noopener nofollow ugc">
    meTchaikovsky
  </a>
  on <a href="https://stats.stackexchange.com/questions/335197/why-kl-divergence-is-non-negative" target="_blank" rel="noopener nofollow ugc">10:43AM - 18 Mar 18 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>The cornerstone of the proof is that for KLDLoss(p, q), sum(q) needs to equal one to make sure the loss is above zero. So even if you have p = log_softmax(tensor), you might still get negative values if your target is not a true distribution: sum(q) != 1</p>

kinwai_cheuk: <p>I am now using pytorch 1.6. The result is 0.0231 instead of 2.3088.</p>

tom: <p>Yeah, I think I might have missed a e-2 in my post.</p>

ube: <pre><code class="lang-auto">f=np.array([5.96157837e-23, 1.00000000e+00, 1.22259395e-20, 1.33735148e-23,
 4.61867503e-18, 1.12406575e-20 ,1.61654594e-18, 4.05129215e-16,
 4.42485638e-19 ,2.47655932e-18])
g=np.array([6.6305534e-17,1.0000000e+00 ,1.1786218e-13 ,2.3428323e-22, 5.9469584e-13,
            5.7867524e-19, 5.0707482e-17, 4.4700832e-13, 1.3639888e-15, 1.2760710e-15])


div_measure= lambda a,b :
print(sum(f), sum(g))
print(np.multiply(f,np.log(np.divide(f,g))).sum())
</code></pre>
<p>This will output about -2.917560</p>

bkoyuncu: <p>These are not clearly summing up to 1</p>

Wu_Zq: <p>To avoid underflow issues when computing this quantity, this loss expects the argument <code>input</code> in the log-space. <a href="https://pytorch.org/docs/stable/generated/torch.nn.KLDivLoss.html" class="inline-onebox" rel="noopener nofollow ugc">KLDivLoss — PyTorch 2.3 documentation</a></p>
<p>here is an example:</p>
<pre><code class="lang-auto">import numpy as np
import torch
import torch.nn as nn


# KL divergence
# D(P,Q): P=real prob dist, Q=detected probability distribution
# D(P,Q) = Σ (- P * logQ + P * logP), 对所有 x求和, 其中 P=P(x), Q=Q(x)
# https://pytorch.org/docs/stable/generated/torch.nn.KLDivLoss.html
def test1():
    targets = [0.2, 0.3, 0.5]  # P-true
    inputs = [0.5, 0.3, 0.2]  # P-predict

    loss_fn = nn.KLDivLoss(reduction='sum')
    log_inputs = np.log(inputs)  # note here, in torch, assume inputs already log
    loss = loss_fn(torch.tensor(log_inputs), torch.tensor(targets))
    print(f'loss_1={loss}')

    # manual
    loss_2 = 0
    for idx, px in enumerate(targets):
        qx = inputs[idx]
        loss_2 += px * np.log(px) - px * np.log(qx)
    print(f'loss_2={loss_2}')
    # loss_1 = 0.2748872074901785
    # loss_2 = 0.2748872195622465


if __name__ == '__main__':
    test1()
</code></pre>