KLDivLoss returns negative value

msqiqi10: <p>With the following code:<br>
<code>s = torch.softmax(torch.randn(100), dim=0)</code><br>
<code>d = torch.zeros(100)</code><br>
<code>d[43] = 1</code><br>
<code>loss = torch.nn.KLDivLoss()</code><br>
<code>loss(s,d)</code><br>
And somehow this returns a negative value, why is it and how am I supposed to fix it?<br>
Thank you.</p>

pierrecurie: <aside class="quote no-group">
<blockquote>
<p>As with <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.NLLLoss" rel="noopener nofollow ugc"> <code>NLLLoss</code> </a>, the input given is expected to contain <em>log-probabilities</em>… The targets are given as <em>probabilities</em> (i.e. without taking the logarithm).</p>
</blockquote>
</aside>
<p>So, dump the softmax</p>

Lin_Jia: <p>I was calculating KL Divergence loss, and it is negative, that leads me to here. Thanks for all your previous answers.</p>
<p>This is the mathematical proof of why KLDLoss should be above zero:</p><aside class="onebox stackexchange" data-onebox-src="https://stats.stackexchange.com/questions/335197/why-kl-divergence-is-non-negative">
  <header class="source">

      <a href="https://stats.stackexchange.com/questions/335197/why-kl-divergence-is-non-negative" target="_blank" rel="noopener nofollow ugc">stats.stackexchange.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stats.stackexchange.com/users/198896/metchaikovsky" target="_blank" rel="noopener nofollow ugc">
    <img alt="meTchaikovsky" src="https://i.stack.imgur.com/e92Ty.jpg?s=256&amp;g=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stats.stackexchange.com/questions/335197/why-kl-divergence-is-non-negative" target="_blank" rel="noopener nofollow ugc">Why KL divergence is non-negative?</a>
</h4>

<div class="tags">
  <strong>information-theory, kullback-leibler</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stats.stackexchange.com/users/198896/metchaikovsky" target="_blank" rel="noopener nofollow ugc">
    meTchaikovsky
  </a>
  on <a href="https://stats.stackexchange.com/questions/335197/why-kl-divergence-is-non-negative" target="_blank" rel="noopener nofollow ugc">10:43AM - 18 Mar 18 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>The cornerstone of the proof is that for KLDLoss(p, q), sum(q) needs to equal one to make sure the loss is above zero. So even if you have p = log_softmax(tensor), you might still get negative values if your target is not a true distribution: sum(q) != 1</p>
<p>Also, see the awesome discussion here:</p><aside class="quote quote-modified" data-post="14" data-topic="16791">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lin_jia/48/22799_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/kl-divergence-produces-negative-values/16791/14">KL Divergence produces negative values</a> 
  </div>
  <blockquote>
    I was calculating KL Divergence loss, and it is negative, that leads me to here. Thanks for all your previous answers. 
This is the mathematical proof of why KLDLoss should be above zero: 
 

The cornerstone of the proof is that for KLDLoss(p, q), sum(q) needs to equal one to make sure the loss is above zero. So even if you have p = log_softmax(tensor), you might still get negative values if your target is not a true distribution: sum(q) != 1
  </blockquote>
</aside>