KL divergence different results from tf

razvanc92: <p>I’ve noticed that the pytorch implementation of KL divergence yells different results from the tensorflow implementation. The results differ significantly (0.20, and 0.14) and I was curios what could be the reason. Below you can find a small example. Any help will be more than appreciated.</p>
<pre><code class="lang-auto">import tensorflow as tf
import numpy as np
import torch
from torch.distributions.kl import kl_divergence
tf.enable_eager_execution()

preds = np.array([1.9417487e-03, 9.9999997e-10, 5.8252434e-03, 9.9999997e-10, 3.8834962e-03,
 8.1553400e-02, 3.6893204e-01, 5.2427185e-01, 7.7669914e-03, 5.8252434e-03,
 9.9999997e-10]).astype('float32')

labels = np.array([1.0362695e-02, 9.9999997e-10, 9.9999997e-10, 9.9999997e-10, 3.1088084e-02,
 9.0673573e-02, 3.4974092e-01, 5.1036268e-01, 2.5906744e-03, 9.9999997e-10,
 5.1813480e-03]).astype('float32')

preds_tf = tf.distributions.Categorical(probs=tf.convert_to_tensor(preds))
labels_tf = tf.distributions.Categorical(probs=tf.convert_to_tensor(labels))
tf_res = tf.distributions.kl_divergence(preds_tf, labels_tf)

preds_torch = torch.distributions.Categorical(probs=torch.from_numpy(preds))
labels_torch = torch.distributions.Categorical(probs=torch.from_numpy(labels))
torch_res = kl_divergence(preds_torch, labels_torch)

print(tf_res.numpy(), torch_res.item())
</code></pre>

Nikronic: <p>Hi,<br>
I have not read the <code>distribution</code> package source code, but from what I know from the C++ source code, I prefer using <a href="https://pytorch.org/docs/master/nn.html?highlight=kldiv#torch.nn.KLDivLoss" rel="noopener nofollow ugc">torch.nn.functional.kl_div</a> function to calculate the divergence.</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/35fed93b1ef05175143f883c6f89f06c6dd9429b/aten/src/ATen/native/Loss.cpp#L71">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/35fed93b1ef05175143f883c6f89f06c6dd9429b/aten/src/ATen/native/Loss.cpp#L71" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/35fed93b1ef05175143f883c6f89f06c6dd9429b/aten/src/ATen/native/Loss.cpp#L71" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/35fed93b1ef05175143f883c6f89f06c6dd9429b/aten/src/ATen/native/Loss.cpp#L71</a></h4>



    <pre class="onebox"><code class="lang-cpp">
      <ol class="start lines" start="61" style="counter-reset: li-counter 60 ;">
          <li>  return apply_loss_reduction(output, reduction);</li>
          <li>}</li>
          <li></li>
          <li>Tensor margin_ranking_loss(const Tensor&amp; input1, const Tensor&amp; input2, const Tensor&amp; target, double margin, int64_t reduction) {</li>
          <li>  auto output =  (-target * (input1 - input2) + margin).clamp_min_(0);</li>
          <li>  return apply_loss_reduction(output, reduction);</li>
          <li>}</li>
          <li></li>
          <li>Tensor kl_div(const Tensor&amp; input, const Tensor&amp; target, int64_t reduction) {</li>
          <li>  auto zeros = at::zeros_like(target);</li>
          <li class="selected">  auto output_pos = target * (at::log(target) - input);</li>
          <li>  auto output = at::where(target &gt; 0, output_pos, zeros);</li>
          <li>  return apply_loss_reduction(output, reduction);</li>
          <li>}</li>
          <li></li>
          <li>Tensor kl_div_backward_cpu(const Tensor&amp; grad, const Tensor&amp; input, const Tensor&amp; target, int64_t reduction) {</li>
          <li>  auto grad_input = at::zeros_like(input);</li>
          <li>  auto grad_expand = grad.expand_as(input);</li>
          <li>  AT_DISPATCH_FLOATING_TYPES(input.scalar_type(), "kl_div_backward_cpu", [&amp;]() {</li>
          <li>    at::CPU_tensor_apply3&lt;scalar_t, scalar_t, scalar_t&gt;(</li>
          <li>        grad_input,</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Based on the source code, you should provide <code>log_probs</code> for the <code>target</code>.<br>
Notice that PyTorch use <code>kl_div</code> like this: <code>kl_div(b, a)</code> for <code>kl_div(a||b)</code>, so it means you need to use following code to get the same result as Tensorflow.</p>
<pre data-code-wrap="python"><code class="lang-python">preds_torch = torch.Tensor(preds)
labels_torch = torch.Tensor(labels)
out = F.kl_div(labels_torch.log(), preds_torch, reduction='sum')
print(out.item())  #0.2038460671901703
</code></pre>
<p>Also, it is equivalent to:</p>
<pre data-code-wrap="python"><code class="lang-python">out = (preds_torch * (preds_torch / labels_torch).log()).sum()
print(out.item())
</code></pre>
<p>In the end, I am really not sure about <code>distribution</code> package yet. I will check it out and let you know if you are interested.</p>
<p>Further reading:</p><aside class="quote quote-modified" data-post="2" data-topic="53024">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/kl-divergence-between-two-multivariate-gaussian/53024/2">KL-divergence between two multivariate gaussian</a> 
  </div>
  <blockquote>
    Hi, 
Yes, this is the correct approach. 
Just be aware that the input a must should contain log-probabilities and the target b should contain probability. 


<a href="https://pytorch.org/docs/stable/nn.functional.html?highlight=kl_div#kl-div" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/nn.functional.html?highlight=kl_div#kl-div</a> 
By the way, PyTorch use this approach: 
 <a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/0a58d819377f67d7c4735e76be5db794ad56f05d.png" data-download-href="https://discuss.pytorch.org/uploads/default/0a58d819377f67d7c4735e76be5db794ad56f05d" title="image">[image]</a> 
<a href="https://pytorch.org/docs/stable/distributions.html?highlight=kl_div#torch.distributions.kl.kl_divergence" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/distributions.html?highlight=kl_div#torch.distributions.kl.kl_divergence</a> 
Good luck 
Nik
  </blockquote>
</aside>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/7337">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/7337" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/7337" target="_blank" rel="noopener nofollow ugc">kldiv loss formula</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2018-05-07" data-time="13:35:28" data-timezone="UTC">01:35PM - 07 May 18 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2018-05-07" data-time="15:50:16" data-timezone="UTC">03:50PM - 07 May 18 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/skynbe" target="_blank" rel="noopener nofollow ugc">
          <img alt="skynbe" src="https://avatars.githubusercontent.com/u/3962617?v=4" class="onebox-avatar-inline" width="20" height="20">
          skynbe
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Does F.kl_div(a, b) mean KL(b||a), not KL(a||b) ?</p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Good luck<br>
Nik</p>

Nikronic: <p><a class="mention" href="/u/razvanc92">@razvanc92</a></p>
<p>I just found the solution using <code>distribution</code> package too.<br>
As I mentioned in the previous post, the <code>target</code> should be <code>log_probs</code>, so based on, we must have these:</p>
<pre><code class="lang-auto">preds_torch = torch.distributions.Categorical(probs=torch.from_numpy(preds))
labels_torch = torch.distributions.Categorical(logits=torch.from_numpy(np.log(labels)))
torch_res = kl_divergence(preds_torch, labels_torch)
</code></pre>
<p>Note that for <code>target(labels_torch)</code> we use <code>logits</code> not <code>probs</code> and also provide <code>log(labels)</code> rather than <code>labels</code> itself.</p>
<p>Good luck<br>
Nik</p>

razvanc92: <p>Could you also help me with the differences between tf/pytorch and numpy. It seems to be working fine when the input is 2d, but when the input has more than 2 dimensions it doesn’t. For example now I’m trying with a 4d array where the distributions are on the last axis. This is my implementation:</p>
<pre><code class="lang-auto"> np.mean(np.sum(preds * np.log(preds / labels), axis=-1))
</code></pre>
<p>Thanks in advance.</p>

Nikronic: <p><a class="mention" href="/u/razvanc92">@razvanc92</a>  Sorry for late reply, I was dealing with a bunch of problems.<br>
If I want to be frank with you, I could not get same output for random generated numbers using both <code>nn.kl_div</code> or formula itself. Can you state your last post as a separate question?<br>
And please mention me there too, so I can understand what is really happening there.<br>
Maybe other experienced could help us too.</p>