Torch.nn.functional.kl_div result different from TF/Scipy implementation

username: sc21
cooked: <p>I thought torch.nn.functional.kl_div should compute KL divergence in <a href="https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence" class="inline-onebox" rel="noopener nofollow ugc">Kullback–Leibler divergence - Wikipedia</a> (the same as scipy.stats.entropy and tf.keras.losses.KLDivergence), but I cannot get the same results from a simple example. Does anyone know why?</p>
<pre><code class="lang-auto"> from scipy.stats import entropy
entropy([0.5,0.5],[0.7,0.3])
0.08717669357238891
kl_tf=tf.keras.losses.KLDivergence()
y_true=[0.5,0.5]
y_pred=[0.7,0.3]
kl_tf(y_true,y_pred)
&lt;tf.Tensor: shape=(), dtype=float32, numpy=0.08717668&gt;
</code></pre>
<p>Those two above gives the same kl divergence value. But when I tried to use torch.nn.functional.kl_div, the result is not the same.</p>
<pre><code class="lang-auto">y_true=[0.5,0.5]
y_pred=[0.7,0.3]
torch.nn.functional.kl_div(torch.FloatTensor(y_true),torch.FloatTensor(y_pred),reduction='sum')
tensor(-1.1109)
</code></pre>

username: tom
cooked: <p>So if we look at the <a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.kl_div" rel="noopener nofollow ugc">documentation for <code>kl_div</code></a> we are instructed to See <a href="https://pytorch.org/docs/stable/generated/torch.nn.KLDivLoss.html#torch.nn.KLDivLoss" rel="noopener nofollow ugc"><code>KLDivLoss</code></a> for details.</p>
<p>There it says:</p>
<blockquote>
<p>the input given is expected to contain <em>log-probabilities</em> and is not restricted to a 2D Tensor. The targets are interpreted as <em>probabilities</em> by default, but could be considered as <em>log-probabilities</em> with <code>log_target</code> set to <code>True</code>.</p>
</blockquote>
<p>In other words, the first argument should be log probs.</p>
<p>Personally, I’d not use FloatTensor, it’s been superseded as the <a href="https://pytorch.org/blog/pytorch-0_4_0-migration-guide/#creating-tensors" rel="noopener nofollow ugc">preferred way to create tensors</a> longer than it had been the way to create them.</p>
<p>Best regards</p>
<p>Thomas</p>

username: sc21
cooked: <p>Thanks – that makes sense. I must have missed the log probability part, although it seems weird not giving an option to specify probabilities directly on input.</p>
<p>And thanks for pointing out the FloatTensor issue. I guess torch.tensor() is the recommended way now.</p>

username: tom
cooked: <p>So the definition that it is log probs has historically grown, but as a rule of thumb, probabilities very close to 1 are difficult to work with (try evaluating <code>1-1e-20</code>, and it’s a lot less for float vs. double). Now if you know that your probabilities are close to 1 but not close to 0, you can avoid this by using <code>1-p</code> consistently, but it seems that using log-probs as the default representation of probabilities is a good idea in general.</p>
<p>Now this of distinctly practical relevance and crops up quite frequently</p>
<ul>
<li>it is much more sane to use log_probs as neural network output if you want to train with negative log likelihood aka cross entropy,</li>
<li>When computing p-values that can be very small, it is a bad idea to compute 1-p first, see e.g. <a href="https://github.com/scipy/scipy/issues/12999" class="inline-onebox" rel="noopener nofollow ugc">Bug in scipy.stats.ks_2samp for two-sided auto and exact modes on a few thousand samples (when n1!=n2) · Issue #12999 · scipy/scipy · GitHub</a> , (where my solution is to compute p directly instead of going via 1-p).</li>
<li>Scipy has separate implementations for the lower (=usual) and <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.gammaincc.html#scipy.special.gammaincc" rel="noopener nofollow ugc">upper incomplete gamma function (1-CDF for the gamma distribution)</a> for stability reasons.</li>
</ul>
<p>So to me, it makes a lot of sense to have log probs whenever actually doing something with it, but I would agree that it might be better to mention it explicitly also in the kl_div documentation (personally, I try to name my parameters <code>log_probs</code> or so, but I guess that’s not trivial to change).</p>
<p>Best regards</p>
<p>Thomas</p>