KL divergence loss pytorch implementation

Krishnan_Jothi: <p>According to the theory kl divergence is the difference between cross entropy (of inputs and targets) and the entropy (of targets).</p>
<p>I compared the kl div loss implementation in pytorch against the custom implementation based on the above theory. But the results are not the same, I am not sure why there is a difference.</p>
<p>import torch</p>
<p>m = nn.Softmax(dim=1)<br>
i_tensor_before_softmax = torch.Tensor([0.1, 0.2, 0.4, 0.3])<br>
i_tensor = m(p_tensor.view(-1,4))<br>
o_tensor_before_softmax = torch.Tensor([0.7, 0.1, 0.1, 0.1])<br>
o_tensor = m(q_tensor.view(-1,4))</p>
<p>import torch.nn.functional as F<br>
kl_loss = nn.KLDivLoss(reduction=“batchmean”)<br>
loss = torch.nn.CrossEntropyLoss()<br>
kl_output = kl_loss(input=F.log_softmax(i_tensor_before_softmax, dim=-1), target=o_tensor)<br>
cross_ent = loss(input=i_tensor_before_softmax.view(-1,4), target=o_tensor)<br>
ent = loss(input=o_tensor_before_softmax.view(-1,4), target=o_tensor)<br>
kl_output_using_ce = cross_ent - ent<br>
print(kl_output, kl_output_using_ce)</p>
<p>output: (tensor(0.0179), tensor(0.0716))</p>
<p>Help me out in identifying the issue <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"><br>
Thanks,<br>
Krishnan</p>

KFrank: <p>Hi Krishnan!</p>
<p>The short story is that your results differ by a factor of four because<br>
<code>KLDivLoss</code> takes the mean over the class dimension, while<br>
<code>CrossEntropyLoss</code> sums.</p>
<aside class="quote no-group" data-username="Krishnan_Jothi" data-post="1" data-topic="166206" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/krishnan_jothi/48/54923_2.png" class="avatar"> Krishnan_Jothi:</div>
<blockquote>
<p>According to the theory kl divergence is the difference between cross entropy (of inputs and targets) and the entropy (of targets).</p>
</blockquote>
</aside>
<p>This is correct.</p>
<aside class="quote no-group">
<blockquote>
<p>I compared the kl div loss implementation in pytorch against the custom implementation based on the above theory. But the results are not the same,</p>
</blockquote>
</aside>
<p>For what are presumably historical reasons (<code>CrossEntropyLoss</code> did not<br>
used to support probabilistic targets.), <code>KLDivLoss</code> computes what it calls<br>
the “pointwise” KL-divergence, while pytorch does not use a notion of<br>
“pointwise” cross entropy.</p>
<p>Let’s call the first dimension of your tensors the “batch” dimension and<br>
the second the “class” dimension.  So in your example your tensors have<br>
shape <code>[nBatch = 1, nClass = 4]</code>.</p>
<p><code>KLDivLoss</code>’s default <code>reduction</code> of <code>mean</code> takes the mean over all<br>
dimensions of the pointwise-KL-divergence tensor, so it takes the mean<br>
over the <code>nClass</code> dimension, dividing by four (and then takes the trivial<br>
mean over the <code>nBatch = 1</code> dimension).</p>
<p><code>CrossEntropyLoss</code>, by definition, sums over the <code>nClass</code> dimension, not<br>
dividing by four (and then, with its default <code>reduction = mean</code>, takes the<br>
trivial mean over the <code>nBatch</code> dimension).</p>
<p>Therefore your two results differ by a factor of four.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.13.0
&gt;&gt;&gt;
&gt;&gt;&gt; m = torch.nn.Softmax (dim=1)
&gt;&gt;&gt; i_tensor_before_softmax = torch.Tensor ([0.1, 0.2, 0.4, 0.3])
&gt;&gt;&gt; i_tensor = m (i_tensor_before_softmax.view (-1,4))
&gt;&gt;&gt; o_tensor_before_softmax = torch.Tensor ([0.7, 0.1, 0.1, 0.1])
&gt;&gt;&gt; o_tensor = m (o_tensor_before_softmax.view (-1,4))
&gt;&gt;&gt;
&gt;&gt;&gt; import torch.nn.functional as F
&gt;&gt;&gt; kl_loss = torch.nn.KLDivLoss (reduction = 'batchmean')
&gt;&gt;&gt; loss = torch.nn.CrossEntropyLoss()
&gt;&gt;&gt; kl_output = kl_loss (input = F.log_softmax (i_tensor_before_softmax, dim=-1), target = o_tensor)
&gt;&gt;&gt; cross_ent = loss (input = i_tensor_before_softmax.view (-1,4), target = o_tensor)
&gt;&gt;&gt; ent = loss (input=o_tensor_before_softmax.view (-1,4), target = o_tensor)
&gt;&gt;&gt; kl_output_using_ce = cross_ent - ent
&gt;&gt;&gt; print (kl_output, kl_output_using_ce)
tensor(0.0179) tensor(0.0716)
&gt;&gt;&gt;
&gt;&gt;&gt; kl_loss = torch.nn.KLDivLoss (reduction = 'none')
&gt;&gt;&gt; loss = torch.nn.CrossEntropyLoss (reduction = 'none')
&gt;&gt;&gt; kl_output_unr = kl_loss (input = F.log_softmax (i_tensor_before_softmax, dim=-1), target = o_tensor)
&gt;&gt;&gt; cross_ent_unr = loss (input = i_tensor_before_softmax.view (-1,4), target = o_tensor)
&gt;&gt;&gt; ent_unr = loss (input=o_tensor_before_softmax.view (-1,4), target = o_tensor)
&gt;&gt;&gt; kl_output_using_ce_unr = cross_ent_unr - ent_unr
&gt;&gt;&gt; print (kl_output_unr)
tensor([[ 0.2151, -0.0271, -0.0686, -0.0478]])
&gt;&gt;&gt; print (kl_output_using_ce_unr)
tensor([0.0716])
&gt;&gt;&gt; print (kl_output_unr.mean(), kl_output_unr.sum(), kl_output_using_ce_unr)
tensor(0.0179) tensor(0.0716) tensor([0.0716])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>