Question of usage of kl_divergence

username: Rizhao_Cai
cooked: <p>When I want to use kl divergence, I find there are some different and strange use cases.<br>
The formulation of KL divergence is<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5db22d149fd5049dcdca267c5e016476a78f37bb.png" alt="image" data-base62-sha1="dmS90rwolLvcKsZRuEfw2ZRJhMf" width="406" height="108"><br>
and the P  should be the target distribution, and Q is the input distribution.</p>
<p>According to the API doc,</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4d95f44b35b5e6619e9f969a0fa6dbef0ca651bf.png" data-download-href="https://discuss.pytorch.org/uploads/default/4d95f44b35b5e6619e9f969a0fa6dbef0ca651bf" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d95f44b35b5e6619e9f969a0fa6dbef0ca651bf_2_479x500.png" alt="image" data-base62-sha1="b4m2wJ5RjdHV7w9uN1XzM6HqKyH" width="479" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d95f44b35b5e6619e9f969a0fa6dbef0ca651bf_2_479x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d95f44b35b5e6619e9f969a0fa6dbef0ca651bf_2_718x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d95f44b35b5e6619e9f969a0fa6dbef0ca651bf_2_958x1000.png 2x" data-dominant-color="F2F4F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1049×1093 92.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I assume the first args `input’ should be Q, and the second args target should be P.</p>
<p>However, I asked GPT, it gave me the answer below<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/4/54646ae4a96682501dac35ad9708fe47fdd036a1.png" data-download-href="https://discuss.pytorch.org/uploads/default/54646ae4a96682501dac35ad9708fe47fdd036a1" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/4/54646ae4a96682501dac35ad9708fe47fdd036a1.png" alt="image" data-base62-sha1="c2zfzlgdcbCabJNCGqT3JRd4VwJ" width="690" height="378" data-dominant-color="676666"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1027×563 25.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It seems reversed. I also see some implementations that used  the target distribution（P）<br>
as the first args. This confuses me.</p>
<p>Why does the first args need to be used with the .log() function? If this is necessary, why don’t implement the .log inside the  function `kl_div’?<br>
Whether the first  or second  args should  be the true target distribution (P)?<br>
Moreover, I also see there is an argument named log_target. I still do not understand why we need this argument.</p>
<p>Appreciate any reply!</p>

username: zhen0qiang
cooked: <p>To avoid underflow issues when computing this quantity, this loss expects the argument ‘input’ in the log_space. The argument ‘target’ may also be provided in the log_space if log_target=True</p>
<p>if not log_target:<br>
loss = target * (target.log() - input)<br>
else:<br>
loss = target.exp() * (target_input)</p>

username: Rizhao_Cai
cooked: <p>Hi Zhen Qiang,</p>
<p>Thanks for your reply. But still confused.</p>
<p>When P is the true distribution, and Q is the predicted distribution,<br>
may I say,<br>
it should be <code>F.kl_div(Q, P)</code>, because Q is input, and P is target.<br>
But usually, to avoid the underflow issue, <code>F.kl_div(P.log(), Q)</code> is often used.</p>
<p>Is my understanding correct?</p>
<p>I</p>

username: zhen0qiang
cooked: <p>Yes, there is no problem with your understanding. See F.kl_div and torch.nn.KLDivLoss for more details.</p>