Can not get gradient of the kl_div

username: Ereebay
cooked: <p>Hi, I have some problems of getting the gradient when I used the kl_div function.</p>
<p>I tried two versions of the kl_div function, one is the official version: F.kl_div, and the other is implemented by myself. The results of the official kl_div and my kl_div are same but the gradient of the F.kl_div is always 0.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F


def kl_div(x, y):
    _kl = torch.sum(y * (torch.log(y) - x), -1)
    return torch.mean(_kl)


logits = torch.Tensor([[1, 2, 3, 4]])

logits.requires_grad = True

label = torch.Tensor([[0.1, 0.1, 0.7, 0.1]])
label2 = torch.Tensor([[0.1, 0.1, 0.7, 0.1]])
label.requires_grad = True
y_qry = torch.LongTensor([1])

logsoftlogits = F.log_softmax(logits, dim=-1)

case = 0
if case == 0:
    loss = kl_div(logsoftlogits, label)
    #print(loss)
else:
    loss = F.kl_div(logsoftlogits, label, reduction='sum') / logsoftlogits.size()[0]
    #print(loss)
grad = torch.autograd.grad(outputs=loss, inputs=logits, create_graph=True)
logits_q = logits - grad[0]

loss_q = F.cross_entropy(logits_q, y_qry)

labelgrad = torch.autograd.grad(outputs=loss_q, inputs=label, grad_outputs=torch.ones(loss_q.size()))
print(labelgrad)
</code></pre>
<p>the official kl_div result:</p>
<blockquote>
<p>(tensor([[0., 0., 0., 0.]]),)</p>
</blockquote>
<p>my implementation reusult:</p>
<blockquote>
<p>(tensor([[-0.2615, -1.1997,  0.1304,  0.1274]]),)</p>
</blockquote>

username: KFrank
cooked: <p>Hi Ereebay!</p>
<aside class="quote no-group" data-username="Ereebay" data-post="1" data-topic="71279" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ereebay/48/21063_2.png" class="avatar"> Ereebay:</div>
<blockquote>
<p>Hi, I have some problems of getting the gradient when I used the kl_div function.</p>
<p>I tried two versions of the kl_div function, one is the official version: F.kl_div, and the other is implemented by myself. The results of the official kl_div and my kl_div are same but the gradient of the F.kl_div is always 0.</p>
</blockquote>
</aside>
<p>I don’t follow fully what you are doing with two layers of losses and grads,<br>
and I will comment in terms of the older 0.3.0 version of pytorch.</p>
<p>I believe what you are seeing is that <code>F.kl_div()</code> doesn’t track gradients<br>
with respect to its second argument (your <code>label</code>).  (I think this is a<br>
“feature” in the interest of efficiency.)  The version of <code>kl_div()</code> that you<br>
wrote isn’t “smart” enough to <em>not</em> compute the gradient with respect<br>
to its second argument.</p>
<p>Please see this related thread:</p>
<aside class="quote quote-modified" data-post="1" data-topic="3693">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/b5e925/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/nn-criterions-dont-compute-the-gradient-w-r-t-targets/3693">Nn criterions don't compute the gradient w.r.t. targets</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    I am implementing a really basic autoencoder in pytorch. 
import torch
import torch.nn as nn
import torch.nn.parallel
import torch.backends.cudnn as cudnn
import torch.optim as optim
from torch.autograd import Variable

BATCH_SIZE = 16

criterion_mse = nn.MSELoss().cuda()
x = Variable(torch.FloatTensor( BATCH_SIZE , 10  )  ).cuda()
l = nn.Linear( 10 , 10 ).cuda() 
y = l(x)
loss = criterion_mse( x , y )

But this code gives the following error. 
AssertionError                            Traceback…
  </blockquote>
</aside>

<p>Here is a simple pytorch 0.3.0 script that illustrates this for <code>F.kl_div()</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__

torch.manual_seed (2020)

# make some log-probabilities and probabilities
logprobs = torch.nn.functional.log_softmax (torch.autograd.Variable (torch.randn (2, 5)), dim = 1).data
probs1 = torch.nn.functional.softmax (torch.autograd.Variable (torch.randn (2, 5)), dim = 1).data
probs2 = probs1.clone()

# make leaf-variable Variables
# input values are log-probabilities
# input  = torch.autograd.Variable (torch.randn (2, 5), is_leaf = True, requires_grad = True), dim = 1)
input = torch.autograd.Variable (logprobs, requires_grad = True)
# target values are probabilities
targ_grad = torch.autograd.Variable (probs1, requires_grad = True)
targ_nograd = torch.autograd.Variable (probs2, requires_grad = False)

loss = torch.nn.functional.kl_div (input, targ_grad)
loss
loss.backward()
input.grad
targ_grad.grad

loss = torch.nn.functional.kl_div (input, targ_nograd)
loss
loss.backward()
input.grad
targ_nograd.grad
</code></pre>
<p>And here is the output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'0.3.0b0+591e73e'
&gt;&gt;&gt;
&gt;&gt;&gt; torch.manual_seed (2020)
&lt;torch._C.Generator object at 0x000002904D7C6630&gt;
&gt;&gt;&gt;
&gt;&gt;&gt; # make some log-probabilities and probabilities
... logprobs = torch.nn.functional.log_softmax (torch.autograd.Variable (torch.randn (2, 5)), dim = 1).data
&gt;&gt;&gt; probs1 = torch.nn.functional.softmax (torch.autograd.Variable (torch.randn (2, 5)), dim = 1).data
&gt;&gt;&gt; probs2 = probs1.clone()
&gt;&gt;&gt;
&gt;&gt;&gt; # make leaf-variable Variables
... # input values are log-probabilities
... # input  = torch.autograd.Variable (torch.randn (2, 5), is_leaf = True, requires_grad = True), dim = 1)
... input = torch.autograd.Variable (logprobs, requires_grad = True)
&gt;&gt;&gt; # target values are probabilities
... targ_grad = torch.autograd.Variable (probs1, requires_grad = True)
&gt;&gt;&gt; targ_nograd = torch.autograd.Variable (probs2, requires_grad = False)
&gt;&gt;&gt;
&gt;&gt;&gt; loss = torch.nn.functional.kl_div (input, targ_grad)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: the derivative for 'target' is not implemented
&gt;&gt;&gt; loss
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
NameError: name 'loss' is not defined
&gt;&gt;&gt; loss.backward()
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
NameError: name 'loss' is not defined
&gt;&gt;&gt; input.grad
&gt;&gt;&gt; targ_grad.grad
&gt;&gt;&gt;
&gt;&gt;&gt; loss = torch.nn.functional.kl_div (input, targ_nograd)
&gt;&gt;&gt; loss
Variable containing:
 0.2735
[torch.FloatTensor of size 1]

&gt;&gt;&gt; loss.backward()
&gt;&gt;&gt; input.grad
Variable containing:
1.00000e-02 *
 -0.6291 -1.5081 -5.4165 -1.8988 -0.5475
 -0.8674 -0.3891 -0.4075 -0.6592 -7.6768
[torch.FloatTensor of size 2x5]

&gt;&gt;&gt; targ_nograd.grad
&gt;&gt;&gt;
</code></pre>
<p>The pytorch 0.3.0 version of <code>F.kl_div()</code> simply refuses to accept a second<br>
argument that has <code>requires_grad = True</code>, and throws an error.  If the<br>
second argument has <code>requires_grad = False</code>, <code>F.kl_div()</code> works, and<br>
you get a gradient for its first argument, but, of course, not its second.</p>
<p>Does this look like the root cause of what you are seeing?</p>
<p>Best.</p>
<p>K. Frank</p>

username: Ereebay
cooked: <p>Thx for your reply, Frank.  Sorry for my poor English.</p>
<p>First, let me explain what I’m doing in this two layer, I have a logits which is a result of a classification, and the fake or synthetic label is the target, and I used to kl_dive to compute the divergence between them. Than I used to gradient to update the logits, than I used the new logits to compute the cross entropy between the new logits and the right label(y_qry). After that, I used the new gradient of the synthetic label to update the synthetic label.</p>
<blockquote>
<p>I believe what you are seeing is that  <code>F.kl_div()</code>  doesn’t track gradients<br>
with respect to its second argument (your  <code>label</code> ). (I think this is a<br>
“feature” in the interest of efficiency.)</p>
</blockquote>
<p>Second, I used the auto.grad function to directly compute the gradient of label, it can track gradients with respect to its arguments normally. But I compute the gradient of label in the cross entropy loss, it can not track. Is this possible that it can not track an tensor in another loss function ?</p>

username: KFrank
cooked: <p>Hello Ereebay!</p>
<aside class="quote no-group" data-username="Ereebay" data-post="3" data-topic="71279" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ereebay/48/21063_2.png" class="avatar"> Ereebay:</div>
<blockquote>
<p>Second, I used the auto.grad function to directly compute the gradient of label, it can track gradients with respect to its arguments normally.</p>
</blockquote>
</aside>
<p>As shown in my post above, in pytorch 0.3.0, <code>F.kl_div()</code> simply<br>
does <em>not</em> track the gradient of its second argument (<code>label</code>).</p>
<p>I downgraded, for whatever reason, to pytorch 0.3.0, so I can’t test<br>
this on later versions.  Could you tell us what version of pytorch you<br>
are using?</p>
<p>(Perhaps someone with pytorch 1.x could run my test on a more<br>
up-to-date version.)</p>
<p>Could you check in a simple script whether or not you can track gradients<br>
with respect to both arguments of <code>F.kl_dv()</code> in your version of pytorch?<br>
(Don’t use your loss-within-a loss code – just make a single call to<br>
<code>F.kl_div()</code> for testing purposes.)</p>
<aside class="quote no-group">
<blockquote>
<p>But I compute the gradient of label in the cross entropy loss, it can not track. Is this possible that it can not track an tensor in another loss function ?</p>
</blockquote>
</aside>
<p>I don’t know for a fact, but I would not expect this to be the case.  As far<br>
as I understand it, loss functions are just regular (autograd-supporting)<br>
tensor functions, so I would expect gradient-tracking to work as well<br>
for your loss-within-a-loss use case as it does for the individual loss<br>
functions separately.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Ereebay
cooked: <p>Hi,Frank, I used pytorch 1.2.0. And I checked the both arguments of F.kl_div() in my version of pytorch, they can be tracked normally. I also met the same problem in the L1 loss function.</p>

username: KFrank
cooked: <p>Hello Ereebay!</p>
<aside class="quote no-group" data-username="Ereebay" data-post="5" data-topic="71279" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ereebay/48/21063_2.png" class="avatar"> Ereebay:</div>
<blockquote>
<p>I used pytorch 1.2.0. And I checked the both arguments of F.kl_div() in my version of pytorch, they can be tracked normally.</p>
</blockquote>
</aside>
<p>I can’t test this myself on pytorch 1.x, but looking at the <a href="https://github.com/pytorch/pytorch/blob/7cc029cb75c292e93d168e117e46a681ace02e79/aten/src/ATen/native/Loss.cpp#L69-L94" rel="noopener nofollow ugc">code</a> for <code>kl_div()</code><br>
(specifically line 86, <code>grad_input_val = -target_val * grad_val;</code>),<br>
I only see the gradient being computed for the first (<code>input</code>, your <code>logits</code>)<br>
argument, and not for the second (<code>target</code>, your <code>label</code>).</p>
<p>(I assume that this code is for the current “stable” pytorch version, so<br>
newer than your 1.2.0.)</p>
<p>Could you post your (short, complete, runnable) test script, together<br>
with its output?  Maybe somebody with a non-archaic copy of pytorch<br>
could help look into this.</p>
<p>Best.</p>
<p>K. Frank</p>