What's wrong about the grad of softmax when I use just some of the inputs to do softmax?

MasLiang: <p>I use softmax and then I use log to mimic cross entropy when I try to implement knowledge distillation.</p>
<pre><code class="lang-auto">soft_old_output = soft_target.clone()/T
soft_new_output = outputs.clone()/T
soft_new_output.retain_grad()
outputs_S = F.softmax(soft_new_output,dim=1)
outputs_S.retain_grad()
outputs_T = F.softmax(soft_old_output,dim=1)
loss2 = outputs_T.mul(-1*torch.log(outputs_S))
loss2 = loss2.sum(1)
loss2 = loss2.mean()
loss = loss2
loss.backward(retain_graph=True)
print(soft_new_output.grad)
</code></pre>
<p>At first the <code>soft_old_output</code> is the same as <code>soft_new_output</code> . So the <code>grad</code> of <code>soft_new_output</code> is all zero. It’s right.<br>
But when I try to use some of the value to do is, like that:</p>
<pre><code class="lang-auto">soft_old_output = soft_target.[:,:2]clone()/T
soft_new_output = outputs[:,:2].clone()/T
soft_new_output.retain_grad()
outputs_S = F.softmax(soft_new_output,dim=1)
outputs_S.retain_grad()
outputs_T = F.softmax(soft_old_output,dim=1)
loss2 = outputs_T.mul(-1*torch.log(outputs_S))
loss2 = loss2.sum(1)
loss2 = loss2.mean()
loss = loss2
loss.backward(retain_graph=True)
print(soft_new_output.grad)
</code></pre>
<p>the grad of soft_new_output will be</p>
<pre><code class="lang-auto">tensor([[-5.8213e-08, -1.3913e-09]])
</code></pre>
<p>Why aren’t they zero?</p>

ptrblck: <p>The small absolute error is most likely created by the limited numerical precision using <code>flaot32</code> and you should get a smaller error using <code>float64</code>.<br>
Also note, that using <code>torch.log(torch.softmax(...))</code> is numerically less stable than <code>F.log_softmax</code>, as the latter applies the <code>log-sum-exp</code> trick to increase the stability.</p>

MasLiang: <p>Thank you very much~<br>
And I have another question now, when I try to find the problem myself, I try to track gradient changes. But I don’t know where to find the grad_fn of each variable. So where can I find them? I mean, I want to know how they work so that I can analyze which part leading to this problem, but I can’t find them.</p>

ptrblck: <p>The <code>grad_fn</code> is an attribute of a tensor, which was created by a differentiable operation and which Autograd has tracked.<br>
Here is a small example:</p>
<pre><code class="lang-python">x = torch.randn(1)
w = torch.randn(1, requires_grad=True)
y = x * w
print(y.grad_fn)
&gt; &lt;MulBackward0 object at ...&gt;
</code></pre>

MasLiang: <p>I know that but what does MulBackward0 do? It is easy to understand it but what about other grad_fn? Do they just work like the mathematical formula says? Or do they have any trick to work? So I want to find the code of them to have a look.</p>

ptrblck: <aside class="quote no-group" data-username="MasLiang" data-post="5" data-topic="109266">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/masliang/48/33566_2.png" class="avatar"> MasLiang:</div>
<blockquote>
<p>Do they just work like the mathematical formula says?</p>
</blockquote>
</aside>
<p>Yes, they should be implemented using the right mathematical derivatives.<br>
You can find the implementations in <a href="https://github.com/pytorch/pytorch/blob/57fb2c0fcce9f5d77816012a60f71f8c2d18d670/tools/autograd/derivatives.yaml">derivatives.yaml</a>, where these methods are either directly defined or are pointing to the name of the implementation.</p>

MasLiang: <p>That’s just what I need. Thank you very much. <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"></p>