Why the gradient of feature passing into CrossEntroyLoss function is different from the theoretical value？

username: CalmLife
cooked: <p>Here is my code, it’s similar to official example.</p>
<pre><code class="lang-auto">import math
import torch
import torch.nn as nn
import torch.nn.functional as F


loss = nn.CrossEntropyLoss()
feature = torch.ones(3, 5, requires_grad=True)
target = torch.empty(3, dtype=torch.long).random_(5)

output_sm = F.log_softmax(feature, dim=1)
output_nll = F.nll_loss(output_sm, target)
output = output_nll
output.backward()

print("Input:\n", feature)
print("Target:\n", target)
print("Gradient in feature:\n", feature.grad)
</code></pre>
<p>The picture below is my results.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/c/acfe583093eda01b9b4db08ef16bc5128fab4aba.png" alt="result" data-base62-sha1="oGn3rfBL64NVCrnZlEyzkHAcar8" width="594" height="252"></p>
<p>The result of theoretical deduction is as follow:</p>
<p>Partial derivative of L to z equals p - y</p>
<p>L represents loss function<br>
z is the input feature<br>
p is the output of softmax<br>
y is the target<br>
Regarding the detailed derivation of this formula, I am not here because it is easy to get.</p>
<p>What confused me is the experimental results do not match the theoretical values?<br>
Can you give me some suggestion? Thank you!</p>

username: albanD
cooked: <p>What is the value you expect?<br>
Given there is a log then softmax then nll then averaging. Plus the backward computations of each. I wouldn’t say it’s a trivial computation to do.</p>

username: CalmLife
cooked: <p>Thanks for your answer!<br>
This is the formula.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d79df21ab44518248c3d76c22242dd3a92f535fc.png" alt="derivate" data-base62-sha1="uLr3iQIw9EjrcURDmnQB6TS46ao" width="295" height="89"></p>
<p>p = [<br>
&nbsp; &nbsp; &nbsp;0.2 0.2 &nbsp;0.2 0.2 -0.8<br>
-0.8 0.2 &nbsp;0.2 0.2 0.2<br>
&nbsp;0.2 0.2 -0.8 0.2 0.2<br>
]</p>
<p>y = [4, 0, 2]</p>
<p>The values I expected  as follows:<br>
［<br>
&nbsp;0.2 0.2 &nbsp;0.2 0.2 -0.8<br>
-0.8 0.2 &nbsp;0.2 0.2 0.2<br>
&nbsp;0.2 0.2 -0.8 0.2 0.2<br>
]</p>

username: albanD
cooked: <p>Could you explain how you get <code>dL/dz = p - y</code> please?</p>

username: CalmLife
cooked: <p>Thank you! I have solved this problem. I did not divide by batches when caculating.<br>
I will explain how I get  <code>dL/dz = p - y</code> later.</p>

username: CalmLife
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/2/5246b7d9c9bccdeb0187a92e561178315395639b.png" data-download-href="https://discuss.pytorch.org/uploads/default/5246b7d9c9bccdeb0187a92e561178315395639b" title="%E6%8D%95%E8%8E%B7"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/5246b7d9c9bccdeb0187a92e561178315395639b_2_345x500.png" alt="%E6%8D%95%E8%8E%B7" data-base62-sha1="bJQFgnmOzgMDQ7WbGpzREMfF2yL" width="345" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/5246b7d9c9bccdeb0187a92e561178315395639b_2_345x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/5246b7d9c9bccdeb0187a92e561178315395639b_2_517x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/2/5246b7d9c9bccdeb0187a92e561178315395639b.png 2x" data-dominant-color="FCFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">%E6%8D%95%E8%8E%B7</span><span class="informations">540×781 13.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>