How can i calculate correct softmax gradient?

hao_xi: <p>How does the softmax gradient in the pytorch framework calculate the gradient of the input data? I followed this link <a href="https://community.deeplearning.ai/t/calculating-gradient-of-softmax-function/1897/3" rel="noopener nofollow ugc">https://community.deeplearning.ai/t/calculating-gradient-of-softmax-function/1897/3</a> to implement the gradient derivative function, why are the gradients of the derivatives all 0? How do I correctly pass the gradient when using the softmax layer?</p>
<p>import torch</p>
<p>def softmax_forward(x):<br>
e_x = torch.exp(x - torch.amax(x, dim=1, keepdim=True))<br>
softmax = e_x / torch.sum(e_x, dim=1, keepdim=True)<br>
return softmax</p>
<p>def softmax_backward(softmax, grad_output):<br>
shape = softmax.shape<br>
softmax = torch.reshape(softmax, (1, -1))<br>
grad_output = torch.reshape(grad_output, (1, -1))<br>
d_softmax = softmax * torch.eye(softmax.numel()) - softmax.T @ softmax<br>
grad = (grad_output @ d_softmax).reshape(shape)<br>
return grad</p>
<p>data = torch.tensor([1,2,3,4], dtype=torch.float32, requires_grad=True).view(1, 4)<br>
data.retain_grad()<br>
out = torch.softmax(data, dim=-1)<br>
out.retain_grad()</p>
<p>sim_out = softmax_forward(data)<br>
print(f"out_diff: {torch.abs(sim_out - out).max()}")</p>
<p>y = torch.sum(out, dim=-1)<br>
y.backward()</p>
<p>print(f"data_grad: {data.grad}“)<br>
sim_grad = softmax_backward(out, out.grad)<br>
print(f"grad_diff: {torch.abs(sim_grad - data.grad).max()}”)</p>
<blockquote>
<blockquote>
<p>out_diff: 0.0<br>
data_grad: tensor([[0., 0., 0., 0.]])<br>
grad_diff: 2.9802322387695312e-08</p>
</blockquote>
</blockquote>

KFrank: <p>Hi Hao!</p>
<aside class="quote no-group quote-modified" data-username="hao_xi" data-post="1" data-topic="212975" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hao_xi/48/73993_2.png" class="avatar"> hao_xi:</div>
<blockquote>
<p>why are the gradients of the derivatives all 0?<br>
…<br>
y = torch.sum(out, dim=-1)<br>
y.backward()</p>
</blockquote>
</aside>
<p>I haven’t looked at the details of your code, but <code>softmax()</code> has a property that<br>
will cause your particular gradients to be zero.  Namely, <code>softmax()</code> returns a<br>
set of probabilities that sum to one, which, being a constant, has zero gradient.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; torch.softmax (torch.tensor ([1., 2., 3., 4.]), dim  = -1)
tensor([0.0321, 0.0871, 0.2369, 0.6439])
&gt;&gt;&gt; torch.softmax (torch.tensor ([1., 2., 3., 4.]), dim  = -1).sum()
tensor(1.)
</code></pre>
<p>Instead of calling backward on the sum of <code>out</code>, you might try <code>out[0].backward()</code>.<br>
An individual component of the result of <code>softmax()</code> is not a constant, so you will<br>
get a non-trivial gradient.</p>
<p>Best.</p>
<p>K. Frank</p>

hao_xi: <aside class="quote no-group quote-modified" data-username="KFrank" data-post="2" data-topic="212975" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Hi Hao!</p>
<aside class="quote no-group quote-modified" data-username="hao_xi" data-post="1" data-topic="212975" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hao_xi/48/73993_2.png" class="avatar"> hao_xi:</div>
<blockquote>
<p>why are the gradients of the derivatives all 0?<br>
…<br>
y = torch.sum(out, dim=-1)<br>
y.backward()</p>
</blockquote>
</aside>
<p>I haven’t looked at the details of your code, but <code>softmax()</code> has a property that<br>
will cause your particular gradients to be zero.  Namely, <code>softmax()</code> returns a<br>
set of probabilities that sum to one, which, being a constant, has zero gradient.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; torch.softmax (torch.tensor ([1., 2., 3., 4.]), dim  = -1)
tensor([0.0321, 0.0871, 0.2369, 0.6439])
&gt;&gt;&gt; torch.softmax (torch.tensor ([1., 2., 3., 4.]), dim  = -1).sum()
tensor(1.)
</code></pre>
<p>Instead of calling backward on the sum of <code>out</code>, you might try <code>out[0].backward()</code>.<br>
An individual component of the result of <code>softmax()</code> is not a constant, so you will<br>
get a non-trivial gradient.</p>
<p>Best.</p>
<p>K. Frank</p>
</blockquote>
</aside>
<p>Thank you very much for your reply。<br>
Yes, what I want to describe is the same as what you said. After I call the backward function of the softmax function, I find that the gradient of the softmax input data obtained by using the softmax output data to differentiate is always 0. So I would like to ask, if softmax is an intermediate layer in a network, how is its corresponding gradient calculated? Is it only the maximum value to calculate the gradient or is there any other method? How to implement this part of the program in pytorch?</p>

KFrank: <p>Hi Hao!</p>
<aside class="quote no-group" data-username="hao_xi" data-post="4" data-topic="212975" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hao_xi/48/73993_2.png" class="avatar"> hao_xi:</div>
<blockquote>
<p>I find that the gradient of the softmax input data obtained by using the softmax output data to differentiate is always 0.</p>
</blockquote>
</aside>
<p>This isn’t true.</p>
<p>First note that applying <code>softmax()</code> to, say, a one-dimensional tensor returns a<br>
one-dimensional tensor.  You can’t compute a gradient of a tensor (of length greater<br>
than one), so you have to take the gradient of some scalar function of the tensor.<br>
If that scalar function <em>happens to be</em> <code>sum()</code>, then the result will be <code>1.0</code>, a constant,<br>
so the gradient of that constant will be zero.</p>
<p>However, if you use some other scalar function that doesn’t always return <code>1.0</code>, you<br>
will get a non-zero gradient, as you expect.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.5.1'
&gt;&gt;&gt; t = torch.arange (5.0, requires_grad = True)
&gt;&gt;&gt; (t.softmax (dim = 0)**2).sum().backward()
&gt;&gt;&gt; t.grad
tensor([-0.0106, -0.0277, -0.0658, -0.1097,  0.2139])
</code></pre>
<p>(Again, the point is that <code>softmax()</code> has the special property that the result of<br>
<code>softmax()</code> sums to one.  It’s the fact that you are summing the result of <code>softmax()</code>,<br>
instead of doing something else with it, that leads to the zero gradient.)</p>
<p>Best.</p>
<p>K. Frank</p>

hao_xi: <p>Thank you very much for your answer. After your suggestion, I also realized that it was because I called the sum function every time, which caused their gradient values ​​to be 1, and thus the gradient of the input tensor to be 0. I just reproduced your code and the gradient is indeed normal. Thank you very much, and I wish you a happy life~</p>