Understanding of gradient values after backward pass

two_Two: <p>Hi,I realize that I have no knowledge of how gradient values we get after calling backward.<br>
for example :</p>
<pre><code class="lang-auto">x = torch.randn((4,5),requires_grad = True)
z = x.mean()
z.backward()
#print(x.grad)
</code></pre>
<p>Assume I have the values of x</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x
tensor([[-0.3571,  0.1481,  0.1713, -1.2597, -0.7667],
        [-0.1553, -0.9620,  0.0103,  3.3494,  0.2220],
        [ 2.1131, -0.2404,  0.4820,  0.3816,  1.9752],
        [ 1.7232, -0.5064, -0.8151,  0.3720,  0.1470]], requires_grad=True)
&gt;&gt;&gt; 
</code></pre>
<p>And the result of x.grad returns</p>
<pre><code class="lang-auto">tensor([[1.1500, 1.1500, 1.1500, 1.1500, 1.1500],
        [1.1500, 1.1500, 1.1500, 1.1500, 1.1500],
        [1.1500, 1.1500, 1.1500, 1.1500, 1.1500],
        [1.1500, 1.1500, 1.1500, 1.1500, 1.1500]])

</code></pre>
<p>And if I change  <code>z = x.mean()</code>  to <code>z = x.sum()</code>  , x.grad becomes</p>
<pre><code class="lang-auto">tensor([[2.1500, 2.1500, 2.1500, 2.1500, 2.1500],
                [2.1500, 2.1500, 2.1500, 2.1500, 2.1500],
                [2.1500, 2.1500, 2.1500, 2.1500, 2.1500],
                [2.1500, 2.1500, 2.1500, 2.1500, 2.1500]])
</code></pre>
<p>I would like to know how do we compute and get the values both 1.1500 and 2.1500</p>
<p>Thanks in advance !</p>

KFrank: <p>Hi Two!</p>
<aside class="quote no-group quote-modified" data-username="two_Two" data-post="1" data-topic="90520" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/two_two/48/11554_2.png" class="avatar"> two_Two:</div>
<blockquote>
<pre><code class="lang-auto">x = torch.randn((4,5),requires_grad = True)
z = x.mean()
z.backward()
#print(x.grad)
</code></pre>
<p>…<br>
And the result of x.grad returns</p>
<pre><code class="lang-auto">tensor([[1.1500, 1.1500, 1.1500, 1.1500, 1.1500],
        [1.1500, 1.1500, 1.1500, 1.1500, 1.1500],
        [1.1500, 1.1500, 1.1500, 1.1500, 1.1500],
        [1.1500, 1.1500, 1.1500, 1.1500, 1.1500]])
</code></pre>
<p>…</p>
<p>I would like to know how do we compute and get the values both 1.1500 and 2.1500</p>
</blockquote>
</aside>
<p>The gradients you report are obviously wrong.  I cannot reproduce this<br>
with a pytorch-version-0.3.0 test script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__

torch.manual_seed (2020)

w1 = torch.randn (4,5)
w2 = w1.clone()

x1 = torch.autograd.Variable (w1, requires_grad = True)
x1
z1 = x1.mean()
z1.backward()
x1.grad

x2 = torch.autograd.Variable (w2, requires_grad = True)
x2
z2 = x2.sum()
z2.backward()
x2.grad
</code></pre>
<p>Here is the output showing the correct gradients:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'0.3.0b0+591e73e'
&gt;&gt;&gt;
&gt;&gt;&gt; torch.manual_seed (2020)
&lt;torch._C.Generator object at 0x000001E2AF8B6630&gt;
&gt;&gt;&gt;
&gt;&gt;&gt; w1 = torch.randn (4,5)
&gt;&gt;&gt; w2 = w1.clone()
&gt;&gt;&gt;
&gt;&gt;&gt; x1 = torch.autograd.Variable (w1, requires_grad = True)
&gt;&gt;&gt; x1
Variable containing:
 1.2372 -0.9604  1.5415 -0.4079  0.8806
 0.0529  0.0751  0.4777 -0.6759 -2.1489
-1.1463 -0.2720  1.0066 -0.0416 -1.2853
-0.4948 -1.2964 -1.2502 -0.7693  1.6856
[torch.FloatTensor of size 4x5]

&gt;&gt;&gt; z1 = x1.mean()
&gt;&gt;&gt; z1.backward()
&gt;&gt;&gt; x1.grad
Variable containing:
1.00000e-02 *
  5.0000  5.0000  5.0000  5.0000  5.0000
  5.0000  5.0000  5.0000  5.0000  5.0000
  5.0000  5.0000  5.0000  5.0000  5.0000
  5.0000  5.0000  5.0000  5.0000  5.0000
[torch.FloatTensor of size 4x5]

&gt;&gt;&gt;
&gt;&gt;&gt; x2 = torch.autograd.Variable (w2, requires_grad = True)
&gt;&gt;&gt; x2
Variable containing:
 1.2372 -0.9604  1.5415 -0.4079  0.8806
 0.0529  0.0751  0.4777 -0.6759 -2.1489
-1.1463 -0.2720  1.0066 -0.0416 -1.2853
-0.4948 -1.2964 -1.2502 -0.7693  1.6856
[torch.FloatTensor of size 4x5]

&gt;&gt;&gt; z2 = x2.sum()
&gt;&gt;&gt; z2.backward()
&gt;&gt;&gt; x2.grad
Variable containing:
 1  1  1  1  1
 1  1  1  1  1
 1  1  1  1  1
 1  1  1  1  1
[torch.FloatTensor of size 4x5]
</code></pre>
<p>Could you post a complete, runnable script that reproduces your issue<br>
and let us know what version of pytorch you are running?</p>
<p>Best.</p>
<p>K. Frank</p>

Sagnik_Mukherjee: <p>Is the question “How are the gradients computed numerically?” ?</p>

two_Two: <p>Dear Frank <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Oh,that’s so terrible,I would like to re-post this again and sorry about wasting your time…</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.5.1+cu101'
&gt;&gt;&gt; x = torch.tensor([[-0.3571,  0.1481,  0.1713, -1.2597, -0.7667],
...         [-0.1553, -0.9620,  0.0103,  3.3494,  0.2220],
...         [ 2.1131, -0.2404,  0.4820,  0.3816,  1.9752],
...         [ 1.7232, -0.5064, -0.8151,  0.3720,  0.1470]], requires_grad=True)
&gt;&gt;&gt; z = x.mean()
&gt;&gt;&gt; z.backward()
&gt;&gt;&gt; x.grad
tensor([[0.0500, 0.0500, 0.0500, 0.0500, 0.0500],
       [0.0500, 0.0500, 0.0500, 0.0500, 0.0500],
       [0.0500, 0.0500, 0.0500, 0.0500, 0.0500],
       [0.0500, 0.0500, 0.0500, 0.0500, 0.0500]])

</code></pre>
<p>And another one would be :</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch 
&gt;&gt;&gt; torch.__version__
'1.5.1+cu101'
&gt;&gt;&gt; x = torch.tensor([[-0.3571,  0.1481,  0.1713, -1.2597, -0.7667],
...         [-0.1553, -0.9620,  0.0103,  3.3494,  0.2220],
...         [ 2.1131, -0.2404,  0.4820,  0.3816,  1.9752],
...         [ 1.7232, -0.5064, -0.8151,  0.3720,  0.1470]], requires_grad=True)
&gt;&gt;&gt; z = x.sum()
&gt;&gt;&gt; z.backward()
&gt;&gt;&gt; x.grad
tensor([[1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.]])
</code></pre>
<p>And once again,Thanks in advance ! !</p>

two_Two: <p>Yeah,It could be haha,I was struggling how to give this post a great title,and it looks like I fail…</p>

Sagnik_Mukherjee: <p>Okay, I can help you a bit there. Gradient is a “fancy” name for derivative (differential calculus). so lets say you have a tensor (2*2) defined as:<br>
W11 W12<br>
W21 W22<br>
When you take torch.mean(), the returned variable say W = (W11 + W21 + W12 + W22)/4.0<br>
And hence the gradient matrix shall look like,<br>
d(W)/d(W11) d(W)/d(W12)<br>
d(W)/d(W21) d(W)/d(W22)</p>
<p>Which is;<br>
.25 .25<br>
.25 .25<br>
I think you can go through the PRML book by chris bishop and gain further insights.<br>
hope that helped.</p>

two_Two: <p>Sure ,why could not I figure this out …<br>
thinks of every element in tensor is a independent variables.and do basic calculus thing<br>
y=x1+x2<br>
dy / dx1 =1</p>
<p>if y=(x1+x2)/2<br>
dy/dx1 = 0.5</p>
<p>And thank you very much!</p>