Why the grad is chagnging

zwj: <p>I am using a Linear network, the structure of the network is following:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/4/241f2bf2c12f63d2cbefd3428ebaaa87260dd35a.png" data-download-href="https://discuss.pytorch.org/uploads/default/241f2bf2c12f63d2cbefd3428ebaaa87260dd35a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/241f2bf2c12f63d2cbefd3428ebaaa87260dd35a_2_690x112.png" alt="image" data-base62-sha1="59xYnOxbLQHx6i2ZX7RLLyq6JC2" width="690" height="112" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/241f2bf2c12f63d2cbefd3428ebaaa87260dd35a_2_690x112.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/241f2bf2c12f63d2cbefd3428ebaaa87260dd35a_2_1035x168.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/241f2bf2c12f63d2cbefd3428ebaaa87260dd35a_2_1380x224.png 2x" data-dominant-color="F3F3F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1384×226 41.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
as you can see, This is a linear network and no activation function.<br>
I think when I am using backward function, The gradient of the input vector is supposed to be constant, but the result seems change. The following image show the grad of full zero vector<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/9/8969b24c18396cdc644a563ad6d019bee90621a4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8969b24c18396cdc644a563ad6d019bee90621a4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/8969b24c18396cdc644a563ad6d019bee90621a4_2_690x245.jpeg" alt="image" data-base62-sha1="jBBRL8E8RF2dr4fuEjgBGY3We2g" width="690" height="245" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/8969b24c18396cdc644a563ad6d019bee90621a4_2_690x245.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/8969b24c18396cdc644a563ad6d019bee90621a4_2_1035x367.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/8969b24c18396cdc644a563ad6d019bee90621a4_2_1380x490.jpeg 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2286×814 209 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
when I change the input vector, The result changes, like the following image<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa1488ad06c0782a67f9d6ee11b54a4841f22e4c.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/fa1488ad06c0782a67f9d6ee11b54a4841f22e4c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa1488ad06c0782a67f9d6ee11b54a4841f22e4c_2_611x500.jpeg" alt="image" data-base62-sha1="zGjmoIx33z8q8KMwPEPToUSTisk" width="611" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa1488ad06c0782a67f9d6ee11b54a4841f22e4c_2_611x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa1488ad06c0782a67f9d6ee11b54a4841f22e4c_2_916x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa1488ad06c0782a67f9d6ee11b54a4841f22e4c_2_1222x1000.jpeg 2x" data-dominant-color="EBEBEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1596×1304 198 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is there something wrong?</p>

Andrei_Cristea: <p>The gradient is not constant, it’s dependent on the input. Gradient is the derivative of the loss w.r.t. the weights. When the input is all zeros, the derivative is going to be zero everywhere (since d(0*W)/d(W) = 0) however when the input is nonzero somewhere, the derivative is no longer zero everywhere.</p>

zwj: <p>Thanks for your answer, but here I am getting the derivative of the loss w.r.t. the input.  It supposed to be d(x*W)/d(x)=W.</p>

Andrei_Cristea: <p>We have two parallel conversations going on the same question, but basically I think when you run output.backward() it computes dy/dw not dy/dx.</p>

zwj: <p>so, maybe my calculation method is wrong?</p>

Andrei_Cristea: <p>I tried this and it seems to give the same answer when I change the x, so I think this might be what you want:</p>
<pre><code class="lang-auto">net = nn.Linear(5, 10)
</code></pre>
<p>Then, for 2 different X, we get the same dy/dx:</p>
<pre><code class="lang-auto">input_tensor = torch.zeros(1, 5)
input_tensor.requires_grad_()
torch.autograd.grad(net(input_tensor).sum(), input_tensor, create_graph=True)

(tensor([[-1.0625, -0.4604, 0.7039, 0.1229, -1.1627]], grad_fn=&lt;MmBackward&gt;),)
</code></pre>
<pre><code class="lang-auto">input_tensor = torch.ones(1, 5)
input_tensor.requires_grad_()
torch.autograd.grad(net(input_tensor).sum(), input_tensor, create_graph=True)

(tensor([[-1.0625, -0.4604, 0.7039, 0.1229, -1.1627]], grad_fn=&lt;MmBackward&gt;),)
</code></pre>
<p>Which exactly match the weight sums:</p>
<pre><code class="lang-auto">net.weight.sum(dim=0)

tensor([-1.0625, -0.4604,  0.7039,  0.1229, -1.1627], grad_fn=&lt;SumBackward1&gt;)
</code></pre>

Andrei_Cristea: <p>Turns out the issue was that the existence of nonlinear layers in the model (e.g. RELU) means that dx/dy does depend on x.</p>

zwj: <p>Yes, I have found the reason. The network has a normalization towards the input and the output of the network, making it no longer a linear network. So the dy/dx is dependent on the x</p>