Taking derivative of Linear module

username: Kakatorch
cooked: <p>Hi, I am trying something very basic with pytorch. I am creating a <code>Linear</code> module and setting its weights to something I desire as follows:</p>
<pre><code class="lang-python">import torch
torch_linfn = torch.nn.Linear(2, 3, bias=True)
torch_linfn.weight = torch.nn.Parameter(
    torch.tensor([[1,0], [-1,14], [5,-9]], dtype=torch.float32))
torch_linfn.bias = torch.nn.Parameter(
    torch.tensor([1, 4, -1], dtype=torch.float32))
</code></pre>
<p>Now, I do a forward pass on the module:</p>
<pre><code class="lang-python">x = torch.tensor([1, -12], dtype=torch.float32)
torch_y = torch_linfn(x)
# the output is: tensor([   2., -165.,  112.], grad_fn=&lt;AddBackward0&gt;)
</code></pre>
<p>Now, I hope to get the gradient with respect to the weights (essentially, get <code>dy/dW</code>):</p>
<pre><code class="lang-auto">torch_y.backward()
</code></pre>
<p>This throws the following error:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch_y.backward()
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/media/kz-wd-ssd/repo/zkynet/venv/zkynet/lib/python3.8/site-packages/torch/_tensor.py", line 363, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/media/kz-wd-ssd/repo/zkynet/venv/zkynet/lib/python3.8/site-packages/torch/autograd/__init__.py", line 166, in backward
    grad_tensors_ = _make_grads(tensors, grad_tensors_, is_grads_batched=False)
  File "/media/kz-wd-ssd/repo/zkynet/venv/zkynet/lib/python3.8/site-packages/torch/autograd/__init__.py", line 67, in _make_grads
    raise RuntimeError("grad can be implicitly created only for scalar outputs")
RuntimeError: grad can be implicitly created only for scalar outputs
</code></pre>
<p>How come? Why doesn’t pytorch’s autograd work for non-scalar outputs?</p>

username: Andrei_Cristea
cooked: <p>I guess you would have three different gradients, w.r.t. each of the elements in y. However, you could do this, which is maybe what you meant?</p>
<pre><code class="lang-auto">torch_y.sum().backward()  # adding .sum()
print(torch_linfn.weight.grad)
print(torch_linfn.bias.grad)
Output:
tensor([[  1., -12.],
        [  1., -12.],
        [  1., -12.]])
tensor([1., 1., 1.])
</code></pre>

username: Kakatorch
cooked: <p>Yes that’s what I meant. Didn’t know you can get <code>grad</code> from the weight and bias. Thanks!</p>

username: Kakatorch
cooked: <p>Actually, is there a reason pytorch does not support getting the gradients at the weight and bias without the <code>.sum()</code> function?</p>

username: Andrei_Cristea
cooked: <p>It does support it, just not default behavior. Here’s an example:</p>
<pre><code class="lang-auto">optim = torch.optim.SGD(params=torch_linfn.parameters(), lr=1e-4)

for i_y in range(torch_linfn.weight.shape[0]):
    optim.zero_grad()
    torch_y[i_y].backward(retain_graph=True)
    print(f"dy_{i_y}/dx")
    print(torch_linfn.weight.grad[i_y], end="\n\n")

Output:
dy_0/dx
tensor([  1., -12.])

dy_1/dx
tensor([  1., -12.])

dy_2/dx
tensor([  1., -12.])
</code></pre>
<p>More info about autograd <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html" rel="noopener nofollow ugc">here</a>.</p>