Function TBackward returned an invalid gradient at index 0 - got [1, 3] but expected shape compatible with [1, 2]

username: Pedro_Coelho
cooked: <p>I’m working on a neural network that can expand in size from one iteration to the next according to changes in the inputs. While developing the training loop, I encountered a problem during back-propagation. I coded a minimal example that reproduces the problem:</p>
<pre><code class="lang-auto">import torch
from torch import nn

l1 = nn.Linear(2, 1)

x = torch.tensor([1.0, 3.0])

# with torch.no_grad():
y_pred = l1(x)

x = torch.tensor([1.0, 3.0, -2.0])
y = torch.tensor([2.0])

l1.weight.data = torch.cat(
    [l1.weight.data, torch.randn((1, 1))],
    dim=1
)
l1.in_features += 1

y_pred = l1(x)
loss = (y - y_pred)**2

loss.backward()
</code></pre>
<p>This gives the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/Users/pedrocoelho/gnn-poc/bug_test.py", line 25, in &lt;module&gt;
    loss.backward()
  File "/usr/local/lib/python3.9/site-packages/torch/_tensor.py", line 255, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/usr/local/lib/python3.9/site-packages/torch/autograd/__init__.py", line 147, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Function TBackward returned an invalid gradient at index 0 - got [1, 3] but expected shape compatible with [1, 2]
</code></pre>
<p>What could be causing this? I know it has to do with the incompatibility of shape of the gradients, but I find that odd since the gradients are not calculated until loss.backward() is called. It might also be related to the intermediate results being stored in the context of a forward pass. How do I access/clear those intermediate results after each forward+backward iteration? Also, what exactly is the TBackward function? I couldn’t find much information about it.</p>
<p>Calling the first y_pred = l1(x)  inside the with torch.no_grad()  fixes the problem but it isn’t an option because I need want to do a backward pass at every iteration, with different network sizes.</p>
<p>The computation graph for the loss looks like this:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a80310e663a3ce1c9444a566a234a4a1a4f095b.png" data-download-href="https://discuss.pytorch.org/uploads/default/6a80310e663a3ce1c9444a566a234a4a1a4f095b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6a80310e663a3ce1c9444a566a234a4a1a4f095b_2_243x500.png" alt="image" data-base62-sha1="fc9gbTDsr88l2FDpp17p5Xm0xWb" width="243" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6a80310e663a3ce1c9444a566a234a4a1a4f095b_2_243x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6a80310e663a3ce1c9444a566a234a4a1a4f095b_2_364x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6a80310e663a3ce1c9444a566a234a4a1a4f095b_2_486x1000.png 2x" data-dominant-color="EEEFEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">656×1348 33.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>You are currently manipulating the <code>.data</code> attribute, which is not supported and can yield unwanted side effects such as probably this one.<br>
If you want to replace the <code>.weight</code> parameter of the linear layer, wrap the assignment in a <code>with torch.no_grad()</code> block and assign a new <code>nn.Parameter</code> to it.</p>

username: Pedro_Coelho
cooked: <p>Thank you,<br>
As you said, the following way of expanding the weights works:</p>
<pre><code class="lang-auto">with torch.no_grad():
    l1.weight = nn.Parameter(
        torch.cat(
            [
                l1.weight.data,
                torch.randn((1, 1))
            ],
            dim=1
        )
    )
</code></pre>