Can't flow gradients through new constructed tensor

username: yoterel
cooked: <p>When I try to construct a tensor (A 3x3 matrix in my case) from multiple other tensors that require grad, the resulting tensor does not have require_grad = True by default, and even if I set it manually, when backproping the tensors that were used to construct the matrix do not get gradients. Why is this so?</p>
<p>example code:</p>
<pre><code class="lang-auto">test0 = torch.ones(1, dtype=torch.float32, requires_grad=True)
test1 = torch.ones(1, dtype=torch.float32, requires_grad=True)
test2 = torch.ones(1, dtype=torch.float32, requires_grad=True)
test_mat = torch.tensor([[test0, test1, 0], [1, 0 ,0], [test2, test1, 1]])
print(test_mat.requires_grad)
sum = test_mat.sum()
sum.backward()
print(test0.grad is None)
</code></pre>
<blockquote>
<p>False<br>
True</p>
</blockquote>
<p>How can I construct a tensor out of other tensors such that gradients flow properly? I want to assemble the individual entries into a matrix and multiply another tensor by it (a 3xN tensor), instead of optimizing a full 3x3 matrix (since some of its entries are constants, and I require it to have some properties such as symmetry, which can’t be enforced directly if I optimize all the entries).</p>
<p>edit: For now I am creating an empty 3x3 matrix and replacing elements directly. Is this the intended way?</p>
<p>Thank you.</p>

username: KFrank
cooked: <p>Hi Yotam!</p>
<aside class="quote no-group" data-username="yoterel" data-post="1" data-topic="155732" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yoterel/48/51137_2.png" class="avatar"> yoterel:</div>
<blockquote>
<p>when backproping the tensors that were used to construct the matrix do not get gradients. Why is this so?</p>
</blockquote>
</aside>
<p>The short answer is that this is the way that the factory function<br>
<code>torch.tensor()</code> works.</p>
<aside class="quote no-group">
<blockquote>
<p>edit: For now I am creating an empty 3x3 matrix and replacing elements directly. Is this the intended way?</p>
</blockquote>
</aside>
<p>This is perfectly fine.</p>
<p>As an alternative – not necessarily better – you can use <code>cat()</code> and<br>
<code>stack()</code> to compose <code>test_mat</code> out of the desired pieces:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.12.0
&gt;&gt;&gt;
&gt;&gt;&gt; test0 = torch.ones (1, dtype = torch.float32, requires_grad = True)
&gt;&gt;&gt; test1 = torch.ones (1, dtype = torch.float32, requires_grad = True)
&gt;&gt;&gt; test2 = torch.ones (1, dtype = torch.float32, requires_grad = True)
&gt;&gt;&gt;
&gt;&gt;&gt; test_mat = torch.stack (
...     (
...         torch.cat ((test0, test1, torch.tensor ([0.0]))),
...         torch.tensor ([1.0, 0.0, 1.0]),
...         torch.cat ((test2, test1, torch.tensor ([1.0])))
...     )
... )
&gt;&gt;&gt;
&gt;&gt;&gt; test_mat
tensor([[1., 1., 0.],
        [1., 0., 1.],
        [1., 1., 1.]], grad_fn=&lt;StackBackward0&gt;)
&gt;&gt;&gt;
&gt;&gt;&gt; test_mat.sum().backward()
&gt;&gt;&gt;
&gt;&gt;&gt; test0.grad
tensor([1.])
&gt;&gt;&gt; test1.grad
tensor([2.])
&gt;&gt;&gt; test2.grad
tensor([1.])
</code></pre>
<p>It would be a matter of style and maybe the details of your use case<br>
as to the preferred approach.</p>
<p>Best.</p>
<p>K. Frank</p>