How to conveniently combine elements of a tensor into a new tensor?

rou: <p>Let’s say that given a tensor of length 3 with <code>requires_grad=True</code>, I want to <em>manually</em> create a 3x3 skew-symmetric matrix for that tensor.</p>
<p>As a PyTorch newbie, this is what I would expect should work:</p>
<pre data-code-wrap="python"><code class="lang-python">def variant_1(x):
    skew_symmetric_mat = torch.tensor([
        [0, -x[2], x[1]],
        [x[2], 0.0, -x[0]],
        [-x[1], x[0], 0.0]
    ])
    return skew_symmetric_mat

vec = torch.rand(3, requires_grad=True)
variant_1(vec).backward(torch.ones(3, 3))  # RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>However, <code>variant_1</code> fails with the runtime error mentioned in the snippet. I guess the underlying problem is that creating the tensor <code>skew_symmetric_mat</code> and populating it with elements of the other tensor <code>x</code> is not a differentiable operation and when initializing <code>skew_symmetric_mat</code>, only the <em>values</em> of <code>x</code> get copied into <code>skew_symmetric_mat</code>, so there is no backward graph being created that would reference the elements of the tensor <code>x</code>.</p>
<p>With this assumption in mind, I was able to write a fully working function <code>variant_2()</code>, which doesn’t assign the element of <code>x</code> to <code>skew_symmetric_mat</code>, but rather multiplies the elements of <code>x</code> by new tensors, which most probably results in proper creation of DAG:</p>
<pre data-code-wrap="python"><code class="lang-python">def variant_2(x):
    skew_symmetric_mat = torch.zeros(3, 3)

    skew_symmetric_mat += x[0] * torch.tensor([
        [0, 0, 0],
        [0, 0, -1.0],
        [0, 1.0, 0]
    ])

    skew_symmetric_mat += x[1] * torch.tensor([
        [0, 0, 1.0],
        [0, 0, 0],
        [-1.0, 0, 0.0],
    ])

    skew_symmetric_mat += x[2] * torch.tensor([
        [0, -1.0, 0],
        [1.0, 0, 0],
        [0, 0, 0.0],
    ])

    return skew_symmetric_mat

vec = torch.rand(3, requires_grad=True)
variant_2(vec).backward(torch.ones(3, 3))  # Computes `vec.grad` just fine
</code></pre>
<p>My question is: <code>variant_2</code> seems a bit too much verbose to my liking and it seem like it’s computationally wasteful too. Surely, there must be a way how to write a code that is as compact as <code>variant_1()</code> while also being computationally efficient. How would you go about writing this?</p>
<p>P.S. Apologies for such a trivial question. I couldn’t even find the right terminology to Google a solution.</p>

InnovArul: <p>If you are open to use external libraries, pl. try using <code>kornia</code>’s <code>vector_to_skew_symmetric_matrix()</code> for this conversion.</p>
<p><a href="https://kornia.readthedocs.io/en/latest/geometry.conversions.html#kornia.geometry.conversions.vector_to_skew_symmetric_matrix" class="onebox" target="_blank" rel="noopener">https://kornia.readthedocs.io/en/latest/geometry.conversions.html#kornia.geometry.conversions.vector_to_skew_symmetric_matrix</a></p>

rou: <p>Thank you for pointing me to the library. My end goal was not necessarily to create a skew-symmetric matrix, but to learn what is the best practice in general of how to construct an intermediary tensor consisting of elements of other tensor(s), which could then be used during computation of a complicated expression.</p>
<p>Thanks to your pointer, I was able to look up the <a href="https://github.com/kornia/kornia/blob/f6ff780d17f21bff1b3b26472ea19dfdc95cf899/kornia/geometry/conversions.py#L1490" rel="noopener nofollow ugc">source code of <code>vector_to_skew_symmetric_matrix()</code></a>, which uses <code>torch.stack()</code> to construct the skew-symmetric matrix. I assume that <code>torch.stack()</code> is autograd-compatible (i.e. differentiable) alternative to creating tensors via <code>torch.tensor()</code>. Based on that, I created <code>variant_3()</code>.</p>
<p>May I ask if this is really the most performant and the “best practice” way of how one would go about implementing such operation? I assume that Kornia devs wouldn’t implement a poor-performing technique into the library, so their approach should be the best way to go about implementing skew-symmetric matrix, right?</p>
<pre data-code-wrap="python"><code class="lang-python">def variant_3(x):
    zero = torch.tensor(0.0)

    skew_symmetric_mat = torch.stack([
        torch.stack([zero, -x[2], x[1]]),
        torch.stack([x[2], zero, -x[0]]),
        torch.stack([-x[1], x[0], zero])
    ])

    return skew_symmetric_mat
</code></pre>

InnovArul: <p>I thnk the main issue is that the gradients are not tracked when constructing a tensor using<br>
<code>torch.tensor(), torch.Tensor()</code> from individual elements of another tensor.<br>
whereas, <code>torch.stack()</code> is keeping the computation graph intact and able to backpropagate.</p>
<p>In my understanding, <code>torch.tensor(), torch.Tensor()</code> are meant to be constructors to create a tensor and they are not meant to keep the computation graph intact. So I would avoid using them in-between end-to-end network operations when there is a need to backpropagate gradients.</p>
<p>Just wanted to document other quirks when using <code>torch.tensor(), torch.Tensor()</code> about sharing memories between source tensor and created tensor.</p>
<h3><a name="p-454176-creation-of-tensor-from-individual-elements-of-source-tensor-1" class="anchor" href="#p-454176-creation-of-tensor-from-individual-elements-of-source-tensor-1"></a>Creation of tensor from individual elements of source tensor</h3>
<ul>
<li><code>torch.tensor(), torch.Tensor()</code>: When creating tensor using individual elements of other tensors, gradient is not tracked.</li>
</ul>
<pre><code class="lang-auto">src = torch.randn(3, requires_grad=True)
clone = torch.tensor([src[0], src[1], src[2]])
clone.sum().backward()
# RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

src = torch.randn(3, requires_grad=True)
clone = torch.Tensor([src[0], src[1], src[2]])
clone.sum().backward()
# RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<h3><a name="p-454176-torchtensorsource_tensor-2" class="anchor" href="#p-454176-torchtensorsource_tensor-2"></a>torch.tensor(source_tensor)</h3>
<ul>
<li><code>torch.tensor()</code> always clones the memory and does not track gradients.</li>
</ul>
<pre><code class="lang-auto">src = torch.randn(3, requires_grad=True)
clone = torch.tensor(src)
clone.sum().backward()
src.grad
# RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

clone[1] = 5.0
print(f"src {src}, clone {clone}")
# src tensor([ 0.3578,  2.4084, -1.3494], requires_grad=True), clone tensor([ 0.3578,  5.0000, -1.3494])
</code></pre>
<h3><a name="p-454176-torchtensorsource_tensor-3" class="anchor" href="#p-454176-torchtensorsource_tensor-3"></a>torch.Tensor(source_tensor)</h3>
<ul>
<li>When using with a source tensor as a whole, <code>torch.Tensor()</code> just creates an alias to source tensor and shares the memory with source tensor.</li>
</ul>
<pre><code class="lang-auto">src = torch.randn(3)
clone = torch.Tensor(src)
print(f"src {src}, clone {clone}")
# src tensor([-0.7193, -2.2648,  1.5096]), clone tensor([-0.7193, -2.2648,  1.5096])

clone[1] = 5.0
print(f"src {src}, clone {clone}")
# src tensor([-0.7193,  5.0000,  1.5096]), clone tensor([-0.7193,  5.0000,  1.5096])

src.requires_grad = True
print(f"src {src}, clone {clone}")
# src tensor([-0.7193,  5.0000,  1.5096], requires_grad=True), clone tensor([-0.7193,  5.0000,  1.5096], grad_fn=&lt;AsStridedBackward0&gt;)

clone.sum().backward()
src.grad
print(src.grad)
# tensor([1., 1., 1.])
# 
</code></pre>

rou: <p>Thank you for pointing out the quirks, this is going to save me a lot of debugging time <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20">.</p>

KFrank: <p>Hi Rou!</p>
<aside class="quote no-group quote-modified" data-username="rou" data-post="1" data-topic="209390" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/c67d28/48.png" class="avatar"> rou:</div>
<blockquote>
<p>Let’s say that given a tensor of length 3 with <code>requires_grad=True</code>, I want to <em>manually</em> create a 3x3 skew-symmetric matrix for that tensor.<br>
…</p>
<pre data-code-wrap="python"><code class="lang-python">def variant_2(x):
    skew_symmetric_mat = torch.zeros(3, 3)

    skew_symmetric_mat += x[0] * torch.tensor([
        [0, 0, 0],
        [0, 0, -1.0],
        [0, 1.0, 0]
    ])

    skew_symmetric_mat += x[1] * torch.tensor([
        [0, 0, 1.0],
        [0, 0, 0],
        [-1.0, 0, 0.0],
    ])

    skew_symmetric_mat += x[2] * torch.tensor([
        [0, -1.0, 0],
        [1.0, 0, 0],
        [0, 0, 0.0],
    ])

    return skew_symmetric_mat

vec = torch.rand(3, requires_grad=True)
variant_2(vec).backward(torch.ones(3, 3))  # Computes `vec.grad` just fine
</code></pre>
<p>My question is: <code>variant_2</code> seems a bit too much verbose to my liking and it seem like it’s computationally wasteful too.</p>
</blockquote>
</aside>
<p>For your particular example of generating a skew-symmetric<br>
matrix from a (<code>requires_grad = True</code>) vector of its independent<br>
components, it will likely to be fastest to compute the matrix, as<br>
in your <code>variant_2()</code>, but to package the “basis” matrices of the<br>
skew-symmetric matrix more efficiently:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
2.4.1
&gt;&gt;&gt;
&gt;&gt;&gt; skew_components =  torch.tensor ([
...     [[0, 0, 0],
...      [0, 0, -1.0],
...      [0, 1.0, 0]],
...     [[0, 0, 1.0],
...      [0, 0, 0],
...      [-1.0, 0, 0.0]],
...     [[0, -1.0, 0],
...      [1.0, 0, 0],
...      [0, 0, 0.0]]
... ])
&gt;&gt;&gt;
&gt;&gt;&gt; vec = torch.randn (3, requires_grad = True)
&gt;&gt;&gt;
&gt;&gt;&gt; skew_mat = skew_components @ vec
&gt;&gt;&gt;
&gt;&gt;&gt; skew_mat.backward (torch.ones (3, 3))
&gt;&gt;&gt;
&gt;&gt;&gt; vec.grad   # zeros are expected
tensor([0., 0., 0.])
</code></pre>
<p>More generally, when you are generating a derived tensor from<br>
a (<code>requires_grad = True</code>) initial tensor, whether it will be more<br>
efficient to “assemble” (e.g., things like <code>stack()</code>) or compute<br>
the derived tensor will depend on the details.  But when the<br>
computation can be packaged as a single tensor operation<br>
(such as <code>skew_components @ vec</code>), that will likely be the<br>
fastest way to go.  (After all, that’s what pytorch is optimized<br>
for.)</p>
<p>Best.</p>
<p>K. Frank</p>