Can PyTorch find a d-dimensional vector perpendicular to d - 1 random d-dimensional vectors?

Richard_Wang: <p>Hi all,<br>
Can I use pytorch or numpy or anything, either by off-the-shelf method, script or library…</p>
<p>Given random d-1 d-dimensional vectors (specifically a random d-1xd matrix), get an d-dimensional vector (specifically (d,) shape tensor) that is perpendicular to every vector mentioned above ?</p>
<p>something like this</p>
<pre><code class="lang-auto">vectors = torch.randn(d-1,d)
perpendicular_vector = torch.get_perpendicular(vectors) # (d,)
assert vectors.matmul(perpendicular_vector).sum().item() == 0
</code></pre>

Arna_Ghosh: <p>Not sure if this functionality already exists, but you could try writing a custom function to do this perhaps.<br>
Let’s say that the given matrix is called A. You figure out the eigenvalues of A and if the smallest (ordered by absolute values) is 0, then the corresponding (right) eigenvector is your desired vector. If not, you can calculate something like <a href="https://en.wikipedia.org/wiki/Gram%E2%80%93Schmidt_process" rel="noopener nofollow ugc">Gram-Schmidt process</a>. Basically, start from some random vector and keep subtracting the component of each eigenvector from it iteratively. That should give you the vector that is perpendicular to all rows of A.<br>
Hope this helps.</p>

KFrank: <p>Hi Arna (and Richard)!</p>
<aside class="quote no-group" data-username="Arna_Ghosh" data-post="2" data-topic="99078" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arna_ghosh/48/37135_2.png" class="avatar"> Arna_Ghosh:</div>
<blockquote>
<p>You figure out the eigenvalues of A</p>
</blockquote>
</aside>
<p>Note that A is not square in this case.</p>
<aside class="quote no-group">
<blockquote>
<p>you can calculate something like <a href="https://en.wikipedia.org/wiki/Gram%E2%80%93Schmidt_process" rel="noopener nofollow ugc">Gram-Schmidt process</a>.</p>
</blockquote>
</aside>
<p>Probably the most numerically satisfactory way to do this is to<br>
compute the singular-value decomposition of A  with <a href="https://pytorch.org/docs/1.9.0/generated/torch.linalg.svd.html" rel="noopener nofollow ugc">torch.linalg.svd()</a>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; _ = torch.manual_seed (2021)
&gt;&gt;&gt; vectors = torch.randn (4, 5)
&gt;&gt;&gt; vectors
tensor([[-0.1452,  0.9747,  0.6023,  1.5775,  0.7978],
        [-0.4101, -1.2622, -0.3932,  0.4675, -0.2879],
        [-0.0742,  0.1680,  1.6226,  0.7453, -2.9535],
        [-1.1143, -0.8463, -0.0393, -1.9088, -0.3105]])
&gt;&gt;&gt; perpendicular_vector = torch.linalg.svd (vectors).Vh[4]
&gt;&gt;&gt; perpendicular_vector
tensor([ 0.4183, -0.4666,  0.7019, -0.1041,  0.3222])
&gt;&gt;&gt; vectors @ perpendicular_vector
tensor([-2.9802e-08, -8.9407e-08, -2.3842e-07, -8.9407e-08])
</code></pre>
<p>Note that if A (<code>vectors</code>) is degenerate (that is, that some linear<br>
combination of its rows is zero), then <code>perpendicular_vector</code><br>
will no longer be (in essence) unique.  In such a case A will have<br>
a multi-dimensional <em>null space</em> (sometimes calls its <em>kernel</em>), and<br>
any vector lying in A’s null space will be (by definition) perpendicular<br>
to A’s rows.</p>
<p>Best.</p>
<p>K. Frank</p>

Arna_Ghosh: <p>You are absolutely right <a class="mention" href="/u/kfrank">@KFrank</a> ! Thanks for pointing out that singular values, and not eigenvalues, are what’s needed here.<br>
A small suggestion: It’s probably worth clarifying the in the code that you used the last right eigenvector, perhaps changing it to <code>torch.linalg.svd(vectors).Vh[-1]</code> adds a little more insight into the entire solution?</p>