Instability of torch.inverse?

username: Raphikowski
cooked: <p>Dear all,</p>
<p>I am trying to do the following: Given I have some column vector of shape (5 x 1) I want to calculate the inverse of (q @ q.t()). This actually shouldn’t be such a difficult task, however, it seems that the inverse of this expression, torch.inverse(q @ q.t()) is way off:</p>
<pre><code class="lang-auto">In [358]: q = torch.randn(5,1)                                                                                                                                                                                 
In [359]: q                                                                                                                                                                                                    
Out[359]: 
tensor([[ 0.2144],
        [ 0.2899],
        [-1.0016],
        [-2.5544],
        [ 0.3549]])

In [360]: (q @ q.t()) @ torch.inverse(q @ q.t())                                                                                                                                                               
Out[360]: 
tensor([[ 0.2019, -1.0537,  0.2293, -0.3905, -0.7937],
        [-0.2614, -0.9735,  0.2418, -0.4921, -1.4343],
        [ 0.4410,  7.5348,  0.6504,  1.9400,  3.7739],
        [ 4.5718, 17.0569, -1.1591,  4.7328, 12.1593],
        [ 0.3166, -1.8443,  0.1681, -0.5732, -0.4462]])
</code></pre>
<p>So this calculation should obviously return something which is close to the identity. So is this some numerical issue?<br>
If I just create a (5 x 5) matrix via w = torch.randn(5,5) and then calculate (w @ w.t()) @ torch.inverse(w @ w.t()) there is no issue and the result is close to the identity.<br>
So is there something special by matrix multiplying a column vector with its transpose to obtain a matrix?</p>
<p>So maybe what I am missing is that q @ q.t() is often singular or close to singular? And hence the inverse is actually not defined?</p>
<p>Best regards</p>

username: KFrank
cooked: <p>Hi Raphikowski!</p>
<aside class="quote no-group quote-modified" data-username="Raphikowski" data-post="1" data-topic="67029" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/ec9cab/48.png" class="avatar"> Raphikowski:</div>
<blockquote>
<p>Given I have some column vector of shape (5 x 1) I want to calculate the inverse of (q @ q.t()). This actually shouldn’t be such a difficult task, however, it seems that the inverse of this expression, torch.inverse(q @ q.t()) is way off:</p>
<p>…</p>
<p>So is there something special by matrix multiplying a column vector with its transpose to obtain a matrix?</p>
<p>So maybe what I am missing is that q @ q.t() is often singular or close to singular? And hence the inverse is actually not defined?</p>
</blockquote>
</aside>
<p>Yes, your suspicion is correct.</p>
<p><em>By construction,</em> your matrix is mathematically non-invertible.<br>
(Numerically, it is likely technically invertible, but singular within<br>
round-off error, so you get nonsense results when you try to invert<br>
it numerically.)</p>
<p>Your matrix – by construction – is a <em>projection</em> matrix; that is, by<br>
matrix-vector multiplication it projects any vector onto the vector <code>q</code>.<br>
As such, it has only one non-zero eigenvalue, with corresponding<br>
eigenvector (proportional to) <code>q</code>.</p>
<p>This is illustrated by the following (version 0.3.0) script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__

torch.manual_seed (2020)
q = torch.randn(5,1)
q   # your random vector
qq = q @ q.t()
qq   # your square "projection" matrix

vals, vecs = torch.eig (qq, True)

vals   # all but one zero eigenvalue

max, ind = vals[:,0].max (0)   # get index of non-zero eigenvalue
vec = vecs[:,ind]   # get non-zero eigenvector

q.div (vec)   #  q is (proportional to) the non-zero eigenvector
</code></pre>
<p>And here is the output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'0.3.0b0+591e73e'
&gt;&gt;&gt;
&gt;&gt;&gt; torch.manual_seed (2020)
&lt;torch._C.Generator object at 0x000001F5A11D6630&gt;
&gt;&gt;&gt; q = torch.randn(5,1)
&gt;&gt;&gt; q   # your random vector

 1.2372
-0.9604
 1.5415
-0.4079
 0.8806
[torch.FloatTensor of size 5x1]

&gt;&gt;&gt; qq = q @ q.t()
&gt;&gt;&gt; qq   # your square "projection" matrix

 1.5307 -1.1882  1.9072 -0.5046  1.0895
-1.1882  0.9224 -1.4805  0.3917 -0.8457
 1.9072 -1.4805  2.3763 -0.6287  1.3574
-0.5046  0.3917 -0.6287  0.1664 -0.3592
 1.0895 -0.8457  1.3574 -0.3592  0.7754
[torch.FloatTensor of size 5x5]

&gt;&gt;&gt;
&gt;&gt;&gt; vals, vecs = torch.eig (qq, True)
&gt;&gt;&gt;
&gt;&gt;&gt; vals   # all but one zero eigenvalue

 0.0000  0.0000
 5.7712  0.0000
 0.0000  0.0000
-0.0000  0.0000
-0.0000  0.0000
[torch.FloatTensor of size 5x2]

&gt;&gt;&gt;
&gt;&gt;&gt; max, ind = vals[:,0].max (0)   # get index of non-zero eigenvalue
&gt;&gt;&gt; vec = vecs[:,ind]   # get non-zero eigenvector
&gt;&gt;&gt;
&gt;&gt;&gt; q.div (vec)   #  q is (proportional to) the non-zero eigenvector

 2.4023
 2.4023
 2.4023
 2.4023
 2.4023
[torch.FloatTensor of size 5x1]

&gt;&gt;&gt;
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: Raphikowski
cooked: <p>Hi Frank,<br>
great thank you very much for your answer!<br>
So maybe another question then:<br>
If I have a multivariate linear regression kind of problem of the form:<br>
X @ b = y<br>
with X being a (N x M) matrix, b being a (M x 1) column vector and y being a (N x 1) column vector. And I want to solve this now for X and not for b, what do I do to make this solution numerically tractable. Since my naive solutions would have been:<br>
X = y @ b.t() @ (b <span class="mention">@b.t</span>())^-1<br>
but here I obviously have the problem that I have to invert the  projection matrix (b @ b.t())<br>
So is there a common work around for that?<br>
Best regards</p>

username: KFrank
cooked: <p>Hello Raphikowski!</p>
<aside class="quote no-group" data-username="Raphikowski" data-post="3" data-topic="67029" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/ec9cab/48.png" class="avatar"> Raphikowski:</div>
<blockquote>
<p>If I have a multivariate linear regression kind of problem of the form:<br>
X @ b = y<br>
with X being a (N x M) matrix, b being a (M x 1) column vector and y being a (N x 1) column vector. And I want to solve this now for X</p>
</blockquote>
</aside>
<p>To be clear:  Given vectors <code>b</code> and <code>y</code>, you wish to find a matrix <code>X</code><br>
such that <code>X @ b = y</code>.</p>
<p>The first thing to note is that there will be many such <code>X</code>.  If <code>Q</code> is a<br>
matrix that has <code>y</code> as an eigenvector (with eigenvalue one) and <code>R</code><br>
is a matrix that has <code>b</code> as an eigenvector (with eigenvalue one),<br>
then <code>Q @ X</code> and <code>X @ R</code> will both also solve your equation, that is,<br>
both <code>(Q @ X) @ b = y</code> and <code>(X @ R) @ b = y</code>.</p>
<p>One solution to your equation is given by</p>
<pre data-code-wrap="python"><code class="lang-python">X = (y @ b.t()) / b.norm()**2
</code></pre>
<p>as illustrated by the following script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__

torch.manual_seed (2020)

# let M = 5 and N = 7, and generate random values for b and y
b = torch.randn(5,1)
y = torch.randn(7,1)
b
y

X = (y @ b.t()) / b.norm()**2
X

X @ b
# check result two ways
X @ b - y
X @ b / y

# try another random vector
c = torch.randn(5,1)
# multiplication by this X always gives a vector proportional to y
X @ c / y
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'0.3.0b0+591e73e'
&gt;&gt;&gt;
&gt;&gt;&gt; torch.manual_seed (2020)
&lt;torch._C.Generator object at 0x0000022EF2646630&gt;
&gt;&gt;&gt;
&gt;&gt;&gt; # let M = 5 and N = 7, and generate random values for b and y
... b = torch.randn(5,1)
&gt;&gt;&gt; y = torch.randn(7,1)
&gt;&gt;&gt; b

 1.2372
-0.9604
 1.5415
-0.4079
 0.8806
[torch.FloatTensor of size 5x1]

&gt;&gt;&gt; y

 0.0529
 0.0751
 0.4777
-0.6759
-2.1489
-1.1463
-0.2720
[torch.FloatTensor of size 7x1]

&gt;&gt;&gt;
&gt;&gt;&gt; X = (y @ b.t()) / b.norm()**2
&gt;&gt;&gt; X

 0.0113 -0.0088  0.0141 -0.0037  0.0081
 0.0161 -0.0125  0.0201 -0.0053  0.0115
 0.1024 -0.0795  0.1276 -0.0338  0.0729
-0.1449  0.1125 -0.1805  0.0478 -0.1031
-0.4607  0.3576 -0.5740  0.1519 -0.3279
-0.2457  0.1908 -0.3062  0.0810 -0.1749
-0.0583  0.0453 -0.0727  0.0192 -0.0415
[torch.FloatTensor of size 7x5]

&gt;&gt;&gt;
&gt;&gt;&gt; X @ b

 0.0529
 0.0751
 0.4777
-0.6759
-2.1489
-1.1463
-0.2720
[torch.FloatTensor of size 7x1]

&gt;&gt;&gt; # check result two ways
... X @ b - y

1.00000e-07 *
  0.0000
  0.0000
  0.2980
  0.0000
 -2.3842
  0.0000
  0.0000
[torch.FloatTensor of size 7x1]

&gt;&gt;&gt; X @ b / y

 1.0000
 1.0000
 1.0000
 1.0000
 1.0000
 1.0000
 1.0000
[torch.FloatTensor of size 7x1]

&gt;&gt;&gt;
&gt;&gt;&gt; # try another random vector
... c = torch.randn(5,1)
&gt;&gt;&gt; # multiplication by this X always gives a vector proportional to y
... X @ c / y

-0.2834
-0.2834
-0.2834
-0.2834
-0.2834
-0.2834
-0.2834
[torch.FloatTensor of size 7x1]
</code></pre>
<p>I will say, however, that I don’t really understand what you are trying<br>
to do.  There’s a lot of freedom in the matrix <code>X</code> – in fact, probably too<br>
much freedom – so I suspect that what you are trying to here isn’t<br>
really what you want to be doing.</p>
<p>Good luck!</p>
<p>K. Frank</p>

username: Raphikowski
cooked: <p>Dear Frank,</p>
<p>thank you so much for your efforts. I think you are right in saying that what I am trying to do is a dead end <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
Best regards!</p>