Pytorch torch.linalg.svd returning U and V^T, which are not orthogonal 

username: Bhartendu_Kumar
cooked: <p>Using U,S, VT = torch.linalg.svd(M), the matrix ‘M’ is large, so I am getting the matrices U and VT as non orthogonal. When I compute torch.norm(torch.mm(matrix, matrix.t()) - identity_matrix)) its ~0.004 and also when I print M.M^T, the diagonal entries are not 1, rather 0.2 or 0.4 and non diagonals are not 0, but ~0.0023. IS there a way to get SVD with orthogonal U and V^T ?</p>
<p>But the singular values i.e. diagonal elements of S are near to 1 only.</p>
<pre><code class="lang-auto">matrix = torch.randn(4096, 4096) 
u, s, vh = torch.linalg.svd(matrix) 
matrix = torch.matmul(u, vh)
print('norm ||WTW - I||: ',torch.norm(torch.mm(matrix, matrix.t()) - torch.eye(matrix.shape[0]))) 
print(matrix)
</code></pre>
<p>Changing precision to float64 helped to make ||M*M^T -I|| order of 1E-13,  also I am using “gesvd” that uses QR decomposition and more stable:</p>
<pre><code class="lang-auto">u, s, vh =  torch.linalg.svd(matrix, driver='gesvd')
</code></pre>
<p>but I am cautious that will it work fine or will  ill-conditioned matrix fail this? So is there a way to get stable SVD in pytorch?</p>

username: KFrank
cooked: <p>Hi Bhartendu!</p>
<aside class="quote no-group" data-username="Bhartendu_Kumar" data-post="1" data-topic="169468" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bhartendu_kumar/48/56092_2.png" class="avatar"> Bhartendu_Kumar:</div>
<blockquote>
<pre><code class="lang-auto">print('norm ||WTW - I||: ',torch.norm(torch.mm(matrix, matrix.t()) - torch.eye(matrix.shape[0]))) 
</code></pre>
</blockquote>
</aside>
<p>You are testing the orthogonality of your original random matrix, <code>matrix</code>.<br>
You should be testing the orthogonality of the result of the decomposition,<br>
<code>u</code> and <code>vh</code>, instead.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Bhartendu_Kumar
cooked: <p><a class="mention" href="/u/kfrank">@KFrank</a> Thanks, that was typo in copy pasting the code. I checked orthogonality of matrix = u *vh and individually of u and vh. But not getting them as orthogonal.</p>

username: KFrank
cooked: <p>Hi Bhartendu!</p>
<aside class="quote no-group" data-username="Bhartendu_Kumar" data-post="3" data-topic="169468" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bhartendu_kumar/48/56092_2.png" class="avatar"> Bhartendu_Kumar:</div>
<blockquote>
<p>I checked orthogonality of matrix = u *vh and individually of u and vh. But not getting them as orthogonal.</p>
</blockquote>
</aside>
<p>It works for me:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.13.0
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2023)
&gt;&gt;&gt;
&gt;&gt;&gt; matrix = torch.randn (4096, 4096)
&gt;&gt;&gt; u, s, vh = torch.linalg.svd (matrix)
&gt;&gt;&gt; uut = torch.matmul (u, u.t())
&gt;&gt;&gt; print ('norm ||uut - I||: ', torch.norm (uut - torch.eye (uut.shape[0])))
norm ||uut - I||:  tensor(0.0003)
&gt;&gt;&gt; vvt = torch.matmul (vh, vh.t())
&gt;&gt;&gt; print ('norm ||vvt - I||: ', torch.norm (vvt - torch.eye (vvt.shape[0])))
norm ||vvt - I||:  tensor(0.0003)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>