How to efficiently and randomly produce an orthonormal matrix?

111414: <p>The following code can produce an orthonormal matrix for me:</p>
<pre><code class="lang-python">import numpy as np
from scipy.linalg import orth

def get_matrix(M, N):
    Phi = np.random.randn(N, N).astype(np.float32)
    return orth(Phi)[:M]

Phi = get_matrix(10, 10)
print(np.matmul(Phi, np.transpose(Phi, [1,0])))  # should be very close to identity matrix
</code></pre>
<p>I want to use it in each network <code>forwarding</code> process, during training, but the above implementation with <code>numpy</code> and <code>scipy</code> may be too slow for me.</p>
<p>How to implement the above function with high efficiency by using PyTorch?</p>

KFrank: <p>Hi Bin!</p>
<aside class="quote no-group" data-username="111414" data-post="1" data-topic="153609" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111414/48/31166_2.png" class="avatar"> 111414:</div>
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">def get_matrix(M, N):
    Phi = np.random.randn(N, N).astype(np.float32)
    return orth(Phi)[:M]
</code></pre>
</blockquote>
</aside>
<p>You can use pytorch’s <code>svd()</code> to do what I am pretty sure is the same<br>
thing as <code>scipy.linalg.orth()</code> is doing:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.11.0'
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt; gaus = torch.randn (5, 5)
&gt;&gt;&gt; svd = torch.linalg.svd (gaus)
&gt;&gt;&gt; orth = svd[0] @ svd[2]
&gt;&gt;&gt; orth
tensor([[-0.5558, -0.6665, -0.2910, -0.1637, -0.3679],
        [-0.1533,  0.2271, -0.8099,  0.4482,  0.2612],
        [ 0.5413, -0.3854,  0.0067,  0.6283, -0.4044],
        [ 0.1234, -0.5894,  0.0896,  0.0411,  0.7923],
        [ 0.5995, -0.0906, -0.5014, -0.6131, -0.0722]])
&gt;&gt;&gt; orth @ orth.T
tensor([[ 1.0000e+00,  5.4137e-08,  1.2895e-07,  1.8028e-07,  7.5459e-08],
        [ 5.4137e-08,  1.0000e+00, -1.2912e-07,  1.9165e-07,  2.2009e-07],
        [ 1.2895e-07, -1.2912e-07,  1.0000e+00,  2.5904e-08, -3.9519e-08],
        [ 1.8028e-07,  1.9165e-07,  2.5904e-08,  1.0000e+00, -1.5845e-08],
        [ 7.5459e-08,  2.2009e-07, -3.9519e-08, -1.5845e-08,  1.0000e+00]])
</code></pre>
<p>I believe* (but it would be worth double checking) that the generated<br>
matrix, <code>orth</code>, is randomly distributed over the space of orthonormal<br>
matrices (with respect to the standard invariant measure).</p>
<p>If you don’t want a square matrix, you can drop some rows or columns,<br>
as in your <code>get_matrix()</code> function.</p>
<p>*)  See, for example, this answer on stackoverflow:</p>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/38426349/how-to-create-random-orthonormal-matrix-in-python-numpy/54307312#54307312">
  <header class="source">

      <a href="https://stackoverflow.com/questions/38426349/how-to-create-random-orthonormal-matrix-in-python-numpy/54307312#54307312" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/5549394/zing-lee" target="_blank" rel="noopener nofollow ugc">
    <img alt="Zing Lee" src="https://www.gravatar.com/avatar/dcade8a7005ce19c46f0893b170a1cb0?s=256&amp;d=identicon&amp;r=PG&amp;f=y&amp;so-version=2" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/38426349/how-to-create-random-orthonormal-matrix-in-python-numpy/54307312#54307312" target="_blank" rel="noopener nofollow ugc">How to create random orthonormal matrix in python numpy</a>
</h4>

<div class="tags">
  <strong>python, numpy, linear-algebra, orthogonal</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stackoverflow.com/users/5549394/zing-lee" target="_blank" rel="noopener nofollow ugc">
    Zing Lee
  </a>
  on <a href="https://stackoverflow.com/questions/38426349/how-to-create-random-orthonormal-matrix-in-python-numpy/54307312#54307312" target="_blank" rel="noopener nofollow ugc">11:30AM - 22 Jan 19 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Best.</p>
<p>K. Frank</p>

111414: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="153609">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p><code>torch.linalg.svd</code></p>
</blockquote>
</aside>
<p>Thank you very much. My question is addressed by your answer.</p>