Find Only Largest Eigenvector of Symmetric Matrix

Jack_Rolph: <p>I am looking to find only the primary eigenvector of a real, positive-definite symmetric 20x2000x2000 tensor:</p>
<pre><code class="lang-auto">K = torch.random.randn(20, 2000, 2000)
K_sym = K+K.transpose(1,2)/2
eig, ev = torch.symeig(K_sym, eigenvectors=True)
ev = ev[:,1999,:]

ev.shape

&gt; torch.Size[20, 2000]

</code></pre>
<p>This implementation is very slow, even on the GPU,</p>
<p>Can anyone suggest a faster way of finding the first eigenvector?</p>
<p>For example, power iteration may be a solution. I have based this on the solution found in <code>torch.nn.utils.spectral_norm</code> :</p>
<pre><code class="lang-auto">def TopEigenvector(self, K, n_power_iterations=400, dim=1, eps=1e-12):
        v = torch.ones(K.shape[0],K.shape[1],1).to(self.device)
        with torch.no_grad():
            for _ in range(n_power_iterations):
                v = normalize(torch.bmm(K, v),dim=dim, eps=eps, out=v)
                if(n_power_iterations &gt; 0):
                    v = v.clone(memory_format=torch.contiguous_format)
        return v
</code></pre>
<p>However, this relies on the dreaded for-loop, and also does not have a gradient for obvious reasons.</p>
<p>I have tested backprop using this function and, so far, it does not complain. Is there, however, a way that is gradient friendly?</p>
<p>EDIT:</p>
<p>The function is differentiable using the following:</p>
<pre><code class="lang-auto">    def TopEigenvector(self, K, n_power_iterations=400, dim=1):
        v = torch.ones(K.shape[0],K.shape[1],1).to(self.device)
        for _ in range(n_power_iterations):
            m = torch.bmm(K, v)
            n = torch.norm(m, dim=1).unsqueeze(1)
            v = m/n
        return v
</code></pre>
<p>Slow, but way faster than the alternative.</p>

KFrank: <p>Hi Jack!</p>
<aside class="quote no-group quote-modified" data-username="Jack_Rolph" data-post="1" data-topic="85185" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jack_rolph/48/20306_2.png" class="avatar"> Jack_Rolph:</div>
<blockquote>
<p>I am looking to find only the primary eigenvector of a real, positive-definite symmetric 20x2000x2000 tensor:<br>
…<br>
Can anyone suggest a faster way of finding the first eigenvector?</p>
</blockquote>
</aside>
<p>You might find that the <a href="https://en.wikipedia.org/wiki/Lanczos_algorithm" rel="noopener nofollow ugc">Lanczos method</a> runs faster.</p>
<p>Also, it may even make sense to bump over into scipy (a numpy<br>
relative), and use an established implementation, for example,<br>
<a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.eigs.html" rel="noopener nofollow ugc">scipy.sparse.linalg.eigs</a>.  The careful implementation <em>might</em> beat<br>
pytorch tensors on the gpu.</p>
<p>(I’m pretty sure that it becomes equivalent to Lanczos for your real<br>
SPD use case.)</p>
<p>As for getting autograd / <code>backward()</code> to work, I don’t have anything<br>
useful to say.</p>
<p>[Note to the Pytorch Mavens:  I think I’ve seen questions like this<br>
before.  I don’t personally have a pytorch use case for the top-k<br>
eigenvectors, but is this common enough that it might make sense<br>
to implement an autograd / gpu-friendly top-k eigenvector function?]</p>
<p>Best.</p>
<p>K. Frank</p>

Jack_Rolph: <p>Hi Frank,</p>
<p>Thanks for the informative answer!</p>
<p>For now, this method seems to solve the speed issue - it’s blazingly fast by comparison. I would be happy using scipy, but how would I ensure the gradients are calculated correctly? Does there exist a backprop method for such things?</p>

KFrank: <p>Hi Jack!</p>
<aside class="quote no-group" data-username="Jack_Rolph" data-post="3" data-topic="85185" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jack_rolph/48/20306_2.png" class="avatar"> Jack_Rolph:</div>
<blockquote>
<p>I would be happy using scipy, but how would I ensure the gradients are calculated correctly?</p>
</blockquote>
</aside>
<p>If you use scipy (or numpy, etc.)  you would have to write your own<br>
custom <code>backward()</code> for your operation in order to get autograd to<br>
work.  For this kind of eigenvalue computation, writing <code>backward()</code><br>
would involve some significant analysis and numerical programming.<br>
(It should, be doable – just a lot more than a one-liner.)</p>
<p>Now that you’ve gotten rid of the  <code>with torch.no_grad():</code><br>
block I <em>think</em> that the version you posted above should give you<br>
legitimate gradients.  (I’m not really clear on how autograd works<br>
with loops like this, but I don’t see any reason it shouldn’t.)</p>
<p>If the loop conflicts with autograd (or if you think the loop is hurting<br>
performance), you could <em>unwrap</em> the loop.  (Although now that<br>
you’ve increased the default value of <code>n_power_iterations</code><br>
to <code>400</code> from <code>5</code>, fully unwrapping the loop might be unattractive.)</p>
<p>Best.</p>
<p>K. Frank</p>