Computing the k-largest singular values/vector

mzimmer: <p>Hello everybody! I have the following problem: Given an arbitrary non-square matrix W, i need to get the k singular vector pairs corresponding to the k largest singular values.</p>
<p>An obvious way of doing so is to just compute the entire SVD and then only take the k first. This is however very inefficient when k is small, say k = 1. Is there any way to get these vector pairs more efficiently?</p>
<p>Thanks a lot!</p>

KFrank: <p>Hi Max!</p>
<aside class="quote no-group" data-username="mzimmer" data-post="1" data-topic="147658" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mzimmer/48/31510_2.png" class="avatar"> mzimmer:</div>
<blockquote>
<p>Given an arbitrary non-square matrix W, i need to get the k singular vector pairs corresponding to the k largest singular values.</p>
</blockquote>
</aside>
<p>You should be able to leverage <a href="https://pytorch.org/docs/1.11/generated/torch.lobpcg.html" rel="noopener nofollow ugc">torch.lobpcg()</a>, which calculates the k<br>
largest eigenvalues of a symmetric-positive-definite matrix.</p>
<p>Let’s assume that <code>W</code> is an <code>m x n</code> matrix, with <code>m &lt;= n</code>.  Then <code>W @ W.T</code><br>
will be an <code>m x m</code> symmetric-positive-semidefinite matrix.  I imagine that<br>
you can fudge this by adding a small epsilon to the diagonal, if needed*.</p>
<p><code>torch.lobpcg()</code> will give you the top k eigenvalues of <code>W @ W.T</code>, which<br>
are the squares of the top k singular values of <code>W</code> (In fact, this is a<br>
common definition of the singular values.), together with the associated<br>
eigenvectors.</p>
<p>*)  I haven’t tried it, but my intuition suggests that if none of the k singular<br>
values of <code>W</code> that you ask for are zero, the <code>lobpcg()</code> algorithm might still<br>
work without the epsilon fudge, even if <code>W</code> does have some singular values<br>
that are zero.</p>
<p>Good luck!</p>
<p>K. Frank</p>

mzimmer: <p>Hey <a class="mention" href="/u/kfrank">@KFrank</a>, thanks for the fast reply. This was actually the first thing I tried, but it seems to be relatively slow, but at least the answer seems to be exact. I resorted to a power iteration approach which gives me the first singular-pair in reasonable time, however it would be nice to use pytorch-optimized functions only, I still rely on a for loop (which is by the way the same approach the spectral norm does in pytorch).</p>
<p>I wonder whether its possible to e.g. modify the torch SVD code directly to fit my needs, however it seems like its in C and not in Python/Pytorch.</p>

KFrank: <p>Hi Max!</p>
<aside class="quote no-group" data-username="mzimmer" data-post="3" data-topic="147658" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mzimmer/48/31510_2.png" class="avatar"> mzimmer:</div>
<blockquote>
<p>it seems to be relatively slow</p>
</blockquote>
</aside>
<p>Does it seem slower than it ought to be, or is it just slower than what<br>
you would like for your use case?  (In general, pytorch does a good job<br>
with these sorts of things, but I wouldn’t rule out a bug or unnecessary<br>
inefficiency, and if there is one, it would be worth tracking down.)</p>
<aside class="quote no-group">
<blockquote>
<p>I wonder whether its possible to e.g. modify the torch SVD code directly to fit my needs</p>
</blockquote>
</aside>
<p>There are two probable issues with doing so:</p>
<p>First, the algorithm for calculating a small number of singular values (or,<br>
analogously, eigenvalues) is rather different than that for performing a full<br>
singular-value decomposition (or, analogously, eigendecomposition).</p>
<p>So starting with pytorch’s SVD code would likely lead you to writing an<br>
algorithm that would not be efficient when calculating just a few singular<br>
values.</p>
<p>Second, do you need to backpropagate through your top few singular<br>
vectors?  If so, I expect that you would also have to implement the gradient<br>
(<code>backward()</code>) companion to your singular-vector calculation.  This is<br>
because I doubt very much that pytorch’s SVD algorithm is written using<br>
differentiable pytorch tensor operations.  Therefore neither pytorch’s SVD<br>
algorithm, nor your slimmed-down “top-k” version will get autograd “for<br>
free.”  Instead, you would have to implement the gradient computation<br>
explicitly.</p>
<p>Best.</p>
<p>K. Frank</p>

mzimmer: <aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="147658">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Does it seem slower than it ought to be, or is it just slower than what<br>
you would like for your use case?</p>
</blockquote>
</aside>
<p>It is significantly slower than using a power iteration (which is approximative though). I was thinking of integrating the methods of Scipy into Pytorch, however they seem to be using the same algorithm in this naive way, i.e. by transforming matrix W into symmetric pos-def. matrix W.t()*W and the computing the eigenpairs.</p>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="147658">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>First, the algorithm for calculating a small number of singular values (or,<br>
analogously, eigenvalues) is rather different than that for performing a full<br>
singular-value decomposition (or, analogously, eigendecomposition).</p>
</blockquote>
</aside>
<p>You are probably right. Right now I am using this in a research context, so I will be fine by using the full SVD as implemented in Pytorch, and then selecting only the top-k entries. This obviously does not yield the speedup, but at least gives me correct results.</p>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="147658">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Second, do you need to backpropagate through your top few singular<br>
vectors?</p>
</blockquote>
</aside>
<p>No, I don’t need gradients at all, actually the SVD is performed on the gradient of my parameters.</p>
<p>Thanks for your help!</p>