Svd source code in torch

andy03: <p>Hi,</p>
<p>I was looking at torch’s codebase and could not exactly find where is it’s SVD implementation.specificallly <code>torch.linalg.svd()</code>.</p>
<p>i was able to find some part of it in here : <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/BatchLinearAlgebra.cpp" class="inline-onebox" rel="noopener nofollow ugc">pytorch/aten/src/ATen/native/BatchLinearAlgebra.cpp at main · pytorch/pytorch · GitHub</a></p>
<p>but not really sure if that is the complete source.</p>

KFrank: <p>Hi Andy!</p>
<aside class="quote no-group" data-username="andy03" data-post="1" data-topic="223367" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andy03/48/78321_2.png" class="avatar"> andy03:</div>
<blockquote>
<p>could not exactly find where is it’s SVD implementation.</p>
</blockquote>
</aside>
<p>Short story:  You might look at <a href="https://netlib.org/lapack/explore-html/df/d22/group__gesdd.html" rel="noopener nofollow ugc">LAPACK’s</a> SVD implementation, maybe for certain<br>
code paths and data types, <a href="https://netlib.org/lapack/explore-html/df/d22/group__gesdd_ga58d8bb0bce9bb2d6224d1955cef7462c.html#ga58d8bb0bce9bb2d6224d1955cef7462c" rel="noopener nofollow ugc">LAPACK: zgesdd.</a></p>
<p>Disclaimer:  I do not have a good understanding of pytorch’s source code and I<br>
have a very difficult time navigating the source tree.</p>
<p>However, in the file you linked to we find, for example:</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">template&lt;&gt; void lapackSvd&lt;c10::complex&lt;double&gt;, double&gt;(char jobz, int m, int n, c10::complex&lt;double&gt; *a, int lda,
                                  double *s, c10::complex&lt;double&gt; *u, int ldu, c10::complex&lt;double&gt; *vt, int ldvt, c10::complex&lt;double&gt; *work, int lwork, double *rwork, int *iwork, int *info) {
  zgesdd_(&amp;jobz, &amp;m, &amp;n, reinterpret_cast&lt;std::complex&lt;double&gt;*&gt;(a), &amp;lda, s, reinterpret_cast&lt;std::complex&lt;double&gt;*&gt;(u), &amp;ldu,
          reinterpret_cast&lt;std::complex&lt;double&gt;*&gt;(vt), &amp;ldvt, reinterpret_cast&lt;std::complex&lt;double&gt;*&gt;(work), &amp;lwork, rwork, iwork, info);
}
</code></pre>
<p>and:</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">// gesdd
extern "C" void zgesdd_(char *jobz, int *m, int *n, std::complex&lt;double&gt; *a, int *lda,
                        double *s, std::complex&lt;double&gt; *u, int *ldu, std::complex&lt;double&gt; *vt, int *ldvt, std::complex&lt;double&gt; *work, int *lwork, double *rwork, int *iwork, int *info);
</code></pre>
<p>From this I conclude that pytorch – at least for some code paths – is using<br>
LAPACK’s SVD implementation.</p>
<p>I would speculate that pytorch is linked to some external LAPACK library, but it’s<br>
also possible that pytorch keeps a copy of the LAPACK source code in its source<br>
tree.  I would also speculate that LAPACK uses different implementations and / or<br>
compiler settings for different architectures.</p>
<p>Again me providing possibly unreliable information:  I believe nvidia provides<br>
a “cuBLAS” library for “BLAS” functionality on its gpus.  This is one level below<br>
LAPACK, so the gpu efficiency may come from having a “generic” LAPACK SVD<br>
call down into the cuBLAS gpu implementation.</p>
<p>In general the pytorch code is multi-layered, with a chain of many function calls<br>
before the python code you call gets dispatched to the actual code that actually<br>
performs various floating-point operations.</p>
<p>Best.</p>
<p>K. Frank</p>

andy03: <p>Hi KFrank,</p>
<p>yeah I did a bit of research and, found that internally torch is probably calling the LAPACK implementations, I was interested in the underlying algorithms actually, got to know that they use a combination of jacobi rotations with QR iteration methods.</p>
<p>thank you for your detailed answer!</p>