[LINEAR ALGEBRA] [QUESTION] Inverse

mathematicalmichael: <p>Quick question about an implementation choice (didn’t feel appropriate to raise on github)<br>
Psuedo-inverses <em>are</em> proper inverses when a matrix is square + full rank.<br>
It seems like the implementation in torch isn’t doing anything to discover rank and so can’t fall back on classical inverse algorithms for efficiency.</p>
<p>I am playing with this example:</p>
<pre><code class="lang-auto">import torch
nrows = 10000
ncols = 10000
torch.inverse(torch.rand(nrows,ncols).cuda())
</code></pre>
<pre><code class="lang-auto">
CPU times: user 3.8 s, sys: 141 ms, total: 3.94 s
Wall time: 927 ms
</code></pre>
<p>swapping that last line for <code>torch.pinverse(torch.rand(nrows,ncols).cuda())</code>  yields</p>
<pre><code class="lang-auto">CPU times: user 2min 17s, sys: 1.95 s, total: 2min 19s
Wall time: 20.7 s
</code></pre>
<p>So, even with added overhead to discover rank with something like Cholesky (for which it seems an implementation exists), large speedups can be gained.</p>
<p>What’s going on under the hood with pseudo-inverse and why is it taking so much longer? mat-mult is sub-second, classical inversion for full-rank operators is sub-second, so even manual-construction wins out:</p>
<pre><code class="lang-auto">A = torch.rand(nrows,ncols).cuda()
torch.mm(torch.inverse(torch.mm(A.t(), A)), A.t())
</code></pre>
<pre><code class="lang-auto">CPU times: user 3.67 s, sys: 167 ms, total: 3.83 s
Wall time: 1.09 s
</code></pre>
<p>Moreover, sometimes <code>pinverse</code> runs out of memory, but the latter won’t (a kernel restart allows <code>pinverse</code> to run again), and the solutions do seem to agree on average:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8ee0d76d6a3c16844509347100ae395b55949b25.png" data-download-href="https://discuss.pytorch.org/uploads/default/8ee0d76d6a3c16844509347100ae395b55949b25" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8ee0d76d6a3c16844509347100ae395b55949b25_2_690x377.png" alt="image" data-base62-sha1="knXwkJQxRly9fziFXYLTmr6oBGB" width="690" height="377" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8ee0d76d6a3c16844509347100ae395b55949b25_2_690x377.png, https://discuss.pytorch.org/uploads/default/original/3X/8/e/8ee0d76d6a3c16844509347100ae395b55949b25.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/e/8ee0d76d6a3c16844509347100ae395b55949b25.png 2x" data-dominant-color="F3F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">787×430 42.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

albanD: <p>Hey,</p>
<p>One thing to be careful of here is that the CUDA api is asynchronous. So without the appropriate <code>torch.cuda.synchronize()</code> calls, you shoud not read too much into these timings.</p>
<p>Still you can see <a href="https://github.com/pytorch/pytorch/blob/6debc28964872e26724764e213c40db366327f95/aten/src/ATen/native/cuda/BatchLinearAlgebra.cu#L693-L714">here</a> that we use LU factorization for the inverse and delegate all the heavy lifting to magma.<br>
The pinverse is <a href="https://github.com/pytorch/pytorch/blob/6debc28964872e26724764e213c40db366327f95/aten/src/ATen/native/LinearAlgebra.cpp#L83-L98">here</a> on the other hand and uses svd (done by magma as well). Note that it needs to handle batches of inputs.</p>
<blockquote>
<p>Quick question about an implementation choice</p>
</blockquote>
<p>Definitely open to discussions!</p>

mathematicalmichael: <p>thanks so much for the clarification. I’m a maths person who has only used CPUs to date, finally got a GPU to play around with. So, I’m not exactly sure where those calls should go.</p>
<p>The differences in algos though does explain <em>a lot</em> of the difference between my one-liner and the psuedo-inverse method; I really appreciate the linking to the source files, as it’s a large library to sort through!</p>
<p>A cholesky-based inverse may provide some major gains; c.f matlab’s benchmarks in this (admittedly decade old, but still relevant I think) paper: <a href="https://arxiv.org/pdf/0804.4809.pdf" rel="nofollow noopener">https://arxiv.org/pdf/0804.4809.pdf</a></p>

mathematicalmichael: <p>correct me if I’m wrong, but as far as I understand, deep-learning networks induce operators that are very ill-conditioned. So, my assumption is that <code>pinverse</code> is a often-called function during training. Is that the case? If not, then I’m not so sure how helpful work on improving it would be.</p>

albanD: <p>I wouldn’t say it is a very commonly used operator in general for neural nets. But it is useful and we do want make sure we’re not making obvious mistake here.<br>
That being said, advanced users do have access to all the “low level” functions like lu, svd, getr etc and can use the one they know fits best their use case.</p>
<blockquote>
<p>So, even with added overhead to discover rank</p>
</blockquote>
<p>Considering that we have both usecases, what would be the overhead here of checking the rank for people that have known non-invertible matrices?</p>

mathematicalmichael: <p>good question. out of curiosity, how does the development team think of overhead? Would numerical tests suffice (perhaps calling <code>.cpu()</code> to collect results at the end of my operations to account for the async?), or are you looking for a Big-O analysis for rank reveal + LU over straight SVD? I’d need to look into the specifics of the magma implementation for that (well, frankly, I have some friends in the department who’d be better suited for that).</p>

albanD: <blockquote>
<p>perhaps calling <code>.cpu()</code> to collect results at the end of my operations to account for the async?</p>
</blockquote>
<p>You can do something like:</p>
<pre><code class="lang-auto">torch.cuda.synchronize()
start = time.time()

# your_op

torch.cuda.synchronize()
elapsed = time.time() - start
</code></pre>
<blockquote>
<p>how does the development team think of overhead?</p>
</blockquote>
<p>I am not working on perfs explicitly so the definitive answer will definitely be given by the relevant people on github <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"><br>
But even though the Big-O notation is a good hint, I think actual runtime will be the most important.<br>
Also such choice can depend on the input size, batch size, etc.</p>
<p>cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> what do you think?</p>

mathematicalmichael: <p>Thanks!<br>
Pretty similar benchmark.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2ea1e3af41a79f6219392f20c25b0a4d2ba211d.png" alt="image" data-base62-sha1="u5PRB4nLacuTKKt5gUZo87viJwF" width="596" height="498"></p>
<p>Card: RTX2080S, don’t know how this may differ on better hadware. But the stark difference is a clue for an opportunity to improve performance out of the box. I can do some more numerical studies and see if I can get insight to the theoretical cost question</p>

albanD: <p>That sounds good.</p>
<p>After a quick offline chat with Vitaly, I think we would be happy to add some heuristics here.<br>
Unfortunately, my linear algebra is a bit rusty but if you could open an issue on github with some details on the heuristic you want to use (and a quick idea why). And a script that can be used to get a first idea of the impact on runtime both for full rank, not full rank matrices and ill conditioned matrices.<br>
I think we can then build on that to make sure that the general penalty in time is small enough for the benefit we get.</p>

VitalyFedyunin: <p>You can quite update pinverse implementation as it is easy to extract rank from <code>svd</code> calls results here <a href="https://github.com/pytorch/pytorch/blob/6debc28964872e26724764e213c40db366327f95/aten/src/ATen/native/LinearAlgebra.cpp#L94" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/6debc28964872e26724764e213c40db366327f95/aten/src/ATen/native/LinearAlgebra.cpp#L94</a></p>

mathematicalmichael: <p>that sounds good. if we can agree on a good benchmarking test, I can devote a little time to experimenting with it.</p>

mathematicalmichael: <p>while I don’t often code in c++, I can totally follow along with that snippet based on the math (also a testament to well-written code).</p>
<p>however, while SVD is rank-revealing, I think what I’d propose is to avoid the cost of computing it altogether potentially, since that seems to be the bottleneck. My demo was just on full-rank operators, and that SVD implementation you linked is more stable for ill-conditioned matrices.</p>