Torch.inverse vs torch.cholesky_inverse - when to use which?

username: AlphaBetaGamma96
cooked: <p>Hi All,</p>
<p>I just have a quick question regarding inverting matrices, is there any convention on which one to used? Is it just to use <code>torch.cholesky_inverse</code> for PSD matrices and <code>torch.inverse</code> for all other cases? Because I’ve used both for a PSD matrix and they give different results? Is this just numerical noise?</p>
<p>Thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: KFrank
cooked: <p>Hi Alpha!</p>
<aside class="quote no-group" data-username="AlphaBetaGamma96" data-post="1" data-topic="136080" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3da27b/48.png" class="avatar"> AlphaBetaGamma96:</div>
<blockquote>
<p>Is it just to use <code>torch.cholesky_inverse</code> for PSD matrices</p>
</blockquote>
</aside>
<p>Yes, because (if you know your matrix is symmetric-positive-definite)<br>
you can use <code>cholesky_inverse()</code> and doing so will be faster.</p>
<aside class="quote no-group">
<blockquote>
<p>and <code>torch.inverse</code> for all other cases?</p>
</blockquote>
</aside>
<p>Yes, because in the non-SPD case <code>cholesky_inverse()</code> won’t work.</p>
<aside class="quote no-group">
<blockquote>
<p>Because I’ve used both for a PSD matrix and they give different results? Is this just numerical noise?</p>
</blockquote>
</aside>
<p>You don’t say how large the discrepancy is, so my default assumption<br>
is that you are seeing floating-point round-off error.  An easy test for<br>
this is to repeat the same computations using <code>float64</code> tensors.  If the<br>
discrepancy is reduced by several orders of magnitude, you are most<br>
likely seeing round-off error.</p>
<p>Best.</p>
<p>K. Frank</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a>,</p>
<p>Here’s an example,</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; 
&gt;&gt;&gt; A = torch.randn(3,3)
&gt;&gt;&gt; M = A@A.transpose(-2,-1)
&gt;&gt;&gt; 
&gt;&gt;&gt; M
tensor([[ 1.6559, -0.9484,  0.6780],
        [-0.9484,  1.2809, -0.0119],
        [ 0.6780, -0.0119,  0.5705]])
&gt;&gt;&gt; torch.cholesky_inverse(M)
tensor([[ 1.0665,  0.3376, -1.2417],
        [ 0.3376,  0.6097,  0.0284],
        [-1.2417,  0.0284,  3.0724]])
&gt;&gt;&gt; torch.inverse(M)
tensor([[ 5.9354,  4.3301, -6.9640],
        [ 4.3301,  3.9398, -5.0643],
        [-6.9640, -5.0643,  9.9240]])
&gt;&gt;&gt; 
&gt;&gt;&gt; M=M.double()
&gt;&gt;&gt; M
tensor([[ 1.6559, -0.9484,  0.6780],
        [-0.9484,  1.2809, -0.0119],
        [ 0.6780, -0.0119,  0.5705]], dtype=torch.float64)
&gt;&gt;&gt; torch.cholesky_inverse(M)
tensor([[ 1.0665,  0.3376, -1.2417],
        [ 0.3376,  0.6097,  0.0284],
        [-1.2417,  0.0284,  3.0724]], dtype=torch.float64)
&gt;&gt;&gt; torch.inverse(M)
tensor([[ 5.9354,  4.3301, -6.9640],
        [ 4.3301,  3.9398, -5.0643],
        [-6.9640, -5.0643,  9.9240]], dtype=torch.float64)
</code></pre>
<p>One thing I’ve just checked is multiplying the inverse by M to get back the identity matrix, but <code>torch.cholesky_inverse</code> doesn’t give back <code>I</code> but <code>torch.inverse</code> does.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; iM_c = torch.cholesky_inverse(M)
&gt;&gt;&gt; iM_i = torch.inverse(M)
&gt;&gt;&gt; 
&gt;&gt;&gt; iM_c @ M
tensor([[ 6.0391e-01, -5.6433e-01,  1.0679e-02],
        [ 3.1153e-17,  4.6053e-01,  2.3788e-01],
        [ 4.1231e-16,  1.1777e+00,  9.1056e-01]], dtype=torch.float64)
&gt;&gt;&gt; iM_i @ M
tensor([[ 1.0000e+00,  4.3730e-16,  5.4966e-16],
        [-2.7408e-16,  1.0000e+00,  2.3931e-16],
        [ 6.0420e-16, -6.4525e-16,  1.0000e+00]], dtype=torch.float64)
</code></pre>
<p>I assume I’m using <code>torch.cholesky_inverse</code> incorrectly and should be using <code>torch.inverse</code> instead?</p>
<p>Thank you for your help!</p>
<p>Edit: So, after re-reading the docs you need to apply <code>torch.cholesky</code> before and call <code>torch.cholesky_inverse</code> to that, rather than applying it directly! (I naively thought it was called internally, <a href="https://pytorch.org/docs/stable/generated/torch.cholesky_inverse.html?highlight=cholesky_inverse#torch.cholesky_inverse" class="inline-onebox" rel="noopener nofollow ugc">torch.cholesky_inverse — PyTorch 1.10.0 documentation</a> )</p>