How can I computes the pseudoinverse of a matrix with mixed-precision in PyTorch

username: S_M
cooked: <p>Hi all, I am trying to compute the pseudoinverse (Moore-Penrose inverse) of a matrix using tensor.pinverse() with mixed-precision mode. But I got this error:</p>
<p>RuntimeError: linalg_pinv(Half{[10, 10]}): expected a tensor with 2 or more dimensions of float, double, cfloat or cdouble types</p>
<p>Could someone please guide how can I fix this problem.<br>
Thanks in advance</p>

username: albanD
cooked: <p>Hi,</p>
<p>I guess we don’t have this implemented atm. Such functions are usually pretty sensitive to numerical precision so might be very unstable with fp16 inputs.<br>
Would it work for you to convert the input to fp32 before calling pinv? If not could you open an issue on github asking for support on that function?</p>

username: S_M
cooked: <p>Hi <a class="mention" href="/u/alband">@albanD</a>, thanks for your advice. I used this block for my training</p>
<pre><code>with torch.autocast(device_type='cuda', dtype=torch.float16, enabled=True):
    my code...
</code></pre>
<p>when tensor.pinverse() was used the tensor was converted to fp32 but I got this error during backward</p>
<pre><code>self.scaler.scale(loss).backward()
</code></pre>
<p>File “/usr/local/lib/python3.9/dist-packages/torch/_tensor.py”, line 307, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)<br>
File “/usr/local/lib/python3.9/dist-packages/torch/autograd/<strong>init</strong>.py”, line 154, in backward<br>
Variable._execution_engine.run_backward(<br>
RuntimeError: Function ‘EigBackward0’ returned nan values in its 0th output.</p>
<p>How can I handle this error?</p>

username: albanD
cooked: <p>I’m afraid this is exactly the kind of issues you would get from what I was saying above about these functions being very sensitive to numerical precision.<br>
I would first check that it runs fine without autocast in fp32 precision. And if it does, you might have to only autocast other pieces of your network and not this one.</p>

username: S_M
cooked: <p>I need to compute pseudoinverse of matrix in fp16 mode because I have a limited GPU to run my network so when I use fp32 I get cuda out of memory error.</p>

username: ptrblck
cooked: <p>Linear algebra operations are often sensitive to the numerical precision (which is why e.g. TF32 is not used in <code>torch.lingalg</code>) and the failure might thus be expected. If you don’t have enough device memory you might need to move the tensor to the CPU instead.</p>

username: S_M
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> when I move to CPU I get “Killed” error.</p>

username: ptrblck
cooked: <p>Your process is most likely also killed by the OS as it’s running out of host RAM so your system also doesn’t seem to have enough RAM to perform the operation on the CPU.</p>

username: S_M
cooked: <p>For such reasons I need to reduce GPU memory usage. This error occurs even in google Colab.</p>

username: Lezcano
cooked: <p>A few things:</p>
<ul>
<li>
<code>pinv</code> is quite literally the most unstable operation in the codebase. In almost all scenarios you don’t want to compute the pseudoinverse directly, but rather you want to compute <code>lstsq</code> (although we don’t have support for non-full-rank matrices on GPU) or <code>solve</code>.</li>
<li>If you <strong>really</strong> need to use <code>pinv</code>, there is no way it’s going to compute accurate results in half precision. It will already struggle in <code>float32</code>. In this case, I would suggest either making your model smaller or getting an instance with a GPU wtih x2 . Now, as Piotr mentioned, if you can’t run your modle on CPU either, it looks like there are some fundamental issues with your model, as it may be using <strong>much</strong> more memory than you think it is. If this is the case, running it in half precision will not solve these issues.</li>
</ul>