Batched LU factorization of sparse tensors on GPU

romanodev: <p>Hi,</p>
<p>I need to solve many sparse linear systems using LU factorization (ideally on GPU). Is it possible within the current implementation of PyTorch? Thanks</p>

albanD: <p>You can use the <a href="https://pytorch.org/docs/stable/torch.html#torch.lu" rel="nofollow noopener"><code>torch.lu</code></a> function for that.</p>

romanodev: <p>Thanks. It seems that it does not work with sparse tensors (trying without batching):</p>
<p>import torch</p>
<p>i = torch.LongTensor([[0, 2], [2, 0], [1, 1]])<br>
v = torch.FloatTensor([3,      4,      5    ])<br>
A = torch.sparse.FloatTensor(i.t(), v, torch.Size([3,3])).cuda()</p>
<p>A_LU = torch.lu(A)</p>
<p>RuntimeError: Didn’t find kernel to dispatch to for operator ‘aten::_lu_with_info’. Tried to look up kernel for dispatch key ‘SparseCUDATensorId’. Registered dispatch keys are: [CUDATensorId, CPUTensorId, VariableTensorId]</p>

vincentqb: <p>I’m getting the same, even without cuda.</p>
<pre><code class="lang-auto">i = torch.LongTensor([[0, 1, 2], [0, 1, 2]])                                                                                                                                                                                                                                                                                                               
v = torch.FloatTensor([1,1,1])                                                                                                                                                                                                                                                                                                                             
t = torch.sparse.FloatTensor(i, v, torch.Size([3,3]))
t.to_dense()                                                                                                                                                                                                                                                                                         

tensor([[1., 0., 0.],
        [0., 1., 0.],
        [0., 0., 1.]])

t.lu()                       
                                                                                                                                                                                                                                                                                                                              
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-25-9f1f2db7f67d&gt; in &lt;module&gt;
----&gt; 1 t.lu()

~/anaconda3/envs/audio-built/lib/python3.7/site-packages/torch/tensor.py in lu(self, pivot, get_infos)
    294         r"""See :func:`torch.lu`"""
    295         # If get_infos is True, then we don't need to check for errors and vice versa
--&gt; 296         LU, pivots, infos = torch._lu_with_info(self, pivot=pivot, check_errors=(not get_infos))
    297         if get_infos:
    298             return LU, pivots, infos

RuntimeError: Didn't find kernel to dispatch to for operator 'aten::_lu_with_info'. Tried to look up kernel for dispatch key 'SparseCPUTensorId'. Registered dispatch keys are: [CPUTensorId, VariableTensorId]
</code></pre>

romanodev: <p>In your case, you will need the following code:</p>
<p>i = torch.LongTensor([[0, 1, 2], [0, 1, 2]])<br>
v = torch.FloatTensor([1,1,1])<br>
t = torch.sparse.FloatTensor(i, v, torch.Size([3,3]))<br>
a = t.to_dense()     <span class="hashtag">#this</span> is necessary</p>
<p>a.lu()</p>

vincentqb: <p>I just meant to confirm I also get such an error <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

romanodev: <p><a href="https://discuss.pytorch.org/u/bamos">bamos</a>: any thought? I noted you posted something similar in the past. Thanks.</p>

albanD: <p>I think the answer is that no we don’t have a LU factorization for sparse Tensors available. You will have to convert to a dense Tensor.<br>
If you are interested in contributing an implementation for LU factorization for sparse Tensor, we would be happy to help and accept it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

romanodev: <p>Thanks for your reply. I will look back into it once I have suitable bandwidth.</p>

MaximeBeauchamp: <p>Hi everyone,</p>
<p>Is there any other developments regarding this problem of linear algebra algorithms for sparse matrices ?</p>
<p>I think most of methods already implemented for instance in scipy or extensions such as scikit-sparse, are not yet implemented in Torch, for instance:</p>
<ul>
<li>Sparse Cholesky decomposition, see e.g.  <code>sksparse.cholmod</code>
</li>
<li>Solving linear systems Ax=b with A a sparse matrix</li>
<li>Product of two sparse matrices</li>
<li>…</li>
</ul>
<p>It could be very useful to allow for automatic differentiation of such operations in Torch.</p>
<p>Thanks</p>