Solving Ax=B for sparse tensors (preferably with backward)

username: Nikronic
cooked: <p>Hi,</p>
<h2><a name="problem-1" class="anchor" href="#problem-1"></a>Problem:</h2>
<p>Based on the issues on Github, PyTorch does not support <code>torch.solve</code> for sparse tensors (neither forward nor backward). The main issue is <code>runtime error: no stride</code>. I wonder is there any workarounds for any special case so I can fix my issue? Based on my experiments, there is no way that I can handle my problem by using dense matrices and also, I need backward compatibility too.</p>
<p>I do not know it is possible, but can I use a third-party libs such as <code>cupy</code> and still maintain computational graph and backward through it?</p>
<aside class="quote" data-post="1" data-topic="62141">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/76d3ee/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/rewriting-spdiags-spsolve/62141">Rewriting spdiags &amp; spsolve?</a> 
  </div>
  <blockquote>
    I’m trying to rewrite some legacy code and switch from scipy to pytorch and I’m having difficulties re-writing the spdiags and spsolve. 
How could I perform these operations on GPUs? 
A = spdiags(B.T,d,k,k)

A = A + A.T + spdiags(D.T, 0, k, k)

return spsolve(A, input)
  </blockquote>
</aside>

<aside class="onebox githubblob" data-onebox-src="https://github.com/Quansight-Labs/rfcs/blob/pearu/rfc0005/RFC0003-sparse-roadmap/SparseSupportState.md#linear-algebra-algorithms">
  <header class="source">

      <a href="https://github.com/Quansight-Labs/rfcs/blob/pearu/rfc0005/RFC0003-sparse-roadmap/SparseSupportState.md#linear-algebra-algorithms" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/Quansight-Labs/rfcs/blob/pearu/rfc0005/RFC0003-sparse-roadmap/SparseSupportState.md#linear-algebra-algorithms" target="_blank" rel="noopener nofollow ugc">Quansight-Labs/rfcs/blob/pearu/rfc0005/RFC0003-sparse-roadmap/SparseSupportState.md#linear-algebra-algorithms</a></h4>


      <pre><code class="lang-md">
This file is auto-generated, do not edit!

# The state of PyTorch tensor layouts support

The following table summarizes the state of PyTorch tensor layouts for
different PyTorch functions from the following namespaces:
- torch
- torch.nn.functional
- torch.sparse

|                              Section                               |                        strided@cpu                        |                      strided@cuda:1                       |                             sparse_coo@cpu                             |                           sparse_coo@cuda:1                            |
| :----------------------------------------------------------------- | :-------------------------------------------------------- | :-------------------------------------------------------- | :--------------------------------------------------------------------- | :--------------------------------------------------------------------- |
| &lt;a href="#tensor-constructors"&gt;Tensor constructors&lt;/a&gt;             | PASSED: 21, SKIPPED: 3                                    | PASSED: 20, SKIPPED: 4                                    | FAILED: 11, N/A: 2, PASSED: 7, SKIPPED: 4                              | FAILED: 12, N/A: 2, PASSED: 6, SKIPPED: 4                              |
| &lt;a href="#trigonometry-functions"&gt;Trigonometry functions&lt;/a&gt;       | PASSED: 78                                                | PASSED: 78                                                | FAILED: 71, PASSED: 7                                                  | FAILED: 71, PASSED: 7                                                  |
| &lt;a href="#arithmetics-functions"&gt;Arithmetics functions&lt;/a&gt;         | NOTIMPL: 1, PASSED: 36                                    | NOTIMPL: 1, PASSED: 36                                    | FAILED: 31, NOTIMPL: 1, PARTIAL: 2, PASSED: 3                          | FAILED: 31, NOTIMPL: 1, PARTIAL: 2, PASSED: 3                          |
| &lt;a href="#linear-algebra-algorithms"&gt;Linear Algebra algorithms&lt;/a&gt; | PASSED: 32                                                | FAILED: 2, PASSED: 30                                     | FAILED: 28, PASSED: 4                                                  | FAILED: 28, PASSED: 4                                                  |
| &lt;a href="#convolution-operations"&gt;Convolution operations&lt;/a&gt;       | PASSED: 9                                                 | PASSED: 9                                                 | FAILED: 9                                                              | FAILED: 9                                                              |
| &lt;a href="#matrix-functions"&gt;Matrix functions&lt;/a&gt;                   | NOTIMPL: 2, PASSED: 24                                    | NOTIMPL: 2, PASSED: 24                                    | FAILED: 24, NOTIMPL: 2                                                 | FAILED: 24, NOTIMPL: 2                                                 |
| &lt;a href="#general-array-functions"&gt;General array functions&lt;/a&gt;     | PASSED: 57                                                | PASSED: 57                                                | FAILED: 38, PASSED: 19                                                 | FAILED: 38, PASSED: 19                                                 |
</code></pre>



  This file has been truncated. <a href="https://github.com/Quansight-Labs/rfcs/blob/pearu/rfc0005/RFC0003-sparse-roadmap/SparseSupportState.md#linear-algebra-algorithms" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p><a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.spsolve.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.spsolve.html</a></p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/9674">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/9674" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/9674" target="_blank" rel="noopener nofollow ugc">The state of sparse Tensors</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2018-07-21" data-time="04:03:19" data-timezone="UTC">04:03AM - 21 Jul 18 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/weiyangfb" target="_blank" rel="noopener nofollow ugc">
          <img alt="weiyangfb" src="https://avatars.githubusercontent.com/u/38509346?v=4" class="onebox-avatar-inline" width="20" height="20">
          weiyangfb
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: sparse
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">This note tries to summarize the current state of sparse tensor in pytorch. It d<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">escribes important invariance and properties of sparse tensor, and various things need to be fixed (e.g. empty sparse tensor). It also shows some details of sparse operators.


## Semantics

### Construct a sparse tensor
```
# create a float sparse tensor in CPU
&gt;&gt;&gt; indices = torch.LongTensor([[0, 0, 1], [0, 1, 1]])
&gt;&gt;&gt; values = torch.FloatTensor([2, 3, 4])
&gt;&gt;&gt; sizes = [2, 2]
&gt;&gt;&gt; torch.sparse_coo_tensor(indices, values, sizes)
torch.sparse.FloatTensor of size (2,2) with indices:
tensor([[0, 0, 1],
        [0, 1, 1]])
and values:
tensor([2., 3., 4.])

# to_dense
&gt;&gt;&gt; torch.sparse_coo_tensor(indices, values, sizes).to_dense()
tensor([[2., 3.],
        [0., 4.]])

# in CUDA
&gt;&gt;&gt; torch.sparse_coo_tensor(indices, values, sizes, device=torch.device('cuda'))
torch.cuda.sparse.FloatTensor of size (2,2) with indices:
tensor([[0, 0, 1],
        [0, 1, 1]], device='cuda:0')
and values:
tensor([2., 3., 4.], device='cuda:0')
```

### More approaches in creating a sparse tensor
```
torch.sparse.FloatTensor()
torch.sparse.DoubleTensor()
torch.zeros(..., layout=torch.sparse_coo, ...)
```
Should we unify/reduce different approaches?


## Sparse representation

Currently, our sparse tensors are hybrid tensors, with a mix of sparse dims and dense dims. We keep track of `nnz`, `sparseDims`, `denseDims`, a `indices tensor of size = (sparseDims, nnz)`, and a `values tensor of size (nnz, size[sparseDims:])`. Additionally, we have a flag to note if the tensor is `coalesced`.

### Should we keep this representation?

- Our currently hybrid representation has some issues. It's difficult to support operators on hybrid tensors, *especially because only embedding uses hybrid*. Furthermore, some functions have ambiguous outputs on hybrid tensors (e.g. transposing a dense dim and a sparse dim is ambiguous). Because of this, it makes sense to have embedding has a special case, and only support “true” sparse tensors with denseDims = 0.

- Many sparse libraries use CSR because it's really efficient for ops such as mm. Ideally, we'd like to make use of these libraries, but it's difficult: CSR can't represent more than 2D, but COO requires a lot of conversions. Potential solution: caching CSR representations?

- Caffe2 uses per-row counts to represent sparse tensors, and they have some level of sparse support.

### A couple caveats with our current system

- Things we should always keep in mind
```
Ideal INVARIANTS:
  _sparseDims: range [0, len(shape)]; _sparseDims + _denseDims = len(shape)
  _denseDims : range [0, len(shape)]; _sparseDims + _denseDims = len(shape)
  _indices.shape: dimensionality: 2,  shape: (_sparseDims, nnz)
  _values.shape:  dimensionality: 1 + _denseDims.  shape: (nnz, shape[_sparseDims:])

Actual INVARIANT differences:
  1) _sparseDims: range [1, len(shape)] (i.e. we don't allow 0 sparse dimensions)
  2) when nnz = 0, there is strange behavior because we lack 0-dimensional sparse tensors.  Namely:
     dimensionality == 0, _sparseDims == 0, _denseDims == 0, _indices.shape == {0}, _values.shape == {0}
  3) For both _indices.shape and _values.shape, the nnz dimension may be larger than nnz
  4) For _values.shape, the non-nnz dimensions may be smaller than the corresponding dimension size, e.g.
     a shape (2,3) sparse tensor with _sparseDims == 1, may have _values.shape: (nnz, &lt;=2, &lt;=3).
```

- We're keeping track of nnz, sparseDims, denseDims and coalesced independently of the indices and values tensors. For instance, we may need to write the following code to maintain the invariance:
```
_get_sparse_impl(r)-&gt;set_indices_and_values(r_indices, r_values);
_get_sparse_impl(r)-&gt;set_nnz(t._nnz());
_get_sparse_impl(r)-&gt;set_coalesced(t.is_coalesced());
```

- Without support for 0-dim tensors, we cannot have proper support for scalar sparse tensors. This makes it unclear how to implement ops like reduction ops, which is currently represented as a dense scalar. Maybe there is nothing wrong with the current representation, we just need to be minded that sparse ops can return a scalar. 

- The nnz and the nnz dimension in two indices and value tensors may not be the same, some cuda kernels rely on this.

- In values, the non-nnz dimensions might not actually match the dimensions of the sparse tensor

- We have dim == sparseDims + denseDims, but sparseDims cannot be 0. On one hand, this makes sense because a sparse tensor should have at least one sparse dim, but at the same time, it's in conflict with the idea of a scalar sparse tensor.

- An empty sparse tensor has indices of dim = 1, which means we have to check `nnz != 0` everywhere in using a TensorAccessor of indices.

### Some of these issues are fixed by @yf225  PR https://github.com/pytorch/pytorch/pull/9279

Current behavoir of an empty sparse tensor:
```
&gt;&gt;&gt; a = torch.sparse_coo_tensor([], [], [2, 3])
&gt;&gt;&gt; i = a._indices()
&gt;&gt;&gt; v = a._values()

&gt;&gt;&gt; print(a)
torch.sparse.FloatTensor of size (2,3) with indices:
tensor([], dtype=torch.int64)
and values:
tensor([])

&gt;&gt;&gt; print('i = %s' % i)
i = tensor([], dtype=torch.int64)

&gt;&gt;&gt; print('v = %s' % v)
v = tensor([])

&gt;&gt;&gt; print('a.dim = %d, i.dim = %d, v.dim = %d' % (a.dim(), i.dim(), v.dim()))
a.dim = 2, i.dim = 1, v.dim = 1

&gt;&gt;&gt; print('a._dimI = %d, a._sparseDims = %d, a._dimV = %d, a._denseDims = %d, a._nnz = %d' % 
          (a._dimI(), a._sparseDims(), a._dimV(), a._denseDims(), a._nnz()))
a._dimI = 0, a._sparseDims = 2, a._dimV = 0, a._denseDims = 0, a._nnz = 0
```

When properly supported:
```
&gt;&gt;&gt; import torch
&gt;&gt;&gt; a=torch.sparse.DoubleTensor()
&gt;&gt;&gt; a
torch.sparse.DoubleTensor of size (0,) with indices:
tensor([], size=(1, 0), dtype=torch.int64)
and values:
tensor([], dtype=torch.float64)

# empty sparse tensor to be a 1-dimensional tensor of size [0], 
# with sparseDims == 1 and denseDims == 0

# invariants:
#   _sparseDims + _denseDims = len(shape)
#   _indices.dim = 2, shape = (_sparseDims, nnz)
#   _values.dim = 1 + _denseDims, shape = (nnz, shape[_sparseDims:])
```

This fixes:

- An empty sparse tensor has indices of dim = 1, which means we have to check `nnz != 0` everywhere in using a TensorAccessor of indices.


### Autograd support
```
&gt;&gt;&gt; a = torch.sparse_coo_tensor(indices, values, sizes, requires_grad=True)
&gt;&gt;&gt; b = a * 2
&gt;&gt;&gt; b.backward(torch.sparse_coo_tensor(indices, values, sizes))
&gt;&gt;&gt; print(a)

torch.sparse.FloatTensor of size (2,2) with indices:
tensor([[0, 0, 1],
        [0, 1, 1]])
and values:
tensor([4., 6., 8.])
```

A lot of operators on sparse tensors have densified gradients. e.g., `log1p` would make all 0 inputs have a gradient of 1, and it densifies the sparse tensor gradients. Currently we have to raise errors in the backward of these operators. Some potential ways to fix:

- Define special sparse operations so that they operate only on the nnz. This works well for functions that take a single input tensor. For example, we can define sparse log1p such that implicit 0s do not participate, and it would allow us to have a sparse gradient tensor. We can call this operator `s_log1p` or `nnz_log1p` (https://github.com/pytorch/pytorch/issues/1369 has similar discussions). However, we are not sure if they are really what users want, because these operators have different behaviors to their dense counterpart.

- Return a dense gradient. We can have backward functions able to handle both of dense and sparse gradients. No special treatment needed during autograd, just more functions to be written.

- Return the gradient as a sparse tensor. Even though we know a tensor might be completely dense, we can still choose to return its sparse form. This is awful for performance.

- No backward for densified gradients of sparse operator. Users need to call `to_dense` and apply dense operators instead. This will not work without backward for `to_dense`, which itself will have a densified gradients, and it brings us back to the 2nd proposal.


## Functions

### Pointwise one tensor math

|functions|dense grad|
|---|:---:|
|pow|N|
|log1p|Y|
|div_ / div(Sparse, Scalar)|Y|
|mul_ /&nbsp;mul(Sparse, Scalar)|Y|

All pointwise one tensor calls dense couterpart ops on `_values`, so maybe a macro can be written to cover them all.


### Pointwise two tensor math

|functions|formula|dense grad|
|---|---|:---:|
|add_ / add(Sparse, Sparse, Scalar)&nbsp;→ Sparse|add(T, S, alpha) = T + alpha * S|Y|
|add_ / add(Dense, Sparse, Scalar)&nbsp;→ Dense|add(T, S, alpha) = T + alpha * S|Y|
|sub_ / sub(Sparse, Sparse, Scalar)&nbsp;→ Sparse|sub(T, S, alpha) = T - alpha * S|Y|
|mul_ / mul(Sparse, Sparse) → Sparse|mul(T, S) = T * S|Y|

All pointwise two tensor functions have properly optimized CUDA kernel except for `mul_ / mul`:
- `add_ / add(Sparse, Sparse, Scalar)&nbsp;→ Sparse` returns cat(Sparse, Scalar * Sparse)
- `add_ / add(Dense, Sparse, Scalar)&nbsp;→ Dense` parallelizes over nnz
- `sub_ / sub(Sparse, Sparse, Scalar)&nbsp;→ Sparse` calls `add`

`mul_ / mul(Sparse, Sparse) → Sparse` needs parallelized CUDA kernels, possible directions:
1. write customized kernel to parallelized over nnz of 1st sparse tensor
2. use cuSPARSE?

### BLAS

|functions|formula|dense grad|
|---|---|:---:|
|addmm(Dense, Sparse, Dense, Scalar, Scalar) → Dense|addmm(T, S, D, beta, alpha) = beta * T + alpha * mm(S, D)|Y|
|sspaddmm(Sparse, Sparse, Dense, Scalar, Scalar) → Sparse|sspaddmm(T, S, D, beta, alpha) = beta * T + alpha * mm(S, D)|Y|
|mm(Sparse, Dense) → Dense|mm(S, D) = mm(S, D)|Y|
|smm(Sparse, Dense) → Sparse|smm(S, D) = mm(S, D)|Y|
|hspmm(Sparse, Dense) → HybridSparse|hspmm(S, D) = mm(S, D)|Y|
|spmm(Sparse, Dense) → Dense|spmm(S, D) = mm(S, D)|Y|

Functions with CUDA kernel optimized are `mm`, `addmm` and `hspmm`. `addmm` and `hspmm` use cuSPARSE (cusparseScsrmm2 and cusparseDcsrmm2) in CUDA kernel, and `mm` calls `addmm`. However, `smm` and `sspaddmm` don't have CUDA support (gets removed?).


### Others

|functions|dense grad|
|---|:---:|
|clone|NA|
|norm|N|
|zero_|N|
|t_ / t|N|


## Optimizers

- optim.SGD (CUDA and CPU)
- optim.SparseAdam (CUDA and CPU) - lazy version of Adam algorithm
- optim.Adagrad (CPU)


## Future work

### TODO functions

|Functions|dense grad|
|---|:---:|
|nonzero|           N|
|sum|               Y|
|copy_|             N|
|narrow|            N|
|select_index|      N|
|mul_ / mul(S, D)|  Y|
|cuda|              NA|
|F.linear|          Y|
|softmax|           N|
|cat|               N|
|max|               N|
|bmm(S, D)|         Y|

- There is a list of pointwise functions for sparse can be implemented by calling dense ops on their `_values`, some helpers or macros can be written to make all of these ops available for sparse.
```
abs, acos, asin, atan, ceil, cos, cosh, erf, exp, expm1, floor, 
log, log10, log2, round, sin, sinh, sqrt, rsqrt, tan, trunc
```


This note was summarized by @li-roy and me. All suggestions/comments are more than welcomed!

cc @soumith @ezyang @colesbury @gchanan @SsnL @zou3519 @yf225</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/8853">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/8853" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/8853" target="_blank" rel="noopener nofollow ugc">Todo functions and autograd supports for Sparse Tensor</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2018-06-25" data-time="17:23:07" data-timezone="UTC">05:23PM - 25 Jun 18 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/weiyangfb" target="_blank" rel="noopener nofollow ugc">
          <img alt="weiyangfb" src="https://avatars.githubusercontent.com/u/38509346?v=4" class="onebox-avatar-inline" width="20" height="20">
          weiyangfb
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: sparse
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Here summarizes a list of requested Sparse Tensor functions and autograd support<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">s from previous PRs. Please feel free to comment on functions that should be added also.

## Functions
- [x] `sum()` with autograd https://github.com/pytorch/pytorch/pull/12430
- [ ] `max()` with autograd
- [x] `log1p()` https://github.com/pytorch/pytorch/pull/8969
- [x] `S.copy_(S) with autograd ` https://github.com/pytorch/pytorch/pull/9005
- [ ] indexing (`gather()`, `index_select()`)
- [ ] `mul_(S, D) -&gt; S`, `mul(S, D) -&gt; S` with autograd
- [x] `cuda()`
- [ ] `nn.Linear` with autograd (SxS, SxD, relies on `addmm` and `matmul`)
- [x] `softmax()` with autograd (same as in [TF](https://www.tensorflow.org/api_docs/cc/class/tensorflow/ops/sparse-softmax): Applies softmax() to a region of a densified tensor submatrix; (2) Masks out the zero locations; (3) Renormalizes the remaining elements. SparseTensor result has exactly the same non-zero indices and shape)
- [x] `to_sparse()` #12171
- [x] `narrow_copy()` https://github.com/pytorch/pytorch/pull/11342
- [ ] `sparse_mm(S, D) -&gt; D` with autograd
- [x] `cat()` https://github.com/pytorch/pytorch/pull/13577
- [x] `unsqueeze()`, `stack()` https://github.com/pytorch/pytorch/pull/13760

### Wish list
- `bmm(S, D)` (add an extra sparse dim at `indices` of SparseTensor as batch dim?)
- broadcasting `mul(S, D) -&gt; S`
- `Dataset`, `Dataloader`
- `save`, `load` for sparse tensors

## Existing
- autograd supported for `values()` via https://github.com/pytorch/pytorch/pull/13001 (Thanks to @SsnL!), that means all element-wise ops are supported in sparse now
- norm (cannot take `dim` args)
- pow
- clone
- zero_
- t_ / t
- add_ / add(Sparse, Sparse, Scalar) -&gt; Sparse
- add_ / add(Dense, Sparse, Scalar) -&gt; Dense
- sub_ / sub(Sparse, Sparse, Scalar) -&gt; Sparse
- mul_ / mul(Sparse, Sparse) -&gt; Sparse
- mul_ / mul(Sparse, Scalar) -&gt; Sparse
- div_ / div(Sparse, Scalar) -&gt; Sparse
- addmm(Dense, Sparse, Dense, Scalar, Scalar) -&gt; Dense
- sspaddmm(Sparse, Sparse, Dense, Scalar, Scalar) -&gt; Sparse
- mm(Sparse, Dense) -&gt; Dense
- smm(Sparse, Dense) -&gt; Sparse
- hspmm(Sparse, Dense) -&gt; HybridSparse
- spmm(Sparse, Dense) -&gt; Dense</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<h2><a name="so-far-2" class="anchor" href="#so-far-2"></a>So far:</h2>
<p>I have found that Tensorflow also does not like sparse operations with the same reasons as PyTorch.<br>
Apparently, there are some libraries such as <a href="https://docs.nvidia.com/cuda/cusolver/#cuSolverSP-function-reference" rel="noopener nofollow ugc">cuSolver</a> which supports forward but still backward needs to be implemented. As there are no python bindings for this lib, I wonder does it make sense to make this happen in PyTorch.</p>
<p>Another approach which I don’t like (nothing is like PyTorch <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=12" title=":heart:" class="emoji" alt=":heart:" loading="lazy" width="20" height="20"> ) is that change framework to Jax as apparently it supports forward/backward for <code>spsolve</code> or <code>cholesky_solve</code>, or even using another language such as Julia.</p>
<p>Please let me know if there is any ongoing project on solving this issue or any other ideas.</p>
<p>Bests</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/nikronic">@Nikronic</a>,</p>
<p>Are there any updates to this? I’m just bumping this thread to the top to see if there are any updates!</p>