Does PyTorch support autograd on sparse matrix?

username: Yiru_Shen
cooked: <p>Hi there,</p>
<p>I am a beginner trying to learn PyTorch and there is one question bugging me. I know PyTorch support sparse x dense -&gt; dense function in <a href="http://torch.mm" rel="nofollow noopener">torch.mm</a>. However, I don’t think it currently supports autograd on sparse variables (say sparse matrix). Examples are:</p>
<pre><code class="lang-python">x = torch.sparse.FloatTensor(2,10)
y = torch.FloatTensor(10, 5)
sx = torch.autograd.Variable(x)
sy = torch.autograd.Variable(y)

torch.mm(sx, sy) # fails
</code></pre>
<p>Errors: TypeError: Type torch.sparse.FloatTensor doesn’t implement stateless method addmm</p>
<p>Am I on the wrong track?</p>
<p>Thanks for any comment!</p>
<p>Yiru</p>

username: nitish
cooked: <p>Same question! Any idea?</p>

username: smth
cooked: <p>you guys should use <code>torch.matmul</code> which is implemented for both sparse and dense.</p>

username: nitish
cooked: <p>I tried <code>torch.mm</code> and <code>torch.matmul</code> for sparse*dense matrix multiplication but get the <code>torch.sparse.FloatTensor doesn't implement stateless method addmm</code> error from both.</p>

username: smth
cooked: <p>it does indeed look like it isn’t working. i’m looking into it.</p>

username: smth
cooked: <p>So the bummer is, autograd is not supported for sparse matrices. Sorry for not looking closely when i replied last.</p>
<p>We dont have any tests in test_autograd for sparse either.</p>
<p>We used sparse directly in nn layers such as <code>nn.Embedding</code> at the implementation level.</p>

username: smth
cooked: <p>We’re tracking the finishing of this feature here: <a href="https://github.com/pytorch/pytorch/issues/2389">https://github.com/pytorch/pytorch/issues/2389</a></p>
<p>For <code>torch.mm</code> in particular, it wasn’t that hard for me to make it work with autograd. you can probably patch these in while I send a PR with unit tests and all: <a href="https://gist.github.com/anonymous/49c10bc17ac4a97307d52c07d01a2870">https://gist.github.com/anonymous/49c10bc17ac4a97307d52c07d01a2870</a></p>

username: nitish
cooked: <p>Thanks Soumith for looking into this. Patched the changes and it looks like <code>torch.mm</code> works on an example I tried.</p>

username: percqdeng
cooked: <p>I tried, <code>torch.matmul</code> works on  a dot b where a is sparse tensor, and b has to have dimension &gt; 1.<br>
So far until 0.2, sparse tensor can’t be the second operand, otherwise there will be some error like <code>"invalid combination of arguments"</code>.<br>
<code>(torch.SparseDoubleTensor source, torch.DoubleTensor mat2)</code> is what is expected.</p>
<p>In order to perform autograd, you may define a dot product inheriting <code>torch.autograd.Function</code>. And also squeeze or unsqueeze the dimension when necessary.</p>
<p>Don’t know when this will be resolved.</p>

username: nitish
cooked: <p>This patch was working fine with SGD. Today I tried Adadelta and RMSprop but got the following error :  <code>'torch.cuda.sparse.FloatTensor' object has no attribute 'addcmul_'</code></p>

username: jdily
cooked: <p>I am curious if the PR <a class="mention" href="/u/smth">@smth</a> mentioned is there yet??<br>
So far I am on version 0.2.0+5de7f9e, and it seems the <code>torch.mm</code> still didnt support the sparse x dense -&gt; dense ?<br>
I also raised some discussion in <a href="https://github.com/pytorch/pytorch/issues/2389#issuecomment-342119147" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/2389#issuecomment-342119147</a><br>
Just curious about what is the fastest way to make this work?<br>
Or it just I found the wrong function to use?</p>
<p>thx a lot guys!</p>

username: laoreja
cooked: <p>I cannot find the file torch/autograd/_functions/blas.py for PyTorch 0.4.0.<br>
Also the variable.py does not contain the stuff in the patch.</p>
<p>So how can we make PyTorch 0.4.0 mm support autograd on Sparse Tensor?</p>
<p>mm works for sparse x dense -&gt; dense, but calling backward() on the result leads to error:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-47-12626f95b38d&gt; in &lt;module&gt;()
----&gt; 1 s.backward()

th04/lib/python3.5/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
     91                 products. Defaults to ``False``.
     92         """
---&gt; 93         torch.autograd.backward(self, gradient, retain_graph, create_graph)
     94
     95     def register_hook(self, hook):

th04/lib/python3.5/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     87     Variable._execution_engine.run_backward(
     88         tensors, grad_tensors, retain_graph, create_graph,
---&gt; 89         allow_unreachable=True)  # allow_unreachable flag
     90
     91

RuntimeError: Expected object of type torch.FloatTensor but found type torch.sparse.FloatTensor for argument #2 'mat2'
</code></pre>

username: laoreja
cooked: <p>Never mind, I find <a href="https://github.com/tkipf/pygcn/blob/master/pygcn/layers.py" rel="nofollow noopener">https://github.com/tkipf/pygcn/blob/master/pygcn/layers.py</a>, which solves the problem!</p>

username: mshuaibi
cooked: <p>How was this resolved? I still get an ‘Expected object of backend CPu but got backedn SparseCPU for argument <span class="hashtag">#2</span> ‘mat2’’ error</p>