Dense_tensor.mm(sparse_tensor)

username: kaiyuyue
cooked: <p>Hi, there</p>
<p>When I run the code</p>
<pre><code class="lang-auto">out = dense_tensor.mm(sparse_tensor)
</code></pre>
<p>it will give the error:</p>
<pre><code class="lang-auto">RuntimeError: Expected object of type Variable[torch.cuda.FloatTensor] but found type Variable[torch.cuda.sparse.FloatTensor] for argument #1 'mat2'
</code></pre>
<p>If I convert the <code>sparse_tensor</code> into dense type by using <code>sparse_tensor.to_dense()</code> first, then run the <code>torch.mm</code> it will tell me to buy a new RAM like</p>
<pre><code class="lang-auto">RuntimeError: $ Torch: not enough memory: you tried to allocate 600GB. Buy new RAM! at /pytorch/torch/lib/TH/THGeneral.c:246
</code></pre>
<p>How to solve this problem? PyTorch version: <code>0.3.0.post4</code><br>
Thanks.</p>

username: richard
cooked: <p>pytorch doesn’t support <code>dense * sparse</code> matrix multiplication right now. Out of curiosity, what are you using this for?</p>

username: kaiyuyue
cooked: <p>thx, and what about <code>sparse * dense</code>?</p>

username: richard
cooked: <p>At least one of <code>sparse_tensor.mm(dense_tensor)</code>, <code>torch.mm(sparse_tensor, dense_tensor)</code> should work.</p>

username: kaiyuyue
cooked: <p>Thx, I think I’m stuck into the trouble of this <a href="https://github.com/pytorch/pytorch/issues/2389" rel="nofollow noopener">issue</a>, <code>torch.autograd.Variable</code> does not support <code>sparse</code> operations.</p>

username: richard
cooked: <p>Yeah, you won’t be able to do a <code>.backward()</code> if you use <code>torch.mm(sparse, dense)</code> because <code>dense * sparse</code> matrix multiplication isn’t implemented.</p>

username: kaiyuyue
cooked: <p>I change the code from <code>dense.mm(sparse)</code> into <code>sparse.mm(dense)</code> as you told me. Then I encounter the same problem in that <a href="https://github.com/pytorch/pytorch/issues/2389" rel="nofollow noopener">issue</a> with the error</p>
<pre><code class="lang-auto">RuntimeError: Expected object of type Variable[torch.cuda.sparse.FloatTensor] but found type Variable[torch.cuda.FloatTensor] for argument #1 'mat2'
</code></pre>
<p>sad~<br>
The trouble is <code>torch.mm(sparse, dense) works with tensors but not variables to sparse ops with autograd</code></p>

username: richard
cooked: <p>Oh, my bad. <code>torch.mm(sparse_variable, dense_variable)</code> is implemented on the master branch but not 0.3. You can try building from source (<a href="https://github.com/pytorch/pytorch">instructions here</a>) to use it.</p>

username: kaiyuyue
cooked: <p>Thank you very much, I will try.  Another question: will this way of <code>sparse_Variable.mm(dense_Variable)</code> avoid the memory error?</p>
<pre><code class="lang-auto">RuntimeError: $ Torch: not enough memory: you tried to allocate 600GB. Buy new RAM! at /pytorch/torch/lib/TH/THGeneral.c:246
</code></pre>
<p>the dim of dense matrix is about <code>100k</code>, I know this is quite large.</p>

username: richard
cooked: <p><code>sparse_variable * dense_variable</code> gives you a <code>dense_variable</code> as the output. If the output is too large to fit into memory then I would say no, it won’t fix it. It depends on the output size <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: kaiyuyue
cooked: <p>sparse <code>torch.Size([2L, 100kL])</code> * dense <code>torch.Size([100kL, 1000L])</code>, the output size is small. So this case can work well, is right?</p>

username: richard
cooked: <p>Your dense tensor has 10^8 elements; assuming a minimum 4 bytes per element you get 400 mb for it.<br>
So it should be fine. I’m not sure why a dense * dense multiplication wouldn’t work for you.</p>

username: kaiyuyue
cooked: <p>in GPU, I inspect that it will cost about 6G gpu memory, just doing the <code>torch.mm()</code></p>

username: yuwenchao
cooked: <p>Hi Richard, does PyTorch support loss.backward() now if <code>loss = torch.mm(sparse, dense)</code>? v0.4.1 or v1.0?</p>

username: Rakshit_Kothari
cooked: <p>Torch version 0.4.1<br>
Weird but torch.mm(Dense, Sparse) does not work but torch.mm(Sparse, Dense) works just fine. backward() also works just fine. I’ve managed to work around this by making a transpose. Thankfully my Sparse matrix is Symmetric.</p>