Torch.mul with sparse and dense arguments

username: whichone
cooked: <p>consider code snippet</p>
<pre><code class="lang-auto">import torch


print(torch.__version__)

u = torch.tensor([0, 1, 0, 2, 0])

v = torch.mul(u.unsqueeze(-1), u.unsqueeze(0))
print(v)

w = torch.mul(u.unsqueeze(-1).to_sparse_coo(), u.unsqueeze(0))
print(w)
print(w.to_dense())
print(w._nnz())
</code></pre>
<p>output for me is as follows</p>
<pre><code class="lang-auto">2.1.1
tensor([[0, 0, 0, 0, 0],
        [0, 1, 0, 2, 0],
        [0, 0, 0, 0, 0],
        [0, 2, 0, 4, 0],
        [0, 0, 0, 0, 0]])
tensor(indices=tensor([[1, 3],
                       [0, 0]]),
       values=tensor([0, 0]),
       size=(5, 5), nnz=2, layout=torch.sparse_coo)
tensor([[0, 0, 0, 0, 0],
        [0, 0, 0, 0, 0],
        [0, 0, 0, 0, 0],
        [0, 0, 0, 0, 0],
        [0, 0, 0, 0, 0]])
2
</code></pre>
<p>is that an expected behaviour? seems rather a bug</p>

username: KFrank
cooked: <p>Hi Maxim!</p>
<aside class="quote no-group" data-username="whichone" data-post="1" data-topic="193608" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/bb73d2/48.png" class="avatar"> whichone:</div>
<blockquote>
<p>seems rather a bug</p>
</blockquote>
</aside>
<p>This does look like a bug to me.</p>
<p>As near as I can tell broadcasting of sparse tensors is not supported,<br>
although I can’t find much in the way of documentation or discussion<br>
of this.</p>
<p>Here is a tweaked version of your example code run on two different<br>
versions of pytorch:</p>
<p>Version 2.1.1:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
2.1.1
&gt;&gt;&gt;
&gt;&gt;&gt; u = torch.tensor ([0, 1, 0, 2, 0])
&gt;&gt;&gt;
&gt;&gt;&gt; torch.mul (u.unsqueeze(-1), u.unsqueeze(0))                                 # works
tensor([[0, 0, 0, 0, 0],
        [0, 1, 0, 2, 0],
        [0, 0, 0, 0, 0],
        [0, 2, 0, 4, 0],
        [0, 0, 0, 0, 0]])
&gt;&gt;&gt; torch.mul (u.unsqueeze(-1).expand (5, 5).to_sparse_coo(), u.unsqueeze(0))   # works on 2.1.1
tensor(indices=tensor([[1, 1, 1, 1, 1, 3, 3, 3, 3, 3],
                       [0, 1, 2, 3, 4, 0, 1, 2, 3, 4]]),
       values=tensor([0, 1, 0, 2, 0, 0, 2, 0, 4, 0]),
       size=(5, 5), nnz=10, layout=torch.sparse_coo)
&gt;&gt;&gt; torch.mul (u.unsqueeze(-1).to_sparse_coo(), u.unsqueeze(0).expand (5, 5))   # error or incorrect result
tensor(indices=tensor([[1, 3],
                       [0, 0]]),
       values=tensor([0, 0]),
       size=(5, 5), nnz=2, layout=torch.sparse_coo)
&gt;&gt;&gt;
&gt;&gt;&gt; u.unsqueeze(-1).to_sparse_coo().expand (5, 5)                               # can't expand sparse tensor
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: expand is unsupported for Sparse tensors
</code></pre>
<p>And version 1.11.0:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.11.0
&gt;&gt;&gt;
&gt;&gt;&gt; u = torch.tensor ([0, 1, 0, 2, 0])
&gt;&gt;&gt;
&gt;&gt;&gt; torch.mul (u.unsqueeze(-1), u.unsqueeze(0))                                 # works
tensor([[0, 0, 0, 0, 0],
        [0, 1, 0, 2, 0],
        [0, 0, 0, 0, 0],
        [0, 2, 0, 4, 0],
        [0, 0, 0, 0, 0]])
&gt;&gt;&gt; torch.mul (u.unsqueeze(-1).expand (5, 5).to_sparse_coo(), u.unsqueeze(0))   # works on 2.1.1
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: mul operands have incompatible sizes
&gt;&gt;&gt; torch.mul (u.unsqueeze(-1).to_sparse_coo(), u.unsqueeze(0).expand (5, 5))   # error or incorrect result
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: mul operands have incompatible sizes
&gt;&gt;&gt;
&gt;&gt;&gt; u.unsqueeze(-1).to_sparse_coo().expand (5, 5)                               # can't expand sparse tensor
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: sparse tensors do not have strides
</code></pre>
<p>I reproduce the error you see on 2.1.1, while pytorch flags the unsupported<br>
operation (albeit somewhat opaquely) on 1.11.0.</p>
<p>Here’s a related github issue:</p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/34355">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/34355" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/34355" target="_blank" rel="noopener nofollow ugc">pytorch sparse_coo_tensor broadcasting in multiplication</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-03-06" data-time="04:32:01" data-timezone="UTC">04:32AM - 06 Mar 20 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2022-10-12" data-time="16:41:32" data-timezone="UTC">04:41PM - 12 Oct 22 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/amirhajibabaei" target="_blank" rel="noopener nofollow ugc">
          <img alt="amirhajibabaei" src="https://avatars.githubusercontent.com/u/25602399?v=4" class="onebox-avatar-inline" width="20" height="20">
          amirhajibabaei
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: sparse
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Using broadcasting in normal tensors, the following works
```
torch.ones(3, 1)<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">*torch.ones(3, 10)
```
I need to extend this behavior to sparse vectors, but I can't:
```
i = torch.tensor([[0, 1, 1],
                          [2, 0, 2]])
a = torch.sparse_coo_tensor(i, torch.ones(3, 1), [2, 4, 1])
b = torch.sparse_coo_tensor(i, torch.ones(3, 10), [2, 4, 10])
# gives RuntimeError: mul operands have incompatible sizes
```
Why shouldn't this work? Is there an pytorch function to do this? If not what is the best alternative algorithm?


cc @vincentqb</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Note the following statement:</p>
<blockquote>
<p><code>torch.mul</code> now supports broadcasting over dense dimensions.</p>
</blockquote>
<p>So it looks like sparse-dense multiplication can broadcast one direction<br>
but not the other on 2.1.1 (as illustrated in the sample code I posted).</p>
<p>Note that <code>expand()</code> is sort of like broadcasting, and you can’t <code>expand()</code><br>
sparse tensors.</p>
<p>I think it would be appropriate to file a github issue for this bug you’ve found.</p>
<p>Best.</p>
<p>K. Frank</p>

username: whichone
cooked: <p>Hi, thanks for response. the statement you’he cited makes sense and clarifyes the accident. that all feels like choking with documentation, which misses details in particular case on supported argument types. following pytorch doc provides a list of functions which support coo_tensorts, among which <code>torch.mul</code> situated, however, not in case when broadcasting needed, it seems. <a href="https://pytorch.org/docs/stable/sparse.html" class="inline-onebox" rel="noopener nofollow ugc">torch.sparse — PyTorch 2.1 documentation</a>.</p>