Sparse Tensors in PyTorch

Ilya_Kostrikov: <p>What is the current state of sparse tensors in PyTorch?</p>

smth: <p>right now, the description / summary of this PR that was merged 11 hours ago gives a good idea of the current state of things:</p>
<aside class="onebox githubpullrequest">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/pull/735#issue-207307698" target="_blank">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Pull Request">
    <svg width="60" height="60" class="github-icon" viewBox="0 0 12 16" aria-hidden="true"><path d="M11 11.28V5c-.03-.78-.34-1.47-.94-2.06C9.46 2.35 8.78 2.03 8 2H7V0L4 3l3 3V4h1c.27.02.48.11.69.31.21.2.3.42.31.69v6.28A1.993 1.993 0 0 0 10 15a1.993 1.993 0 0 0 1-3.72zm-1 2.92c-.66 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2zM4 3c0-1.11-.89-2-2-2a1.993 1.993 0 0 0-1 3.72v6.56A1.993 1.993 0 0 0 2 15a1.993 1.993 0 0 0 1-3.72V4.72c.59-.34 1-.98 1-1.72zm-.8 10c0 .66-.55 1.2-1.2 1.2-.65 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2zM2 4.2C1.34 4.2.8 3.65.8 3c0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/pull/735" target="_blank">sparse tensor operations</a>
    </h4>

    <div class="branches">
      <code>pytorch:master</code> ← <code>martinraison:sparse</code>
    </div>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2017-02-13" data-time="18:57:34" data-timezone="UTC">06:57PM - 13 Feb 17 UTC</span>
      </div>

      <div class="user">
        <a href="https://github.com/martinraison" target="_blank">
          <img alt="martinraison" src="https://avatars3.githubusercontent.com/u/2560662?v=4" class="onebox-avatar-inline" width="20" height="20">
          martinraison
        </a>
      </div>

      <div class="lines" title="6 commits changed 52 files with 2556 additions and 271 deletions">
        <a href="https://github.com/pytorch/pytorch/pull/735/files" target="_blank">
          <span class="added">+2556</span>
          <span class="removed">-271</span>
        </a>
      </div>
    </div>

  </div>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


apaszke: <p>But we’re not documenting them on purpose, because they might undergo some more changes in the future. The first step was to implement sprase updates for Embedding. Can I ask what’s your use case?</p>

Ilya_Kostrikov: <p>I need just basic sparse matrix multiplication in order to implement a Graph ConvNet model. For example:</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://tkipf.github.io/favicon-32x32.png" class="site-icon" width="32" height="32">
      <a href="https://tkipf.github.io/graph-convolutional-networks/" target="_blank" rel="nofollow noopener">tkipf.github.io</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/357;"><img src="http://tkipf.github.io/graph-convolutional-networks/images/gcn_web.png" class="thumbnail" width="690" height="357"></div>

<h3><a href="https://tkipf.github.io/graph-convolutional-networks/" target="_blank" rel="nofollow noopener">How powerful are Graph Convolutional Networks?</a></h3>

<p>Many important real-world datasets come in the form of graphs or networks: social networks, knowledge graphs, protein-interaction networks, the World Wide Web, etc. (just to name a few). Yet, until recently, very little attention has been devoted to...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>and</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/mdeff/cnn_graph/blob/master/lib/models.py#L898" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/mdeff/cnn_graph/blob/master/lib/models.py#L898" target="_blank" rel="nofollow noopener">mdeff/cnn_graph/blob/master/lib/models.py#L898</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="888" style="counter-reset: li-counter 887 ;">
<li>L = tf.SparseTensor(indices, L.data, L.shape)</li>
<li>L = tf.sparse_reorder(L)</li>
<li># Transform to Chebyshev basis</li>
<li>x0 = tf.transpose(x, perm=[1, 2, 0])  # M x Fin x N</li>
<li>x0 = tf.reshape(x0, [M, Fin*N])  # M x Fin*N</li>
<li>x = tf.expand_dims(x0, 0)  # 1 x M x Fin*N</li>
<li>def concat(x, x_):</li>
<li>    x_ = tf.expand_dims(x_, 0)  # 1 x M x Fin*N</li>
<li>    return tf.concat([x, x_], axis=0)  # K x M x Fin*N</li>
<li>if K &gt; 1:</li>
<li class="selected">    x1 = tf.sparse_tensor_dense_matmul(L, x0)</li>
<li>    x = concat(x, x1)</li>
<li>for k in range(2, K):</li>
<li>    x2 = 2 * tf.sparse_tensor_dense_matmul(L, x1) - x0  # M x Fin*N</li>
<li>    x = concat(x, x2)</li>
<li>    x0, x1 = x1, x2</li>
<li>x = tf.reshape(x, [K, M, Fin, N])  # K x M x Fin x N</li>
<li>x = tf.transpose(x, perm=[3,1,2,0])  # N x M x Fin x K</li>
<li>x = tf.reshape(x, [N*M, Fin*K])  # N*M x Fin*K</li>
<li># Filter: Fin*Fout filters of order K, i.e. one filterbank per feature pair.</li>
<li>W = self._weight_variable([Fin*K, Fout], regularization=False)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>The Laplacian matrix is extremely sparse is this case.</p>

apaszke: <p>You need <code>sparse x sparse -&gt; sparse</code> multiplication, right? Right now we only have <code>sparse x dense -&gt; dense</code> and <code>sparse x dense -&gt; sparse</code>, because that’s what we needed for sparse Embedding updates. You can open a feature request if you want.</p>

Ilya_Kostrikov: <p>I need sparse x dense -&gt; dense. So I can use PyTorch in this case. Thanks a lot!</p>

apaszke: <p><code>torch.mm</code> and <code>torch.spmm</code> should work then</p>

Ilya_Kostrikov: <p>But it’s not in autograd yet?</p>
<p>Is there a fast way to add it?</p>

smth: <p>a fast and local way is for you to write an autograd function for yourself.</p>

hconroy: <aside class="quote no-group" data-username="Ilya_Kostrikov" data-post="1" data-topic="859" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ilya_kostrikov/48/93_2.png" class="avatar"> Ilya_Kostrikov:</div>
<blockquote>
<p>What is the current state of sparse tensors in PyTorch?</p>
</blockquote>
</aside>
<p>That’s my question too (now on 21st Sept).  Can anyone comment on the current state of sparse tensors in PyTorch?</p>
<p>Thank you</p>

11131: <p>I would like to update variable with sparse gradients. and it’s normal.  I know that wasn’t support by tensorflow. so how about pytorch/. Thank you!</p>

pradeepgopalakrishna: <p>What is the status of Sparse Support in PyTorch? I’d like to contribute to the module. What are the current design strategies in place? There seems to be no detail or comprehensive discussion on this aspect.</p>

rajeshpiryani: <p><a href="https://pytorch.org/docs/stable/sparse.html#sparse-coo-tensors" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/sparse.html#sparse-coo-tensors</a></p>