Error while using Sparse Convolution Function (Conv2d with sparse weights)

username: DrImpossible
cooked: <p>Hi,</p>
<p>I implemented a SparseConv2d (with sparse weights and dense inputs) to reimplement <a href="https://github.com/drimpossible/Deep-Expander-Networks" rel="nofollow noopener">my paper</a> however while trying to train, I am getting this issue:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train_test.py", line 169, in &lt;module&gt;
    optimizer.step()
  File "/home/drimpossible/installs/3/lib/python3.6/site-packages/torch/optim/sgd.py", line 106, in step
    p.data.add_(-group['lr'], d_p)
RuntimeError: set_indices_and_values_unsafe is not allowed on Tensor created from .data or .detach()
</code></pre>
<p>The SparseConv2d function is given below:</p>
<pre><code class="lang-auto">class SparseConv2d(torch.nn.Module):
    def __init__(self, inWCin, inWCout, kernel_size, stride=1, padding=0, dilation=1, sparse_size=0, sparse_mode='Expander'):
        super(SparseConv2d, self).__init__()
        self.kernel_size = kernel_size
        self.stride = stride
        self.padding = padding
        self.dilation = dilation
        self.out_channels = inWCout

        mask = make_sparse(in_dim=inWCin, out_dim=inWCout, size=sparse_size, mode=sparse_mode)
        weight = torch.zeros((inWCout, inWCin))
        weight = 0.01*torch.nn.init.kaiming_normal_(weight)
        weight = torch.mul(weight, mask)
        weight = weight.unsqueeze(2).unsqueeze(3).repeat(1, 1, kernel_size, kernel_size)
        weight = weight.view(weight.size(0), -1)
        weight = weight.to_sparse().cuda()
        self.weight = torch.nn.Parameter(weight, requires_grad=True)
    
    def forward(self, x):
        out = (x.size(2)+2*self.padding-self.dilation*(self.kernel_size-1)-1)//self.stride+1
        x_unf = torch.nn.functional.unfold(x, (self.kernel_size, self.kernel_size)).transpose(1,2)
        x_unf = torch.sparse.mm(self.weight, x_unf.reshape(-1, x_unf.size(2)).t()).t().reshape(x.size(0),-1,self.out_channels).transpose(1,2)
        x_unf = x_unf.view(x_unf.size(0), x_unf.size(1), out, out)
        return x_unf
</code></pre>
<p>The make_sparse function just returns an  Erdös-Rényi random expander on LeNet, CIFAR10 with SGD as per <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="nofollow noopener">the tutorial</a>.</p>
<p>The full code for reproduction is available here: (<a href="https://pastebin.com/0PKN0EbJ" rel="nofollow noopener">https://pastebin.com/0PKN0EbJ</a>). I tried hunting, SGD seems to <a href="https://github.com/pytorch/pytorch/issues/9674" rel="nofollow noopener">support sparse ops</a>. I traced the issue to <a href="https://github.com/pytorch/pytorch/blob/master/test/test_sparse.py" rel="nofollow noopener">here</a>. How do I fix this? <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"><br>
Thanks!</p>

username: AtenaNguyen
cooked: <p>Hello,</p>
<p>Do you get any update on this issue?</p>
<p>Bests<br>
Atena</p>

username: DrImpossible
cooked: <p>Hi,</p>
<p>I didn’t get any yet. If you get any workarounds, let me know. Getting SGD to work seems like the last step essentially.</p>
<p>Regards,<br>
Ameya</p>

username: littlebirdoflargehea
cooked: <aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/29814" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/29814" target="_blank" rel="nofollow noopener">SGD fails on sparse matrix</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-11-14" data-time="15:40:01" data-timezone="UTC">03:40PM - 14 Nov 19 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/antonior92" target="_blank" rel="nofollow noopener">
          <img alt="antonior92" src="https://avatars0.githubusercontent.com/u/16557411?v=4" class="onebox-avatar-inline" width="20" height="20">
          antonior92
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🐛 Bug
One SGD step for optimizing a model with a sparse parameter matrix gives me a RuntimeError
To Reproduce
import torch
from torch import...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: optimizer</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: sparse</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/29816" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/29816" target="_blank" rel="nofollow noopener">AdamSparse fails to run</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-11-14" data-time="15:53:27" data-timezone="UTC">03:53PM - 14 Nov 19 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/antonior92" target="_blank" rel="nofollow noopener">
          <img alt="antonior92" src="https://avatars0.githubusercontent.com/u/16557411?v=4" class="onebox-avatar-inline" width="20" height="20">
          antonior92
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🐛 Bug
One AdamSparse step for optimizing a model with a sparse parameter matrix gives me a RuntimeError
To Reproduce
# %%
import torch
from torch...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: sparse</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
This guy has solved the problem.