Pytorch backward error: sparse tensor

username: bobo_Popo
cooked: <p>ask for help: I was using pytorch to train a GAN-based model. During the training process, I need to use a sparse tensor, which is calculated by the learnable parameters of the generator, which means that when I update the generator, I need to pass a sparse tensor with gradients to the discriminator, i.e. required_grads=True.</p>
<p>However, when doing backward() on the loss of the generator:</p>
<pre><code class="lang-auto">loss_g = generator_loss(pred_label, pred_real, pseudo_label, pseudo_train, is_real)
loss_g.backward()
</code></pre>
<p>I have the following problem:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm 2021.2\plugins\python\helpers\pydev\pydevd.py", line 1483, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm 2021.2\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "D:/myprojects/Mywork/experiment_logs/2022_11_5/gan_demo.py", line 185, in &lt;module&gt;
    loss_g.backward()
  File "D:\anaconda3\envs\myenv\lib\site-packages\torch\_tensor.py", line 255, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "D:\anaconda3\envs\myenv\lib\site-packages\torch\autograd\__init__.py", line 149, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: sparse tensors do not have strides
</code></pre>
<p>I tried to use the torchviz tool to view the computation graph of the generator loss, and the results are as follows:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/6/76dc222c65ab2c836ac32cbca7f03bf1b4c677f0.png" data-download-href="https://discuss.pytorch.org/uploads/default/76dc222c65ab2c836ac32cbca7f03bf1b4c677f0" title="Digraph.gv"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/76dc222c65ab2c836ac32cbca7f03bf1b4c677f0_2_152x500.png" alt="Digraph.gv" data-base62-sha1="gXtYtcQft2KN8KvRSuCE0FWOHm0" width="152" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/76dc222c65ab2c836ac32cbca7f03bf1b4c677f0_2_152x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/76dc222c65ab2c836ac32cbca7f03bf1b4c677f0_2_228x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/76dc222c65ab2c836ac32cbca7f03bf1b4c677f0_2_304x1000.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Digraph.gv</span><span class="informations">1864×6112 111 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: clerkie
cooked: <p>what is the shape of your input tensor?</p>

username: bobo_Popo
cooked: <p>pred_label: shape=[2735, 7], dense_tensor, grad_fn=SparseAddmmBackward<br>
pred_real: shape=[2735,2], dense_tensor, grad_fn=SparseAddmmBackward<br>
pseudo_label: shape=[2735], dense_tensor,  no grad ( i.e. each sample’s class)<br>
pseudo_train: shape=[2735], dense_tensor, no grad  ( i.e. train idx )<br>
is_real: shape=[2735], dense_tensor, no grad (i.e. is real or generated)</p>
<p>Here is my <code>generator_loss</code> function:</p>
<pre><code class="lang-auto">def generator_loss(_pred_label, _pred_real, _pseudo_label, _pseudo_train, _is_real, loss_fn=nn.CrossEntropyLoss()):
    pseudo_idx = [ _ for _ in range(_is_real.nonzero()[-1]+1, _is_real.shape[0])]
    num_pseudo = len(pseudo_idx)
    loss_class = loss_fn(_pred_label[pseudo_idx].squeeze(), _pseudo_label[pseudo_idx].squeeze())
    _is_real_rev = torch.tensor([1]*num_pseudo).squeeze()
    loss_real = loss_fn(_pred_real[pseudo_idx].squeeze(), _is_real_rev)

    _loss_g = loss_real + loss_class
    print(f"---gen_loss:{_loss_g.item():.4f}")
    return _loss_g
</code></pre>
<p>Thank you for your reply.</p>

username: bobo_Popo
cooked: <p>I use a dense tensor and a sparse tensor to compute <code>pred_label</code> and <code>pred_real</code>:</p>
<pre><code class="lang-auto">pred_label, pred_real = discriminator.forward(pseudo_x, pseudo_adj)
loss_g = generator_loss(pred_label, pred_real, pseudo_label, pseudo_train, is_real)
</code></pre>
<p>The <code>forward </code> process of discriminator:</p>
<pre><code class="lang-auto">def forward(self, x, adj):
    x = self.conv1(x, adj)
    x = self.relu(x)
    x = self.dropout(x)
    pred_label = self.conv2(x, adj)
    pred_real = self.conv3(x, adj)

    return pred_label, pred_real
</code></pre>
<p>and <code>conv</code> layer is as follows:</p>
<pre><code class="lang-auto">def forward(self, x, adjacency):
    x = self.linear(x)
    x = torch.sparse.mm(adjacency, x)
    return x
</code></pre>