Sparse tensor is not supported by aten::ne

username: rrkarim
cooked: <p>How can I find the operation in the code which is using aten::ne op in backward that doesn’t support sparse tensors? Is there any method for debugging the backward?</p>
<p>I have an untrackable exception when I’m trying to backpropogate gradients:<br>
<code>RuntimeError: Could not run 'aten::ne.Tensor' with arguments from the 'SparseCUDATensorId' backend. 'aten::ne.Tensor' is only available for these backends: [CUDATensorId, QuantizedCPUTensorId, CPUTensorId, VariableTensorId]</code>.</p>
<p>Maybe there is a list of python torch ops that have <code>aten:ne</code> in the backward c++ implementation?</p>
<p>Unfortunately, I can’t generate a simple code snippet for testing, because the issue is not about the support of sparse tensor in <code>aten::ne</code> but about the python op that contains the usage of it. Still, the code that I’m trying to work with is the pytorch implementation of nerf: <a href="https://github.com/yenchenlin/nerf-pytorch" rel="nofollow noopener">yenchenlin/nerf-pytorch</a>. I’m just inserting sparse <code>nn.Embedding</code> in module list on the line <a href="https://github.com/yenchenlin/nerf-pytorch/blob/master/run_nerf_helpers.py#L83" rel="nofollow noopener">here/line#83</a>.</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can try to enable anomaly mode: <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.detect_anomaly">https://pytorch.org/docs/stable/autograd.html#torch.autograd.detect_anomaly</a></p>
<p>This will show you which forward function is causing the error in the backward.</p>

username: rrkarim
cooked: <p><code>detect_anomaly</code> doesn’t help here, stack trace stop at the sparse embedding forward fn call. It doesn’t provide any info, since I already know that the problem is caused by sparse tensors.</p>

username: albanD
cooked: <p>When you say sparse embedding, you mean the nn.Embedding with sparse = True?<br>
If so, that means that it generates a sparse gradient. So the op just before needs to be able to handle such gradient properly.</p>

username: rrkarim
cooked: <p>Yeah, but the op before is the backward for <code>nn.BatchNorm1d</code>, which should support sparse gradients. Ok, I guess I need to dig a little more then. Thanks so much. Will write here, if something interesting is found.</p>

username: rrkarim
cooked: <p>Are there any difficulties for supporting sparse tensors in <code>aten:ne</code>? Maybe I can open PR on it later if not?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I don’t think there is any specific reason. And we would be happy to accept PR to increase sparse support <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>But I would say that many functions don’t support sparse Tensors, so you might get into more trouble adter added <code>aten::ne</code>, you might want to check before.</p>

username: rrkarim
cooked: <p>So there was hidden <code>set_detect_anomaly</code> flag in the code. That was the issue.</p>

username: albanD
cooked: <p>Note that if you use the latest version, this should have been fixed: <a href="https://github.com/pytorch/pytorch/issues/28649">https://github.com/pytorch/pytorch/issues/28649</a></p>

username: qiuli380
cooked: <p>Hi,I have the same problem when using the GAT framework:RuntimeError: Could not run ‘aten::gt.Scalar’ with arguments from the ‘SparseCPUTensorId’ backend. ‘aten::gt.Scalar’ is only available for these backends: [CPUTensorId, QuantizedCPUTensorId, VariableTensorId]. can you tell me how to solve this problem</p>

username: qiuli380
cooked: <p>Hi，what does  set_detect_anomaly means，What’s in the code，how can i solve it？ <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>

username: albanD
cooked: <p>It is used to find nan and other issues in the backward pass.<br>
It was using specific functions to do that detection that were not supported by sparse Tensors hence the issue.<br>
But your error is not related as we don’t use gt() in there.</p>
<p>This error just means that this function is not implemented for sparse Tensors and you cannot use it.</p>