Sparse Semi-Structured Conv2d tensors?

username: Michelvl92
cooked: <p>Dear reader,</p>
<p>Is it possible with the Sparse Semi-Structured Tensors, or is this not yet supported?</p>
<p>Currently all the examples like: [(beta) Accelerating BERT with semi-structured (2:4) sparsity] (<a href="https://pytorch.org/tutorials/advanced/semi_structured_sparse.html#pruning-bert-to-be-2-4-sparse" class="inline-onebox" rel="noopener nofollow ugc">(beta) Accelerating BERT with semi-structured (2:4) sparsity — PyTorch Tutorials 2.3.0+cu121 documentation</a>) talk about nn.Linear layer, and not about nn.Conv2d. Furthermore, according to: <a href="https://pytorch.org/docs/2.1/sparse.html#constructing-sparse-semi-structured-tensors" rel="noopener nofollow ugc">torch.sparse — PyTorch 2.1 documentation</a> the tensors should be 2D, while a nn.Conv2d is usually 4D? Is it possible to prune on the channel dimension, just like nvidia is doing with their <a href="https://github.com/NVIDIA/apex/tree/master/apex/contrib/sparsity" rel="noopener nofollow ugc">ASP (Automatic SParsity), Channel Permutations for N:M Sparsity</a></p>
<p>Could you give me advice how to apply Sparse Semi-Structured on Conv2d layers if possible, and if yes,  provide an example, or should i still use Nvidia’s framework for this?</p>

username: jcaip
cooked: <p>Currently, we only have support for accelerated sparse matmul, and not sparse convolutions. It would be possible if you can rewrite your convolutions into a matmul (<a href="https://stackoverflow.com/questions/16798888/2-d-convolution-as-a-matrix-matrix-multiplication" class="inline-onebox" rel="noopener nofollow ugc">neural network - 2-D convolution as a matrix-matrix multiplication - Stack Overflow</a>), but this may materialize a much larger tensor and thus won’t lead to further speedups.</p>
<p>ASP only simulates sparsity, it should be possible to simulate sparsity with the WeightNormPruner, but this will not show any speedup unfortunately <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

username: leimao
cooked: <p>Does PyTorch have a plan to officially incorporate or implement ASP (Automatic Structured Sparsity) for Conv2D?<br>
I believe it could be as “simple” as creating a <code>WeightNormSparsifier</code> for Conv2D filters specifically.<br>
Having no speedup in PyTorch is fine. We could export the structured sparsity enabled model from PyTorch and use other acceleration software for inference.<br>
Thanks.</p>

username: LilyAvo
cooked: <p>Sparse semi-structured Conv2D tensors sound intriguing. Could you provide more details or examples of the use case you’re considering? This would help in understanding the requirements better.</p>

username: leimao
cooked: <p>You could check out the <a href="https://github.com/NVIDIA/apex/tree/master/apex/contrib/sparsity" rel="noopener nofollow ugc">NVIDIA ASP package</a>. It’s almost no different from the weight tensor sparsification that PyTorch already has for MatMul. In Conv2D, we do 2:4 sparsification on the input channel.</p>

username: jcaip
cooked: <p>Sorry for the late reply, got lost in my inbox. Currently we have no plans to add support for something similar to ASP in pytorch core. If this is something that you’re interested in adding though, please file an issue here: <a href="https://github.com/pytorch/ao/tree/main/torchao" class="inline-onebox" rel="noopener nofollow ugc">ao/torchao at main · pytorch/ao · GitHub</a></p>
<p>Something like <a href="https://github.com/pytorch/ao/issues/633" class="inline-onebox" rel="noopener nofollow ugc">[RFC] Add LayerSkip to AO · Issue #633 · pytorch/ao · GitHub</a> would be helpful <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>