Is PyTorch fully equipped to support network training using sparse tensors?

stark: <p>Are PyTorch layers and operations within the <code>nn.Module</code> capable of fully supporting sparse tensors by default?</p>
<p>I am aiming to train a CNN for both 2D and 3D images. There are external libraries available, such as <a href="https://github.com/NVIDIA/MinkowskiEngine" rel="noopener nofollow ugc">Nvidia’s Minkowski Engine</a>, <a href="https://github.com/numenta/nupic.torch/" rel="noopener nofollow ugc">SPConv</a>, <a href="https://github.com/facebookresearch/SparseConvNet" rel="noopener nofollow ugc">Numenta</a>, and <a href="https://github.com/rusty1s/pytorch_sparse" rel="noopener nofollow ugc">PyTorch Sparse</a> which efficiently handle sparse data and can accelerate the training process. However, most of these projects are, on average, around 4 years old, and their compatibility with Windows and particularly Ubuntu 22.02 is either limited or non-existent. The compilation of these libraries and their integration into existing projects poses considerable challenges.</p>
<p>I have come across several issues on the <a href="https://github.com/pytorch" rel="noopener nofollow ugc">PyTorch GitHub repository</a> that pertain to requests for features related to sparse data. Notably, a few of these mentions are highlighted below in chronological order:</p>
<ul>
<li>[Nov 14, 2020] <a href="https://github.com/pytorch/pytorch/issues/47915" rel="noopener nofollow ugc">sparse filter layers (more specifically convolutions)</a></li>
<li>[Sep 6, 2021] <a href="https://github.com/pytorch/pytorch/issues/64544" rel="noopener nofollow ugc">Feature Request: Any plan to add ‘Sparse Convolution’ as default nn.module?</a></li>
<li>[May 26, 2022] <a href="https://github.com/pytorch/pytorch/issues/78262" rel="noopener nofollow ugc">Request for adding the possibility for training on sparse tensors</a></li>
</ul>
<p>I would greatly appreciate any comments or suggestions you may have.</p>

KFrank: <p>Hi Stark!</p>
<aside class="quote no-group" data-username="stark" data-post="1" data-topic="187225" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stark/48/29013_2.png" class="avatar"> stark:</div>
<blockquote>
<p>Are PyTorch layers and operations within the <code>nn.Module</code> capable of fully supporting sparse tensors by default?</p>
</blockquote>
</aside>
<p>To the best of my knowledge, no.</p>
<aside class="quote no-group">
<blockquote>
<p>I would greatly appreciate any comments or suggestions you may have.</p>
</blockquote>
</aside>
<p>Pytorch’s support for sparse tensors remains quite incomplete.</p>
<p>It is, for example, possible in pytorch version 2.0.1 (with some work and a lot<br>
of caveats) to pass a sparse tensor through a <code>Linear</code>, but I don’t believe that<br>
the analogous is possible with a <code>Conv2d</code>.</p>
<p>Also, regardless of what pytorch supports, as a tensor with lots of zeros (that<br>
is, a tensor that is sparse mathematically, rather than in terms of how it is<br>
stored) passes through the layers of a network, it typically will become less<br>
and less sparse.  So the bulk of mainstream pytorch use case – complicated<br>
networks for machine learning – will likely not get any benefit from sparse-tensor<br>
support.</p>
<p>Best.</p>
<p>K. Frank</p>

stark: <p>Thank you <a class="mention" href="/u/kfrank">@KFrank</a> for providing valuable insight.</p>
<p>I wonder if those working on 3D scene reconstruction (or similar problems dealing with sparse point clouds) spend as much time training models as those working with dense data.</p>
<p>Can replacing a standard convolution layer with a sparse convolution (or submanifold sparse convolution) layer speed up the whole training process? I read that such a technique is friendlier on TPU than a GPU, and your reply also indicates the same!?</p>