Purpose of pytorch pruning

username: ori
cooked: <p>I have seen and also tried pytorch pruning, but have not seen any improvements in inference time or reduction of model size. By considering the old posts i have seen pruning is like a expermental feature. Is it still like expermental tool or is there any real impact of inference time.</p>

username: andrea_bragagnolo
cooked: <p>Hi, I did some reaserach on pruning and might help you clear some doubts.</p>
<p>Pruning by itself only sets to zero some elements of the weight tensor, but such elements are still part of the nertwork. This is the reason why you do not see any difference in the model size nor in the inference time.</p>
<p>That said there are ways to exploit a pruned model to gain some advantages depending on the pruning strategies that was used.<br>
If you use an <strong>unstructured</strong> pruning approach (leads to tensors that have zeroes peppered throughout), you could rely on some ad hoc hardware or sotware (i.e. <a href="https://pytorch.org/docs/stable/sparse.html" class="inline-onebox" rel="noopener nofollow ugc">torch.sparse — PyTorch 1.11.0 documentation</a>) which should be optimized to work with sparse tensors.<br>
If you use a <strong>structured</strong> approach (removing entire rows or columns of the weight matrices) you might be able to remove the pruned neurons, leading to smalle and faster models without the need for particular hardware.</p>
<p>For the second case we’ve actually built a PyTorch-compatible library which does exactly that. You are welcome to try it out.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/EIDOSlab/simplify">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/EIDOSlab/simplify" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/baab586ae8671df6ba8dfe9387d6b7bcd2f48240258f6f0061c68bbb4ec2c4df/EIDOSlab/simplify" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/EIDOSlab/simplify" target="_blank" rel="noopener nofollow ugc">GitHub - EIDOSlab/simplify: Simplification of pruned models for accelerated...</a></h3>

  <p>Simplification of pruned models for accelerated inference | SoftwareX https://doi.org/10.1016/j.softx.2021.100907 - GitHub - EIDOSlab/simplify: Simplification of pruned models for accelerated infer...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>If you have any questions feel free to ask.</p>

username: ori
cooked: <p>Thanks for your answer, i really appreciate the work you put on this. Sure will check the simplify. I am <strong>unstructured</strong> pruning hence lot of my weights are set to zero. Does this library support transfromers in general??</p>

username: andrea_bragagnolo
cooked: <p>Our library is currently tested on standard torchvision classifcation models, up to v0.10 (I should probably specify it in the docs since now they added more models).<br>
If you’d like to test it on your specific case and open an issue for any eventual problem, we’ll be happy to assist with it and possibly improve the library.</p>
<p>Just notice that unstructured pruning may not lead to any change since it does not ensure that entire neurons are pruned. In pytorch, for standard Conv2d or Linear, neurons are mapped by the 0 dimension of the weight tensor, so you might want to check if you have some of them which are completely zeroed-out.</p>
<p>Here a naive way to do this:</p>
<pre><code class="lang-auto">layer = Linear(500, 10)
idx = torch.tensor([0, 2, 5])
with torch.no_grad():
    layer.weight[idx] = layer.weight[idx] * 0
print(torch.where(layer.weight.abs().sum(dim=1) == 0)[0])  # dim=(1,2,3) for Conv2d
</code></pre>

username: SofiaCP
cooked: <p>Hi! Can the simplify library handle dependecies among layers such as skip connections and concatenations? Thanks!</p>

username: andrea_bragagnolo
cooked: <p>Hello, sorry for the late reply.<br>
Yes we did handle residual connection (with the sum operation) as they are implemented in the ResNet models contained in torchvision.</p>