Pruning doesn't affect speed nor memory for Resnet-101

username: cyril9227
cooked: <p>Hello,</p>
<p>Not sure what I’m doing wrong or if it’s an expected behavior.</p>
<p>My experiment is to take a trained ResNet-101 and then prune it using the pytorch functionality in <code>torch.nn.utils.prune</code>, I’m currently using the code provided in the official turorial so nothing fancy yet.</p>
<p>The issue is that, as far as I monitored, the inference speed and the memory footprint remain exactly the same as the original unpruned model, which is weird for me.</p>
<pre><code class="lang-auto"># model has been loaded properly before
for name, module in model.named_modules():
    # prune 20% of connections in all 2D-conv layers
    if isinstance(module, torch.nn.Conv2d):
        prune.l1_unstructured(module, name='weight', amount=0.2)
        prune.remove(module, 'weight') # make it permanent
    # prune 40% of connections in all linear layers
    elif isinstance(module, torch.nn.Linear):
        prune.l1_unstructured(module, name='weight', amount=0.4)
        prune.remove(module, 'weight')
</code></pre>
<p>For timing I use :</p>
<pre><code class="lang-auto">    start = time.time()
    # get predictions
    output = model(inputs)
    end = time.time()  
    print(end - start)
</code></pre>
<p>For memory I use :</p>
<pre><code class="lang-auto">def get_memory(model):
    memory = 0
    for p in model.parameters():
        memory += 32 * p.nelement()
    return memory
</code></pre>
<p>Am I doing something wrong ? How can I achieve speed-up using <code>torch.nn.utils.prune</code> ?</p>
<p>Thank you very much for your help,</p>
<p>Cyril</p>

username: ra_schoonhoven
cooked: <p>Hello Cyril,</p>
<p>As far as I am aware the pruning module does not prune models by removing weights or something. Instead, it adds new parameters called <code>weight_orig</code> and <code>weight_mask</code>. The <code>weight_orig</code> parameter stores the original weights. The pruning is performed by applying the <code>weight_mask</code>, which is a tensor of 0s and 1s depending on whether that weight is pruned, in the forward and backward passes to mask them out (by using <code>forward_pre_hook</code> etc.). So pruning currently actually requires a bit more memory and a bit more computation.</p>
<p>I asked a question myself recently on what would be the most straightforward way to obtain speedups but it is not trivial unfortunately (<a href="https://discuss.pytorch.org/t/discussion-of-practical-speedup-for-pruning/75212/2" class="inline-onebox">Discussion of practical speedup for pruning</a>).</p>

username: Michela
cooked: <p><a class="mention" href="/u/ra_schoonhoven">@ra_schoonhoven</a>’s answer is correct. For more information, see my answer to a similar question (+ links therein):</p><aside class="quote quote-modified" data-post="2" data-topic="83429">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michela/48/18830_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/weight-pruning-on-bert/83429/2">Weight Pruning on BERT</a> 
  </div>
  <blockquote>
    The point of PyTorch pruning, at the moment, is not necessarily to guarantee inference time speedups or memory savings. It’s more of an experimental feature to enable pruning research. 
In any case, answers to questions similar to yours were given <a href="https://github.com/pytorch/tutorials/pull/605#issuecomment-585994076" rel="noopener nofollow ugc">here</a> and <a href="https://discuss.pytorch.org/t/how-to-improve-inference-time-of-pruned-model-using-torch-nn-utils-prune/78633/4">here</a>. 
TL;DR: You can save space by calling .to_sparse() which brings your sparse tensor into coordinate representation. You cannot expect any inference speedups unless you use a custom sparse matrix algebra library to power your computation. t…
  </blockquote>
</aside>


username: rwightman
cooked: <p>Since the msg board is full of pruning today… a while back, a researcher submitted a PR to one of my repos with some compelling pruned models (including a ResNet101 variant). They used a technique based on this paper: <a href="https://arxiv.org/abs/2002.08258" rel="nofollow noopener">https://arxiv.org/abs/2002.08258</a></p>
<p>The pruned variants are created by applying an overlay in a txt file that defines the modifications to make to the original network, which was really easy to integrate. The end result is a network that is faster than the original and typically a model from the family one step down, while mainting a higher accuracy than the step down model. Unfortunately the code to generate he pruned models wasn’t included, but perhaps if someone asked they’d be willing to share.</p>
<p>See the original PRs: <a href="https://github.com/rwightman/pytorch-image-models/pull/132" rel="nofollow noopener">https://github.com/rwightman/pytorch-image-models/pull/132</a> and <a href="https://github.com/rwightman/pytorch-image-models/pull/136" rel="nofollow noopener">https://github.com/rwightman/pytorch-image-models/pull/136</a></p>
<p>Another researcher has recently reached out about integrating this pruning technique with my models: <a href="https://github.com/cmu-enyac/LeGR" rel="nofollow noopener">https://github.com/cmu-enyac/LeGR</a> … definitely some interesting progress in this area</p>