Pruning vs Dropout

aulakh83: <p>How weight pruning is different from dropout?</p>

tom: <p>Dropout drops certain activations stochastically (i.e. a new random subset of them for any data passing through the model). Typically this is undone after training (although there is a whole theory about test-time-dropout).</p>
<p>Pruning drops certain weights, i.e. permanently drops some parts deemed “uninteresting”.</p>

aulakh83: <p>Thank you for quick response. I applied global pruning over my model and then retrain it, during test the weight parameter is not there. i think those are renamed. then how can i test the pruned model??</p>

KFrank: <p>Hi Tom!</p>
<aside class="quote no-group" data-username="tom" data-post="2" data-topic="130574" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>Dropout drops certain activations stochastically (i.e. a new random subset of them for any data passing through the model).</p>
</blockquote>
</aside>
<p>What is your intuition about why (and when) dropout works?</p>
<p>Thanks.</p>
<p>K. Frank</p>

tom: <aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="130574">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>What is your intuition about why (and when) dropout works?</p>
</blockquote>
</aside>
<p>I’m a Bayesian at heart, so to me <a href="https://arxiv.org/abs/1506.02142" rel="noopener nofollow ugc">Gal and Ghahramani</a> interpretation seems to be a thing.<br>
In a later paper they show that for RNNs it makes sense to keep the dropout mask fixed between the timesteps to use this interpretation and Gal’s theses has a lot of things on the theme (including “why can we not optimize the dropout probability somewhere, I think”). More generally, it would seem that the hypothetical variational model must make sense as well as the approximation.</p>
<p>What is your take? I always love to hear your thoughts on the more theory-grounded topics.</p>
<p>Best regards</p>
<p>Thomas</p>

KFrank: <p>Hi Thomas!</p>
<aside class="quote no-group quote-modified" data-username="tom" data-post="5" data-topic="130574" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>I’m a Bayesian at heart, so to me <a href="https://arxiv.org/abs/1506.02142" rel="noopener nofollow ugc">Gal and Ghahramani</a> interpretation seems to be a thing.<br>
…<br>
What is your take?</p>
</blockquote>
</aside>
<p>Thanks for the Gal and Ghahramani link.  I can’t say that I’ve absorbed<br>
much of it yet.</p>
<p>I’m not well-studied on dropout, and I just don’t have any intuition about<br>
it that feels “right” to me, so I don’t really have a take.</p>
<p>Best.</p>
<p>K. Frank</p>

aulakh83: <p>Can anyone please help in this??<br>
I applied global pruning over my model and then retrain it, during test the weight parameter is not there. i think those are renamed. then how can i test the pruned model??</p>

tom: <p>Sorry for diverting your thread.<br>
Yes, the pruning seems to rename weights (appending <code>_orig</code> probably is the most common), you can get the current set of weight names using</p>
<pre><code class="lang-auto">print([n for n, p in module.named_parameters()])
</code></pre>
<p>There is an equivalent <code>named_buffers</code> for non-parameters that gets masks.</p>
<p>Also, the tutorial section on <a href="https://pytorch.org/tutorials/intermediate/pruning_tutorial.html#remove-pruning-re-parametrization" rel="noopener nofollow ugc">removing the reparametrization</a> in the tutorial might be of use.</p>
<p>Best regards</p>
<p>Thomas</p>

aulakh83: <p>Tom thank you for providing relevant content. I will try this and will share the result <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

aulakh83: <p>I tried to apply global pruning over my model which is having one lstm layer.  I am getting<br>
UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters().</p>
<p>then i put flatten_parameters in forward of the module. But getting the same. Can u plz help</p>

tom: <p>I think this is inherent in the way pruning is implemented using reparametrization. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=10" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>One way to overcome the error message could be to have a workflow where you do pruning and then remove the reparametrization.</p>
<p>The other option is to make it work with non-flattened inputs, i.e. either</p>
<ul>
<li>to silence or avoid the warning using the Python <code>warnings</code> mechanism or</li>
<li>to disable CuDNN, if the flattening and Python warning overhead is larger than the savings by CuDNN optimizations,</li>
<li>rolling your own JIT fused RNN might be another option, but sadly it has never become as much the standard as one might have hoped.</li>
</ul>
<p>I can see how neither option is terribly attractive but it seems that things are not working together as well as they should.</p>
<p>Best regards</p>
<p>Thomas</p>

aulakh83: <p>ok. Theoretically removing the reparameterization is appealing. so I am trying with the same and Will inform about results when done.</p>

aulakh83: <p>removing reparamterization worked well for me. thanks Thomas.</p>

aulakh83: <p>Next issue i am facing is: if i 90% globally prune the model in one-shot it is pruned approximately 90% with different pruning rate in different layers. But iteratively pruning the model with factor 0.1 for 10 iterations, i am getting the model pruned approximately upto 65.6 %. What is the reason behind it?? I thought it to be pruned approximately 99%.</p>

tom: <p>Are you randomly pruning? If so I would suspect that the pruning is considering all weights (including previously pruned) in the computation: If I randomly set 0.1 of the weights to 0 for 10 iterations, I expect (in lieu of LaTeX support in the forum) <code>sum([0.1 * (0.9**i) for i in range(10)])</code> ~ 65.1% of weights to be set to 0.<br>
This appears to be consistent with the documentation in <a href="https://pytorch.org/tutorials/intermediate/pruning_tutorial.html#iterative-pruning" class="inline-onebox" rel="noopener nofollow ugc">Pruning Tutorial — PyTorch Tutorials 1.9.1+cu102 documentation</a> . If you have some other criterion, maybe it is applied to the original weights. I have to admit that I don’t know how it exactly works from the top of my head.</p>
<p>Best regards</p>
<p>Thomas</p>

aulakh83: <p>I am pruning the model weights using<br>
prune.global_unstructured(<br>
parameters_to_prune,<br>
pruning_method=prune.L1Unstructured,<br>
amount=0.10,<br>
)</p>
<p>for 10 iterations. Your explanation makes sense that it must be considering the pruned weights also. Then how can i make the model not to consider those again in other iterations. One more question how can i calculate the inference time of the model</p>

tom: <aside class="quote no-group" data-username="aulakh83" data-post="16" data-topic="130574">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aulakh83/48/41746_2.png" class="avatar"> aulakh83:</div>
<blockquote>
<p>Then how can i make the model not to consider those again in other iterations.</p>
</blockquote>
</aside>
<p>One thing that might be worthwhile to try is progressively increasing the amount if that matches what you want to achieve.</p>
<aside class="quote no-group">
<blockquote>
<p>One more question how can i calculate the inference time of the model</p>
</blockquote>
</aside>
<p>I don’t have a good answer, but two thoughts:</p>
<ul>
<li>I don’t think unstructured sparsity is something lending itself easily to performance gains. (Though at 90%ish it might just work, but I would not know how to do it.)</li>
<li>I would probably measure time.</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

aulakh83: <p>Thank you Thomas, I started experimenting these thoughts that why took long to reply. But today i got the results,  progressive increasing the amount till i reach threshold worked well and i am able to speed up the model by measuring the time before inference call and after that. My aim was not to have performance gains but just to reduce model size that i am  able to achieve thanks</p>