Global structured pruning

username: Alexander_Shekhovtso
cooked: <p>It’s nice to see the new <code>torch.nn.utils.prune.*</code> module in 1.4.0 which is going to be very helpful!<br>
But only "<a href="https://pytorch.org/docs/stable/nn.html#torch.nn.utils.prune.global_unstructured" rel="nofollow noopener">global unstructured"</a> method is implemented in the module. I think, for real applications better to have “global structured” pruning because it’ll help reduce computation complexity along with parameters number avoiding manual tuning of pruning ratio for each layer.<br>
So should we expect <code>torch.nn.utils.prune.global_structured</code> later?<br>
Is it possible to implement custom <code>global_structured</code> or it is impossible for some reason?</p>
<p>Thanks!</p>

username: Jayant_Parashar
cooked: <p>“Since global structured pruning doesn’t make much sense unless the norm is normalized by the size of the parameter, we now limit the scope of global pruning to unstructured methods.”<br>
This is written in Pruning tutorial. It says that the norm used to prune globally does not take into account the size of the parameter. Thus, it would just remove small parameters.<br>
Check this <a href="https://github.com/JJGO/shrinkbench" rel="nofollow noopener">https://github.com/JJGO/shrinkbench</a>.  Maybe they have global structured pruning.</p>

username: Michela
cooked: <p>You can absolutely implement it (and even contribute it back to pytorch, if you wish). We didn’t implement it in the first release for the exact reason that <a class="mention" href="/u/jayant_parashar">@Jayant_Parashar</a> mentioned, as explained in the tutorial: since most models have different-size layers (with a different number of units or channels), global structured pruning across layers does not have a straightforward definition, logically speaking. Does it make sense to compare the norm of a channel with <code>N</code> kernels of size <code>LxL</code> in a given layer with a channel with <code>n&lt;&lt;N</code> kernels of size <code>lxl</code> with <code>l&lt;&lt;L</code> in another layer?</p>
<p>I mean, the comparison is valid, and you can definitely perform it an implement it as a pruning technique, but is it really a good proxy for importance of those channels? or is it just a measure of the size of their associated weight tensors? It might make more sense to try to normalize these norms by the number of total parameters that go into the computation of each norm, or something like that. Perhaps if you find a good paper that implements global structured pruning, we can see how they do it there, and implement their version of it.</p>

username: mzimmer
cooked: <blockquote>
<p>but is it really a good proxy for importance of those channels?</p>
</blockquote>
<p>You are correct, however I think that the same argument could be put forward in the unstructured pruning case, couldn’t it?</p>
<p>In unstructured global pruning we compare weights in entirely different layers and positions within the network. A small weight in one of the first layers might be much more important than a larger one in one of the last layers. Or am I missing something here?</p>

username: Shashank_Priyadarshi
cooked: <p>Hi all,<br>
So I am trying to add a custom global pruning method by extending PyTorch pruning class. So I already have the mask for each layer stored in the dict where key is the module name like conv1 and the value is the masks for the specific module. But I want to use the torch class so that in future when I sometime the model then also those weights remain 0.<br>
Any idea how can I do it. It will be a great help.</p>