CNN weights pruning methods in PyTorch

username: minhoha
cooked: <p>Hi.</p>
<p>I wanna implement network pruning using PyTorch.</p>
<p>I made a weight histogram to find out pruning point.</p>
<p>And then make weight, which can be pruned by histogram, zero.</p>
<p>At this point I have a few question.</p>
<ol>
<li>
<p>Is there any method to make weights zero? I made a for 2~4 loop… Here is my code</p>
<pre><code> for m in net.modules():
 if isinstance(m, nn.Linear):
     wv_fc = m.weight.data

     if number_wv == 4:
         print("pruning FC1 weights")
         number = 0
         for n in wv_fc:
             for i in range(0, 100):
                 for j in range(0, 64*3*3):
                     if m.weight.data[i][j] &lt;= 0.0404 and m.weight.data[i][j] &gt;= -0.0404:
                         m.weight.data[i][j] = 0
                     print(m.weight.data[i][j])                             
             number = number + 1
         number_wv = number_wv + 1

     else:
         print("pruning FC2 weights")
         number = 0
         for n in wv_fc:
             for i in range(0, 10):
                 for j in range(0, 100):
                     if m.weight.data[i][j] &lt;= 0.0404 and m.weight.data[i][j] &gt;= -0.0404:
                         m.weight.data[i][j] = 0
                     print(m.weight.data[i][j])
             number = number + 1
         number_wv = number_wv + 1
</code></pre>
</li>
<li>
<p>If I done masking, is there any simple method to freeze the zero weight specifically? Not per layer like "reqires_grad=FALSE"<br>
I wanna freeze only zero weights in entire network.<br>
In this context, freeze means that freezed weights cannot be trained anymore.</p>
</li>
</ol>

username: SpandanMadan
cooked: <p>You can access parameters by doing a .parameters()</p>
<p>So, model.children() gives you layers, and for each layer, layer.parameters() gives you access to each parameter. It’s a generator basically so it lets you run in a loop. Set the parameters to 0 and the requires_grad to False for any nodes you want to prune.</p>

username: minhoha
cooked: <p>can you explain with small example?</p>

username: SpandanMadan
cooked: <p>I have a tutorial which shows this in great detail. It doesn’t set the weight to zero and requires grad to False, but shows how to access the parameters. Here’s the github repo for the tutorial - <a href="https://github.com/Spandan-Madan/A-Collection-of-important-tasks-in-pytorch" rel="nofollow noopener">https://github.com/Spandan-Madan/A-Collection-of-important-tasks-in-pytorch</a></p>
<p>In the tutorial go to cell 22 and modify the lines to be.</p>
<pre><code>for child in model.children():
    for param in child.parameters():
        param =  torch.zeros(param.size())
        param.requires_grad = False
</code></pre>
<p>This will basically prune all nodes of the model. To specify which nodes are to be pruned just add if statements in the above code to prune only the required nodes!</p>
<p>Hope this helps!</p>

username: minhoha
cooked: <p>I think it is impossible to configure specific weights. The tutorial what you recommend is also about just layer configuration.</p>

username: minhoha
cooked: <p>I modified the code like that</p>
<pre><code>for child in net.children():
for param in child.layer[0].parameters():
    for i in range(0,16):
        for j in range(0,1):
            for k in range(0,5):
                for l in range(0,5):
                    if param.data[i][j][k][l] &lt;= 0.0404 and param.data[i][j][k][l] &gt;= -0.0404:
                        param.data[i][j][k][l] = 0
                        param[i][j][k][l].reguired_grand = False
</code></pre>
<p>But I cause other error like that</p>
<pre><code>Traceback (most recent call last):
  File "main3.py", line 229, in &lt;module&gt;
    param[i][j][k][l].requires_grad = False
RuntimeError: you can only change requires_grad flags of leaf variables. If you want to use a computed variable in a subgraph that doesn't require differentiation use var_no_grad = var.detach().
</code></pre>
<p>If i use detach(), there is no change…</p>

username: SpandanMadan
cooked: <p>You’ll need to use hooks to get access to intermediate gradients. Read this for an example - <a href="https://discuss.pytorch.org/t/why-cant-i-see-grad-of-an-intermediate-variable/94/2?u=spandanmadan">Why cant I see .grad of an intermediate variable?</a></p>

username: minhoha
cooked: <p>You means that make gradients zero. right?</p>

username: SpandanMadan
cooked: <p>That’s one way to do it. It’s a little inefficient as you’re still computing things just that they’re zero so they don’t add up to the equation. But yes it makes sense because it’s obviously impossible to drop specific elements of a matrix from the computation graph. If you get this to work, do drop a reply below I’m curious if this can be done <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: TonightHitTiger1
cooked: <p>Hi，I have the same problem as you. How do you solve it in the end?</p>

username: AlexKoff88
cooked: <p>Hi, look at what is available here <a href="https://github.com/opencv/openvino_training_extensions/tree/develop/pytorch_toolkit/nncf" rel="nofollow noopener">https://github.com/opencv/openvino_training_extensions/tree/develop/pytorch_toolkit/nncf</a>. There are two pruning methods and quantization support.</p>