Network pruning error

Flock1: <p>Hello,</p>
<p>I am very new to this topic but I am trying to prune the model I am working with. For reference, I am using <a href="https://pytorch.org/tutorials/intermediate/pruning_tutorial.html" rel="noopener nofollow ugc">this</a> page. The model is quite big, containing different encoders, ResNet modules, and decoders. So, I’m guessing that I have to prune each network individually (I couldn’t find a reference where the whole model is being pruned together, but please attach some links where it’s being done). The list of different modules are like:</p>
<pre><code class="lang-auto">module.model_enc1.1.weight
module.model_enc1.1.bias
module.model_enc1.2.weight
module.model_enc1.2.bias
module.model_enc1.4.weight
module.model_enc1.4.bias
module.model_enc1.5.weight
.
.
.
</code></pre>
<p>So I’m only taking the <code>module.model_enc1.1.weight</code> using the following code:</p>
<pre><code class="lang-auto">test = netM.module.model_enc1
</code></pre>
<p>where <code>netM</code> contains the model weights (<code>&lt;class 'torch.nn.parallel.data_parallel.DataParallel'&gt; </code>).</p>
<p>So <code>test</code> contains the following model:</p>
<pre><code class="lang-auto">Sequential(
  (0): ReflectionPad2d((3, 3, 3, 3))
  (1): Conv2d(3, 64, kernel_size=(7, 7), stride=(1, 1))
  (2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (3): ReLU(inplace=True)
  (4): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
  (5): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (6): ReLU(inplace=True)
)
</code></pre>
<p>And when I run the pruning method by pytorch<br>
<code>prune.random_unstructured(test, name='1.weight', amount=0.3)</code>, I get the following error:</p>
<blockquote>
<hr>
<p>AttributeError                            Traceback (most recent call last)<br>
 in <br>
----&gt; 1 prune.random_unstructured(test, name=‘1.weight’, amount=0.3)</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/nn/utils/prune.py in random_unstructured(module, name, amount)<br>
851<br>
852     “”"<br>
 → 853     RandomUnstructured.apply(module, name, amount)<br>
854     return module<br>
855</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/nn/utils/prune.py in apply(cls, module, name, amount)<br>
473         “”"<br>
474         return super(RandomUnstructured, cls).apply(<br>
 → 475             module, name, amount=amount<br>
476         )<br>
477</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/nn/utils/prune.py in apply(cls, module, name, *args, **kwargs)<br>
155         # starting from the state it is found in prior to this iteration of<br>
156         # pruning<br>
 → 157         orig = getattr(module, name)<br>
158<br>
159         # If this is the first time pruning is applied, take care of moving</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in <strong>getattr</strong>(self, name)<br>
592                 return modules[name]<br>
593         raise AttributeError(“‘{}’ object has no attribute ‘{}’”.format(<br>
 → 594             type(self).<strong>name</strong>, name))<br>
595<br>
596     def <strong>setattr</strong>(self, name, value):</p>
<p>AttributeError: ‘Sequential’ object has no attribute ‘1.weight’</p>
</blockquote>
<p>How do I fix this? Is there any better way to prune these networks?</p>

ani0075: <aside class="quote no-group" data-username="Flock1" data-post="1" data-topic="88553">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/flock1/48/12220_2.png" class="avatar"> Flock1:</div>
<blockquote>
<p>del is quite big, containing different encoders, ResNet modules, and decoders. So, I’m guessing that I have to prune each network individually (I couldn’t find a reference where the whole model is being pruned together, but please attach some links where it’s being done). The list of different modules are like:</p>
</blockquote>
</aside>
<p>print out test and look at the structure of the module. You’ll need to index test and use name=‘weight’. ‘1.weight’ is not an acceptable parameter name.</p>
<p>Try prune.random_unstructured(test[0], name=‘weight’, amount=0.3) or any other index in place of 0.</p>

jerryzh168: <aside class="quote no-group" data-username="Flock1" data-post="1" data-topic="88553">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/flock1/48/12220_2.png" class="avatar"> Flock1:</div>
<blockquote>
<p>I am very new to this topic but I am trying to prune the model I am working with. For reference, I am using <a href="https://pytorch.org/tutorials/intermediate/pruning_tutorial.html" rel="noopener nofollow ugc">this</a> page. The model is quite big, containing different encoders, ResNet modules, and decoders. So, I’m guessing that I have to prune each network individually (I couldn’t find a reference where the whole model is being pruned together, but please attach some links where it’s being done). The list of different modules are like:</p>
<pre><code class="lang-auto">
</code></pre>
</blockquote>
</aside>
<p><a class="mention" href="/u/michela">@Michela</a> could you take a look?</p>

Flock1: <p>Hi. I got <code>1.weight</code> by running this <code>print(list(netM.module.model_enc1.named_parameters()))</code>. The output is:</p>
<pre><code class="lang-auto">[('1.weight', Parameter containing:
tensor([[[[ 2.5521e-02,  5.2238e-02,  4.7848e-04,  ...,  5.6985e-02,
            5.1901e-02,  5.1235e-02],
.
.
.
.
</code></pre>
<p>And then we have values for <code>1.bias</code>, <code>2.weight</code>, <code>2.bias</code> as mentioned above.<br>
This is the error I got for the code you mentioned:</p>
<blockquote>
<p>AttributeError: ‘ReflectionPad2d’ object has no attribute ‘weight’</p>
</blockquote>

ani0075: <p>Your first Conv layer in the Sequential module is at index 1. Try prune.random_unstructured(test[1], name=‘weight’, amount=0.3). I think that should work. Let me know if it doesn’t.</p>

Flock1: <p>Hi. This worked. Thank you so much. Can you also tell me if there’s some way I can prune such a big network in one go? Or do I need to iterate through every layer and prune it individually?</p>

ani0075: <p>Try this <a href="https://pytorch.org/tutorials/intermediate/pruning_tutorial.html#global-pruning" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/pruning_tutorial.html#global-pruning</a></p>

Flock1: <p>Thank you very much. I’ll check this out</p>

Michela: <p><a class="mention" href="/u/flock1">@Flock1</a> the issue was related to how you were accessing the parameter.  Either do <code>dict(netM.module.model_enc1.named_parameters())['1.weight']</code> or do <code>netM.module.model_enc1[1].weight</code>.<br>
<a class="mention" href="/u/ani0075">@ani0075</a>’s solution is correct (and related to the second option in the previous line of this answer) for when you want to refer to that module/parameter combination for the sake of pruning.</p>
<p>Beyond that, what do you mean by pruning “in one go”? Global pruning will allow you to prune the network by pooling all parameters together and comparing them with each other while deciding which ones to prune. That’s not the same thing as pruning each layer individually, but in an efficient way, without mixing weights across layers. Which one of the two were you interested in?</p>

Flock1: <p>Hi <a class="mention" href="/u/michela">@Michela</a>. Firstly, big fan of your work when it comes to network pruning and ML applications in physics. I work on ML applications for quantum computing and astrophysics. I am so glad you replied.</p>
<p>I think I am looking for Global pruning. I was kinda thinking that pruning each layer might not be an effective way compared to global pruning. I was trying one layer just to see how to go about it since it was my first time. But I want to prune the whole network. Moreover, do you recommend pruning the <code>BatchNorm</code> layer?</p>

Michela: <p>Global pruning is generally more flexible and has empirically been shown to have better performance – though be careful not to let it prune entire layers thus disconnecting your network!<br>
Re: batch norm – that’s a more complicated issue, it depends what you want to achieve. Pruning batch norm params won’t really help you significantly reduce the number of params in the network. But if you prune an entire output, does it make any sense to keep its corresponding batch norm parameters? Btw, on the other hand, some even directly use batch norm to figure out which channels to prune in the respective layer. I’d recommend checking out the literature for this.</p>

Flock1: <p>Thank you. I have one question. Can you elaborate on “entire output”? Do you mean the final layer output or something else. Please let me know.</p>

Michela: <p>I mean an entire output dimension in any of the hidden layers. Batch norm layers compute <code>y = γx + β</code> with parameters <code>γ,β</code> for each normalized <code>x</code> at that layer.<br>
If you pruned the previous layer such that a specific <code>x</code> is now always 0, does it make sense to keep its corresponding <code>γ,β</code> around?</p>

Mallika: <p>Hello! I am facing similar problem when I try to apply global pruning for keyword spotting which utilises the vision transformer model.</p>
<p><strong>The code implemented is:</strong><br>
##################################<br>
if <strong>name</strong> == “<strong>main</strong>”:</p>
<pre><code>entries = globals().copy()
</code></pre>
<p>def prune_ViT(model, prune_ratio):<br>
parameters_to_prune = [<br>
(model.pos_embedding, ‘weight’),<br>
(model.cls_token, ‘weight’),<br>
(model.to_patch_embedding[1].weight, ‘weight’),<br>
(model.to_patch_embedding[1].bias, ‘weight’),<br>
(model.transformer.layers[0][0].norm.weight, ‘weight’),<br>
(model.transformer.layers[0][0].norm.bias, ‘weight’),<br>
(model.transformer.layers[0][0].fn.to_qkv.weight, ‘weight’),<br>
(model.transformer.layers[0][1].norm.weight, ‘weight’),<br>
(model.transformer.layers[0][1].norm.bias, ‘weight’),<br>
(model.transformer.layers[0][1].fn.net[0].weight, ‘weight’),<br>
(model.transformer.layers[0][1].fn.net[0].bias, ‘weight’),<br>
(model.transformer.layers[0][1].fn.net[3].weight, ‘weight’),<br>
(model.transformer.layers[0][1].fn.net[3].bias, ‘weight’),<br>
(model.transformer.layers[1][0].norm.weight, ‘weight’),<br>
(model.transformer.layers[1][0].norm.bias, ‘weight’),<br>
(model.transformer.layers[1][0].fn.to_qkv.weight, ‘weight’),<br>
(model.transformer.layers[1][1].norm.weight, ‘weight’),<br>
(model.transformer.layers[1][1].norm.bias, ‘weight’),<br>
(model.transformer.layers[1][1].fn.net[0].weight, ‘weight’),<br>
(model.transformer.layers[1][1].fn.net[0].bias, ‘weight’),<br>
(model.transformer.layers[1][1].fn.net[3].weight, ‘weight’),<br>
(model.transformer.layers[1][1].fn.net[3].bias, ‘weight’),<br>
# Add more parameters here<br>
(model.mlp_head[0].weight, ‘weight’),<br>
(model.mlp_head[0].bias, ‘weight’),<br>
(model.mlp_head[1].weight, ‘weight’),<br>
(model.mlp_head[1].bias, ‘weight’),<br>
]</p>
<pre><code># Prune parameters
</code></pre>
<h1><a name="for-param-name-in-parameters_to_prune-1" class="anchor" href="#for-param-name-in-parameters_to_prune-1"></a>for param, name in parameters_to_prune:</h1>
<pre><code>#    torch.nn.utils.prune.l1_unstructured(param, amount=prune_ratio)
    
prune.global_unstructured(
    parameters_to_prune,
    pruning_method=prune.L1Unstructured,
    amount=prune_ratio,
)

return model
</code></pre>
<p>def main(args):<br>
config = get_config(args.conf)<br>
model = get_model(config[“hparams”][“model”])<br>
model = model.to(config[“hparams”][“device”])<br>
print(f"Created model with {count_params(model)} parameters.")<br>
prune_ViT(model, 0.3)<br>
for name, param in model.named_parameters():<br>
print(name)<br>
print(“Finished!”)<br>
###################################################</p>
<p><strong>The error is as follows:</strong><br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/utils/prune.py”, line 1097, in global_unstructured<br>
[<br>
File “/usr/local/lib/python3.10/dist-packages/torch/nn/utils/prune.py”, line 1098, in <br>
importance_scores.get((module, name), getattr(module, name))<br>
AttributeError: ‘Parameter’ object has no attribute ‘weight’</p>
<p><strong>and my model defination is as follows:</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/8/482243d0d4dfc40a212aa0850f2f1099fc8ed51e.png" data-download-href="https://discuss.pytorch.org/uploads/default/482243d0d4dfc40a212aa0850f2f1099fc8ed51e" title="Screenshot from 2024-04-04 17-29-00"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/482243d0d4dfc40a212aa0850f2f1099fc8ed51e_2_564x500.png" alt="Screenshot from 2024-04-04 17-29-00" data-base62-sha1="ai7MZ7lWJQ0xKRqZBTFU4ld0DlY" width="564" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/482243d0d4dfc40a212aa0850f2f1099fc8ed51e_2_564x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/482243d0d4dfc40a212aa0850f2f1099fc8ed51e_2_846x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/482243d0d4dfc40a212aa0850f2f1099fc8ed51e_2_1128x1000.png 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-04-04 17-29-00</span><span class="informations">1150×1018 120 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How to fix this error specific to transfomer model?</p>

Mallika: <p><a class="mention" href="/u/michela">@Michela</a> <a class="mention" href="/u/ani0075">@ani0075</a> can you please take a look.</p>

Mallika: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> please take a look.</p>

HDCharles: <p>its saying that the parameter object has no attribute weight. it looks like your parameters_to_prune list isn’t specified correctly.</p>