Global-local structured pruning

username: yan_minou
cooked: <p>Hello,</p>
<p>I’m struggling with the implementation of a global-local structured pruning method. I read its explanation in this paper <a href="https://openaccess.thecvf.com/content/WACV2024/papers/Gupta_Torque_Based_Structured_Pruning_for_Deep_Neural_Network_WACV_2024_paper.pdf" rel="noopener nofollow ugc">paper</a>. In the global pruning, an entire filter is removed by comparing the sum of absolute weights of each filter (further normalized by the shape of filter) globally across all the filters throughout the model. In the local pruning, a minimum filter rule is imposed, which ensures that every conv layer has at least the minimum number of filters. One single MF value (equals to 5) is applied to all the layers of the model.</p>
<p>I tested my implementation with a pruning ration of 50% on a pre-trained vgg19 model on cifar10 test dataset and yielded an accuracy of 89% with the original model but 10% with the pruned model. However, it was reported in the paper that accuracy of the pruned model is 68%.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3c64e036c4389051e369903256a96d520be1bd78.png" data-download-href="https://discuss.pytorch.org/uploads/default/3c64e036c4389051e369903256a96d520be1bd78" title="Screenshot from 2024-02-16 09-54-29"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3c64e036c4389051e369903256a96d520be1bd78_2_539x500.png" alt="Screenshot from 2024-02-16 09-54-29" data-base62-sha1="8CgLY8iWYlZmgzWJ0zMFw6Kax2E" width="539" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3c64e036c4389051e369903256a96d520be1bd78_2_539x500.png, https://discuss.pytorch.org/uploads/default/original/3X/3/c/3c64e036c4389051e369903256a96d520be1bd78.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/c/3c64e036c4389051e369903256a96d520be1bd78.png 2x" data-dominant-color="E7E7E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-02-16 09-54-29</span><span class="informations">670×621 57.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
(Figure2. (a)).</p>
<p>Here is my implementation:</p>
<pre><code class="lang-auto">import torch
import numpy as np

# Define the percentage of filters to prune
pruning_percentage = 0.5

# Collect all the filter norms in the convolutional layers
filter_norms = []
for name, module in original_model.named_modules():
    if isinstance(module, torch.nn.Conv2d):
        # Compute the sum of absolute weights divided by the filter shape
        f_shape = module.weight.data.size()
        f_shape = f_shape[2] * f_shape[3]
        weights = module.weight.data.view(module.weight.size(0), -1)
        f_norm = torch.sum(torch.abs(weights), dim=1) / f_shape
        filter_norms.extend(f_norm.cpu().numpy())

# Determine the threshold for pruning
threshold = np.percentile(np.array(filter_norms), pruning_percentage * 100)

# Prune the filters in the convolutional layers
for name, module in original_model.named_modules():
    if isinstance(module, torch.nn.Conv2d):
        # Compute the sum of absolute weights divided by the filter shape
        f_shape = module.weight.data.size()
        f_shape = f_shape[2] * f_shape[3]
        weights = module.weight.data.view(module.weight.size(0), -1)
        f_norm = torch.sum(torch.abs(weights), dim=1) / f_shape

        # Compute a mask based on the threshold
        mask = (f_norm &gt; threshold).float().view(-1, 1, 1, 1)

        # Ensure at least five filters are kept
        if mask.sum() &lt; 5:
            _, indices = torch.topk(f_norm, 5)
            new_mask = torch.zeros_like(f_norm)
            new_mask[indices] = 1
            new_mask = new_mask.view(-1, 1, 1, 1)
            mask = new_mask

        # Apply the mask to the weights
        module.weight.data.mul_(mask)

# Save the state dict of the pruned model
torch.save(original_model.state_dict(), './vgg19/global_pruned_cifar10_vgg19.pth')

</code></pre>
<p>I’ll be so happy for your help!</p>