Mutual Information Preserving Neural Network Pruning

Authors: Charles Westphal, Stephen Hailes, Mirco Musolesi
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: structured pruning, model compression, mutual information

Abstract:
Model pruning is attracting increasing interest because of its positive implications in terms of resource consumption and costs. A variety of methods have been developed in the past years. In particular, structured pruning techniques discern the importance of nodes in neural networks (NNs) and filters in convolutional neural networks (CNNs). Global versions of these rank all nodes in a network and select the top-$k$, offering an advantage over local methods that rank nodes only within individual layers. By evaluating all nodes simultaneously, global techniques provide greater control over the network architecture, which improves performance. However, the ranking and selecting process carried out during global pruning can have several major drawbacks. First, the ranking is not updated in real time based on the pruning already performed, making it unable to account for inter-node interactions. Second, it is not uncommon for whole layers to be removed from a model, which leads to untrainable networks. Lastly, global pruning methods do not offer any guarantees regarding re-training. In order to address these issues, we introduce Mutual Information Preserving Pruning (MIPP). The fundamental principle of our method is to select nodes such that the mutual information (MI) between the activations of adjacent layers is maintained. We evaluate MIPP on an array of vision models and datasets, including a pre-trained ResNet50 on ImageNet, where we demonstrate MIPP’s ability to outperform state-of-the-art methods. The implementation of MIPP will be made available upon publication.

Decision: reject
Reviewer scores: 5, 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1730458555550

thread_type:
official_review

<thread_object>
creator: reviewer_znsm

<content>
summary:
This paper introduces MIPP (Mutual Information Preserving Pruning), a structured pruning method for neural networks. The key idea is to preserve mutual information between adjacent layers' activations during pruning by selecting nodes that transfer entropy to the subsequent layer. 
The method operates by iteratively pruning from outputs to inputs, using transfer entropy redundancy criterion (TERC) with MI ordering to select nodes. Comprehensive experiments validate MIPP's effectiveness on both trained and untrained networks.

strengths:
1. The overall writing is clear for effective visualizations and well-structured presentation.
2. The paper conducts a wide range of experiments to validate the algorithm.

weaknesses:
1. Some important references [a] are missing, which makes the novelty of the paper questionable. For example, [a] is also about using the mutual information to do filter pruning, what is the difference? Whether the proposed paper achieve higher performance? Why and How?

2. The compared method is rather old. The authors claim "For models trained on datasets smaller than ImageNet, we compare the performance of our method to SynFlow (Tanaka et al., 2020), GraSP (Wang et al., 2022), ThiNet (Luo et al., 2017) and SOSP-H (Non- nenmacher et al., 2022),". Why not include paper published in 2024 [b].

3. Performance is not good. In [a], the performance on ResNet-50 on ImageNet is much higher than Thinet. Why just compare Thinet in Figure 6?

[a] Enhancing CNN efficiency through mutual information-based filter pruning, Digital Signal Processing 2024
[b] Auto-Train-Once: Controller Network Guided Automatic Network Pruning from Scratch

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730644888357

thread_type:
official_review

<thread_object>
creator: reviewer_ns2s

<content>
summary:
The paper introduces Mutual Information Preserving Pruning (MIPP), an activation-based pruning method that maintains mutual information between adjacent layers in neural networks, ensuring retrainability. Unlike traditional methods, MIPP dynamically selects nodes based on their contribution to information transfer, addressing limitations such as layer collapse and lack of adaptability. Experimental results demonstrate that MIPP outperforms state-of-the-art pruning techniques on various vision models, including ResNet50 on ImageNet, with implementation details to be released upon publication.

strengths:
This paper provides an interesting perspective on neural network pruning. It considers the activations of the downstream layers, which allows pruning on trained and untrained networks; the idea is interesting.

weaknesses:
1. Authors claim the method is compared with state-of-the-art techniques, yet most literature is from before 2022; many recent works, such as PHEW or NPB in Pruning at Initialization, and indeed, there are more works on pruning on trained networks in recent years. I strongly suggest that authors provide more valid reviews of recent works.
2. Although the method is interesting because it works for both trained and untrained networks, the motivation for this is not clear. 
For PaI, tasks aim to find networks before training to reduce training costs, while post-trained pruning aims to preserve the best performance on trained networks.  Is MIPP better than SoTA methods on each side?

questions:
1. Can the author compare MIPP in PaI and post-trained pruning in separate settings with SoTA methods in recent years, such as 2023 or 2024?
2. Most experiments are performed on small datasets and networks; showing the pruning task on more computationally intensive structures or datasets, such as the Efficentnet B7 and Imagenet1K tasks, would be better. As the author claimed, the methods work for trained networks; pruning on a network that is pre-trained in a large-scale dataset such as Imagenet21K could be interesting. 
3. Will this method work in Vision transformers?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730649985921

thread_type:
official_review

<thread_object>
creator: reviewer_7fol

<content>
summary:
This paper proposes a pruning approach based on mutual information between the activations of adjacent layers. The proposed approach has been evaluated on a number of models and datasets.

strengths:
+ The motivation of the paper is clear - global pruning approaches do have their limitations and the proposed approach can effectively avoid those. 
+ The idea of looking at the MI between activations of adjacent layers is interesting. It also makes sense to consider nodes that can maintain such MI. 
+ The theoretical analysis seems to make some good points of the observations.

weaknesses:
- The proposed approach has only been tested on some early architectures. It is not clear how this can be generalized to other models and datasets? 
- It is also not clear if the proposed approach is sensitive to different activation functions.
- The comparison between baselines seems to be quite limited to only a few approaches.

questions:
* Can this approach work on other types of architectures like ViT?
* How it may perform with different activation functions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730792290238

thread_type:
official_review

<thread_object>
creator: reviewer_3cvs

<content>
summary:
The authors propose MIPP to enable real-time pruning, whole-layer pruning and global re-training guarantees for improving the performance of network pruning. Through comprehensive experimental evaluation, they demonstrate that MIPP can effectively prune networks.

strengths:
The authors provide a detailed analysis on the motivation and how the method works, especially on how the mutual information is preserved. And they conduct a lot of experiments to demonstrate the effectiveness of their methods.

weaknesses:
1) Experimental settings and result presentation is not clear. 
 - What is untrained network, trained network, pretrained network meaning in Figure 2 and Figure 4? 
 - What is LC in Figure 3?
 - In Figure 2, for column 3 and 4, seems the proposed method does not perform well significantly than the baselines. 
 - Besides, the latest baseline is year 2022, is there any recent works in 2023 or 2024 to compare?

2) in Figure 4, to my best knowledge, state-of-the-art ResNet50 for ImageNet achieves about 76% accuracy however the proposed approaches can achieve nearly 88% accuracy. Can you explain the settings in detail and what is the percentage of parameters reduced and what is the MACs reduced? 

In summary, it is quite unclear how is the comparisons between SOTA and proposed approaches. Besides, some common metrics in comparisons are missing, e.g., FLOPS, #params, MACs. Besides, the baselines seems out-dated.  If the authors could address my concern, I can improve my rating.

questions:
As mentioned in Weakness, I have posed concrete questions for authors. Thanks.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The submission proposes Mutual Information Preserving Pruning (MIPP), a method to prune filters/nodes in neural networks that aims to preserve the mutual information between adjacent layers in a neural network.
After the inital round of reviews, this submission received scores of 5, 5, 3, 3. The issues raised by the reviewers are summarized in the section below, and were found to be valid by the ACs.
After the rebuttal and discussion, the reviewers remained unconvinced. 
The ACs did not find sufficient reason to overturn the negative consensus.