Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new approach for kernel pruning, by leveraging group-wise sparsity to obtain a compact architecture that can be parallelly computed. The approach first clusters the kernels based on the defined measurement via recent findings in the lottery ticket hypothesis (LTH), and then develop a simple and efficient greedy approximation algorithm for filter pruning.
main_review: 
Strengths:

1. The proposed approach is technically sound in general. Each part of the methodology (i.e., clustering, finding LTH, and pruning) is well motivated. The idea to leverage group convolution for kernel pruning is practical.

2. The paper is well written and easy to follow.

Weakness:

1. The empirical evaluations are not self-contained, somewhat weak, and incomplete:
  - Table 1 is not referred to. More experimental results and ablation studies should be included in the main text.
  - The performance gain on CIFAR-10 can be minor compared with existing baselines, e.g., ResNet-32 has 92.82% acc, which is even worse than other baselines.
  - The accuracies of models before pruning on TinyImageNet are apparently lower than other baselines, which is unfair. 
  - There are no results on ImageNet.

2. More ablations are in need to fully verify the proposed approach.
  - It is not verified if the findings in Zhou et.al.(2019) still hold in filter pruning. I wonder if the magnitudes of grouped filters still correlate with the final accuracies.
  - The effect of hyper-parameters $\alpha$ and $\beta$ is seldom discussed in the experiment.

3. The proposed pruning pipeline involves multiple steps, which is kind of complex to me. One needs to group the filters and find the optimal group division, identify the lottery tickets and formulate pruning as a maximum edge-weight connected subgraph problem. What is the time consumption for each step in such a pipeline, aside from network fine-tuning after pruning?


Other comments:

1. "we expect such relationship will expand to a filter-level where filters with large TMI scores may help on accuracy retention and therefore be deemed “more important.”: It would be more convincing if such assumptions are empirically verified first before moving on to the next steps.

2. "we must first hypothesize a subset of grouped kernels that are most “distinctive” from each other, which helps to preserve the representation power of the original filter group." : Is this empirically verified? In other words, what if you switch the strategy to keep the least "distinctive" grouped kernels?

3. Have you tried the practical speed-up of pruned networks via grouped convolutions?
summary_of_the_review: Overall this is a technically sound paper. However, my major concerns are two-fold: 1. The proposed methodology is over complicated in my view, yet there is little gain in accuracy. 2. The experiment still needs to be enriched to verify the proposed solution from multiple aspects.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The proposed work focuses on kernel pruning. The core idea revolves around grouping filters using a similarity criterion and removing common convolutional kernels with the group. The proposed work explores optimal grouping schemes for filters and after pruning unwanted filters, the retained filters are restructured and the network can be fine-tuned to recoup prediction accuracy. 
main_review: Strengths
- The core idea and contribution is stated and explained clearly. This helps the reader assimilate the paper quickly.

Weaknesses
- There are multiple instances of language issues throughout the manuscript e.g., Pg. 3, Section 2, Paragraph 2, first sentence. I encourage the authors to correct them to help the reader process the work seamlessly.
- Could the authors clarify the meaning of "Permuting the retained filters" ? Is it similar to replicating the connections of common input nodes across multiple groups? If so, doesn't it add more FLOPs to the forward pass?
- The nomenclature of dense, sparse and densely structured have a lot of overloaded meaning. It is a little difficult to parse the exact meaning of these terms. I encourage the authors to rethink the terminology and possibly clarify them.
- Pg. 3, last bullet point: The term "cleaner" and other abstract terms can be avoided to maintain focus on the argument at hand.
- A common theme across the Related works and other sections  is the highlighting of computational cost involved in complex pruning heuristics. Could the authors discuss the time-complexity involved in a) the TMI score generation at $W_t$ for the winning ticket and b) TMI score generation across the entire DNN (l layers) and c) TMI scores for multiple groupings, which itself is computationally expensive?
- If we factor in the performance of unsupervised clustering approaches from the Appendices, the Table maintains the same numbers for performance, params and FLOPs across all tested methods. Could the authors clarify these results?
- Could the authors clarify the magnitude increase formula , from Pg. 5 and subsequently used everywhere?
- When discussing the choice of windows for lottery tickets, the authors mention they lookup values from Renda et al.(2020). However, this leaves the choice of windows for new dataset/DNN architectures open. Could the authors discuss more on how they would approach such a target?
- Mathematical notations for k  have been overloaded (Alg. 1 and previously). Also, the variable names have a mix of mathematical notation and pseudo-code. I encourage the authors to reassess their choice of variable names.
- Could the authors discuss their choice of layer pruning ratios ?
- Could the authors clarify the special fine-tuning setup discussed in Section 3.2.2 and the deviation of other baselines from it in more detail? This could add more strength to the argument of loss computational overhead.
- If the pruning and retaining settings mirror the training setup, the terminology of fine-tuning might be incorrectly used. Could the authors clarify if their training and pruning setups are the same?
- The results posted in Table 1 are lower than R. Yu, A. Li, C.-F. Chen, J.-H. Lai, V. I. Morariu, X. Han, M. Gao, C.-Y.Lin, and L. S. Davis, “Nisp: Pruning networks using neuron importance score propagation,” in IEEE Conference on Computer Vision and PatternRecognition, 2018, N. Gkalelis and V. Mezaris, “Fractional step discriminant pruning: A filter pruning framework for deep convolutional neural networks,” in IEEE International Conference on Multimedia & Expo Workshops, 2020 and M. R. Ganesh, J. J. Corso, and S. Y. Sekeh, “Mint: Deep network compression via mutual information-based neuron trimming,” in IEEEInternational Conference on Pattern Recognition, 2020. Could the authors update their results and compare against methods with high performance.
- Overall, the amount of results and discussion presented in the main body of the paper seem insufficient. I encourage the authors to re-organize their manuscript to further support their claims and analyze their results.

After Rebuttal
- I appreciate the author's clarifications on a number of the comments posted.
- After considering the revised version of the paper, I have updated my scores across multiple metrics.
- The update of Table 1 to include a number of results helps emphasize the applicability of the proposed work to a number of datasets. The inclusion of "Amc: Automl for model compression and acceleration on mobile devices" , a top performing method on ResNet50-ILSVRC2012 would help strengthen the comparisons drawn from Table 1. 
- At a higher level, a pruning methodology's performance is based on pushing the limits of performance at extreme sparsity levels. 
In this context, comparisons to the most sparse models mentioned in works, like GAL, DHP, etc. would help highlight the importance of the current work further. 
summary_of_the_review: The method proposed in the work provides an alternative pruning approach that should help decrease the computational complexity. However, overheads involved in repeated formulation of groups and TMI evaluation need to be discussed and compared. The results presented in the manuscript do not possess sufficient depth and are missing analyses.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a novel approach in the well-established space of structured pruning methods (kernel-pruning) post-training, which primarily consists of three stages: (i) clusters the filters in a convolution layer into predefined number of groups, (ii) prune the unimportant kernels from each group, and (iii) permute remaining kernels to form a grouped convolution operation and then finetune. This paper is motivated by the idea of achieving grouped convolution which can be efficiently deployed on low-compute end devices, and definitely has huge practical implications.
main_review: Pros:
1. Dense Structured pruning: this paper has addressed a well-known and crucial problem of kernel-pruning in a structured way yielding a dense pruned network. They have empirically shown how this approach can help improve the representation power of a pruned network (delta increase in performance) and can be easily deployed (less FLOPs and hardware-friendly due to grouped-convolution).

2. Improved Accuracy without Iterative Fine-tuning: one important contribution of this paper is the exhaustive empirical study of their approach with related works (Table 2) with respect to i) need for iterative pruning or ii) special finetuning. Overall, their results imply that the proposed approach of grouped kernel pruning indeed improves the generalisation, but the FLOPs improvement is still either negligible or at par performance as compared to the baselines ( SFP [He et al., 2018] and 3D [Wang et al., 2021b].

3. Enhanced Clustering Scheme: Another important contribution is the better evaluation of filter-clusters by taking filter importance into account. Before this, it was done using a classical clustering Silhouette metric which doesn't take any network context (filter importance) into account.

Cons:
1. Filter Importance and Accuracy Retention? Authors claim that their method of filter-clustering takes accuracy retention into account by following two themes: i) filter importance and ii) "balance between the filter groups" (not all important filters should be aggregated in the same group). However, they have not proven/established any kind of direct relationship between accuracy retention and these aforementioned themes (especially for convolutional filters). For example, any experiment to prove the empirical relation between their defined metric TMI score (to evaluate cluster quality) and accuracy would be enough to prove the point.

2. Loopholes in notations (Algorithm 1): The algorithm 1 on page 7 maintains the matrix M of size (C_in-1 X C_in at Line 3) where C_in represents the number of kernel channels. As per the understanding, it should be of size C_in X C_in. Also, following the proposed algorithm, once a pruning ratio is reached, the algorithm should break the loop instead of continuing (Line 14).

3. Technical issue in algorithm 1: The greedy approach proposed by the algorithm always picks the next best kernel K given the current set of selected kernels (nodes) in a filter-group (graph). 
Example: Suppose a filter-group has 6 kernels A,B,C,D,E,F with the distance-matrix (D) as defined below: D(A,B) = 100, D(C,D) = 100
D(A,C) = D(A,D) = D(B,C) = D(B,D) = -10
D(E,F) = 0
All other kernels have a distance value of -2. In such a scenario, if we try to select 4 kernels of maximum value set (subgraph), the answer should be (A,B,C,D) while the algorithm's greedy approach will never take these 4 kernels (nodes) together. It would be better if the authors can clarify this corner case.

Typos:
1. "Abstract" at Page 1: "structured pruning due to it" looks like an incomplete sentence.
2. "Introduction" at Page 1: "methods lie in between" looks grammatically incorrect.
3. "Related Work" at Page 3: Line "Our method is inspired by group convolution, a is well convolutional architecture" has issue in sentence formation

summary_of_the_review: Overall, I vote for borderline rejection (score 5). Specifically, I liked the idea of introducing a new metric of ticket magnitude increase score (TMI score) to cluster the filters in the first stage based on the "lottery ticket hypothesis". But there are some unproven claims and inconsistencies as pointed out in the cons section.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper consults the empirical findings on the Lottery Ticket Hypothesis and proposes a kernel pruning framework with densely structured outputs. The proposed method seems to give noticeable improvement over SOTA methods with a rather elegant design.
main_review: Strengths:
* The paper addresses an important research problem in the field of model compression as most kernel pruning methods suffer from the curse of sparsity and are therefore not computational friendly. 
* The paper is well written. The intuition behind each proposed component of the framework (e.g., the proposed TMI score) is well discussed and most of them seem straightforward.
* The ablation study on different filter clustering schemes is insightful.

Weaknesses:
* The experiment setup is somewhat limited. The experiments are conducted on CIFAR10 and TinyImageNet only, so it is unclear whether it would work well on datasets like ImageNet. Moreover, only ResNet experiments are reported. It would be more convincing if the results on other popular CNN architectures are also reported (e.g., densely connected blocks and NAS-searched blocks like EfficientNet).
* As finding winning tickets is sensitive to the learning rate, is there any empirical justification for the learning rate decision?
* Although the proposed greedy approximation algorithm seems to be cost-efficient, it would be great to show the training time compared to baselines.


summary_of_the_review: In summary, I feel that although the proposed method seems to be novel, the evaluation part is not yet convincing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present a new metric to determine the similarity between different grouped kernels and prune the unimportant $k\times k$ slices out of a 3D filter. They utilize the Lottery Ticket Hypothesis and propose a greedy search strategy to overcome the challenge of a huge search space. The experiment results show that the one-shot scheme can still be comparable to two-stage leading methods on the CIFAR-10 dataset, with a slightly lower training cost. The empirical success of this paper may serve as proof of the existence of the Lottery Ticket Hypothesis.
main_review: pros.
1. The proposed method seems simple yet effective, though it has not been fully evaluated on the large dataset.
2. Figure 1 clearly illustrates the implementation of TMI-GKP. We may easily deploy it on general-purpose devices like CPU/GPU, which benefits from fine-grained and coarse network pruning schemes simultaneously.
3. The technique part of this paper is easy to follow. The combination of the Lottery Ticket Hypothesis and structured pruning seems interesting. This paper may shed some light on this new direction.

cons.
1. The main concern is the experiment part of this manuscript. Since post-training grouped pruning has been frequently discussed in the recent literature (in static mode or dynamic mode) [1,2,3,4,5], a direct comparison with those methods is more appropriate. Besides, ImageNet has been a common setting in recent works. Though I fully understand the authors are stuck with limited computing resources, a solid result on ImageNet can be more convincing than CIFAR-10.
2. I expect the real speedup reported in Table 1 to verify the superiority of learned group convolutions.
3. It would be better to include the real time cost of the "Ticket Magnitude Increase scoring system" and "Qualifying cluster results with TMI scores". It seems that the computing complexity of those steps are $\mathcal{O}(\prod_{i=0}^{K-1} \binom{C_{out}-i\cdot\frac{C_{out}}{K}}{\frac{C_{out}}{K}})$ where $C_{out}$ is the number of filters, $K$ is the number of groups, which is still very large for $C_{out}=512$.
4. I have checked section A.3.1. The proposed "TMI Clustering w/Greedy" with $+0.14$% on CIFAR-10 is somewhat incremental when compared with K-PCA+k-means.
5. It would be better to verify the effectiveness of Algorithm 1 via a toy experiment, such as setting the channel number to $16$. The brute force solution to this problem can be obtained and serves as a baseline.
6. Since the authors apply the Lottery Ticket Hypothesis to structured pruning, I expect an ablation study under the same setting as *winning ticket* or *weights rewinding* to prove the existence of the ticket. It is hard to evaluate the correlation between $TMI$ and the Lottery Ticket Hypothesis, given the existing experiments.  
7. According to Table 1, GAL seems to outperform TMI with $12$% Params and $38$% FLOPs?
8. There seems to be no ablation study on $\beta$ (in Eq.(5)). Did I miss anything?

ref:
* [1] Dynamic Group Convolution for Accelerating Convolutional Neural Networks. ECCV2020
* [2] Fully Learnable Group Convolution for Acceleration of Deep Neural Networks. CVPR2019
* [3] Building Efficient Deep Neural Networks with Unitary Group Convolutions. CVPR2019
* [4] Differentiable Learning-to-Group Channels via Groupable Convolutional Neural Networks. ICCV2019
* [5] Efficient Structured Pruning and Architecture Searching for Group Convolution. ICCVw2019

summary_of_the_review: In general, I believe this work is somehow different from existing works but the current draft makes it challenging for these ideas to reach their full potential. The authors are encouraged to address the weaknesses above.
review_title: null

Meta-review: This paper studies structured pruning methods, called kernel-pruning in the paper which is also known as channel pruning for convolutional kernels.  A simple method is proposed that primarily consists of three stages: (i) clusters the filters in a convolution layer into predefined number of groups, (ii) prune the unimportant kernels from each group, and (iii) permute remaining kernels to form a grouped convolution operation and then fine-tune the network. Although the novelty of the method is not high, it is simple and effective in experiments after the supplementary sota results in the long rebuttal. Majority of reviewers increase their ratings after the rebuttal (though one reviewer promised this but forgot to act), while some reviewers have concerns on the fairness to other authors by adding lots of new results in unlimited rebuttal and refuse to check more. In terms of the top end of performance, a reviewer thinks that "the authors haven't quite exceeded the results from existing works ("Discrimination-aware channel pruning for deep neural network" and "Learning-compression” algorithms for neural net pruning" for CIFAR-10 and many others on ImageNet)". In all, this work indeed lies on the boundary. After a discussion with other committee members, we recommend the acceptation of this work, if the authors could incorporate all the new results in rebuttal and get the reproducible codes released in the final version.
Decision: Accept (Poster)