Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The work proposes a novel method for network pruning by co-training a network to predict filter importance given model weights. This allows the model learn hidden correlations between the model weights and channel importance. They achieve strong empirical results on pruning resnet networks on CIFAR-10 and Imagenet. The method naturally learns E2E and doesn't require per-layer sparsity values and is tuned by a global cost-weighting/function. To satisfy the requirements of their attention network, they propose a novel activation function.
strength_and_weaknesses: Strengths:
- Achieves very strong possibly SOTA empirical results on CIFAR-10 and Imagenet.
- Proposes novel method for channel pruning using an attention mechanism that allows accounting of layer interdependencies.

Weaknesses:
- Little to no discussion of the stability or robustness of the approach.
- Possibly limited novelty due to the similarities to existing work in pruning and NAS and would benefit from significantly more analysis and ablation of their method and the cost.

Questions:
The paper would benefit from discussion of the cost of pruning compared to competing methods. Do other methods prune/fine-tune the network for as many epochs? How expensive is the score model?
Since the training method is quite lengthy and approaching NAS, it would be quite useful to run the pruning results multiple times to give a better idea of how stable the method is.
How much hyperparameter tuning did you conduct and do you have results from different weightings of the flops/parameter cost? The work would benefit from more discussion of how controllable the sparsity is.
Is a single score model trained used for every layer or are separate networks used per layer?
I don't understand the section on the threshold value. Table 3 shows that you get a more accurate, and sparser network with a value of 0.2 or 0.3. Why was 0.5 chosen?

Nits:
On page 7: "Table 2 compares the performance of PAAM (KQ) to the SOA on ImagNet" is missing Ts
Question, but a bit separate from the paper. Can the channel importance estimator transfer between models or during training or does it only give useful signal when co-trained with the network? Would be a useful area to explore if possible.

clarity,_quality,_novelty_and_reproducibility: Some claims in the paper may be somewhat misleading.

"PAAM can also train and generate a pruned network from scratch in a straightforward, one-stage training process without requiring a pre-trained network. "
This is somewhat misleading since you don't require a pre-trained network since your entire training process is rather more lengthy than a normal training process and ends with fine-tuning the pruned network to convergence.
summary_of_the_review: This work proposes a novel method for structured pruning which achieves significantly better SOTA results on Imagenet and Cifar10. Some aspects may of be limited novelty however, due to similarities to methods explored in other NAS and channel pruning methods. Since the empirical results are quite important, the work would benefit from more exploration of it's robustness and more detailed analysis of the method and training cost. I believe this work is currently marginally below the acceptance threshold, but would be willing to increase my score with additional  analysis, experimental results, and details of the hyperparameter search
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors propose a filter-importance-scoring concept named pruning by active attention manipulation. The proposed method is a one-stage training process training network from scratch without requiring a pre-trained network.
strength_and_weaknesses: Strength:
The authors proposed a one-stage training process training network from scratch without requiring a pre-trained network. The proposed method could achieve good performance.

Weaknesses:
1. There are many activation-aware pruning methods, whose idea is very similar to the proposed method. Thus, I am concerned about the lack of novelty in this paper.
2. The proposed method cannot compare with the latest SOTA methods. In fact, the proposed method cannot achieve the best performance compared with some recent works.
3. From the experimental results, one could see that the activation attention of the proposed method cannot gain much improvement.
4. The ablation study is actually a study of different activations. It is better to add the experimental results of the variant without activation.
clarity,_quality,_novelty_and_reproducibility: The writing of the paper could be improved for a clearer presentation. The novelty is limited since the utilized techniques are very similar to other works.
summary_of_the_review: There are many activation-aware pruning methods, whose idea is very similar to the proposed method. Thus, I am afraid that this work lacks novelty. The experiments are not solid to demonstrate the effectiveness and efficiency of the proposed method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to compute the importance of filters with attention weights. Although the authors utilize existing attention mechanisms (i.e., additive attention and scaled dot-product attention), they properly design a nonlinear activation that considers the range of values and gradients, an alternating training process of model parameters and attention weights, a histogram-based analysis to determine the pruning threshold. Experiments on CIFAR and ImageNet with ResNets show the superiority of the proposed method.
strength_and_weaknesses: Strength
- The idea that utilizes attention mechanisms to compute the importance scores of filters while considering the correlations between them is well-motivated.
- Although the authors used the existing attention mechanisms of previous literature, they introduce several interesting elements to properly exploit them for structured pruning. 
- I think the experimental validation is quite good. The baselines are strong, and the results of the proposed method are either SOTA or competitive.
- I appreciate the visualization of Figure 4 that shows the selected filters after pruning and the related discussions. 
- The paper is well backed-up by a comprehensive supplementary material that contains the in-depth discussion and ablation study on the proposed method as well as the details required for reproducibility. 
- The paper is generally quite thorough and generally feels complete: I believe it is generally ready for publication if it is decided by the reviewers that significance is sufficient to warrant publication.
 
Weakness
- The proposed method was validated with only ResNets. Is this method also effective for small-size or other networks, including MobileNets and VGG? It would be helpful to present additional results with other architectures.
- I think this paper includes only the performance after finetuning, and it is difficult for me to identify whether good accuracy comes from the power of finetuning or the effectiveness of PAAM. It would be good to add the performance before finetuning (but after channel pruning) and compare it with at least one baseline method. 
- My major concern is that the proposed method requires numerous hyperparameters including the number of warm-up epochs, the number of PAAM-training epochs, the number of alternating training epochs, the regularization weighting, and different optimizer schedules for AN/CNN parameters. How did the authors set these values? Although the paper includes quite detailed numbers, it would be very helpful to provide some guidelines to determine hyperparameters.
- I would highly recommend the authors to make their codes publicly available for the research community. Implementing the proposed method with attention networks seems not to be simple, and releasing the codes would be very helpful for reproduction and future studies.

clarity,_quality,_novelty_and_reproducibility: - The main idea using attention mechanisms for filter pruning is intuitive and well-motivated.
- Although existing attention mechanisms are used, interesting elements (e.g., nonlinear activation, alternating training procedure) are introduced.
- The effectiveness of the method is well supported by extensive experiments.
- The implementation and reproducibility concern me a little bit, but I believe this can be relieved if the authors release their codes.
- Typo: 7p sate-of-the-art -> state-of-the-art

summary_of_the_review: In general, I believe this work did a good job in terms of applying attention mechanisms for structured pruning. Although there exists a concern regarding the reproducibility, I believe it can be resolved if the authors can release their codes. The experimental results and analyses are quite good and thorough.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper develops a method for pruning neural networks in which an auxiliary subnetwork learns to predict filter importance scores from filter weights.  After a discussion with substantial engagement between reviewers and authors, two reviewers are positive, while one reviewer maintains a reject rating.  The Area Chair shares some of the concerns raised by Reviewer 12a5 and views the central unresolved issue as follows:

The paper proposes a more sophisticated, yet more computationally expensive, method of learning which filters to prune.  However, almost all of the experimental results show minor differences in performance compared to competing (and possibly simpler) methods.  Review rXzi notes that "results on CIFAR-10 are much less clear" as Table 1 shows marginal benefits over previous publications.  In replying to Reviewer wYHx's comment that "proposed method was validated with only ResNets", the authors show results for VGG, but again there are at best minor improvements with respect to prior work.  Only ResNet50 results on ImageNet (in reply to Reviewer rXzi) hint at a nontrivial improvement.  But, one good result against a backdrop of more minor differences is short of a resounding experimental validation for a more complicated methodology.

With this context, the Area Chair is not comfortably brushing aside Reviewer 12a5's concerns and believes more work is required to build a broader (in terms diversity of tasks and network architectures) and stronger experimental case for the method.
justification_for_why_not_higher_score: Experimental advantages are not sufficiently demonstrated.
Decision: Reject