Venue: AutoML 2022 Track Main

Reviews:

final_score: rating: 1: Strong reject
review_content: summary_of_contributions: N/A for reproducibility reviewers
overall_review: N/A for reproducibility reviewers
review_summary: N/A for reproducibility reviewers
review_title: null

final_score: rating: 3: Marginally below the acceptance threshold (use sparsely)
review_content: summary_of_contributions: The paper proposed an algorithm to learn a non-uniform pruning policy for compressing the deep neural network and maintaining good performance under adversarial attacks. Pruning policy is learned by reinforcement learning where each state is the parameter features of each layer, i.e., convolutional kernel sizes, and the action space is the range for compression ratio. With the learned policy, the paper shows that they can achieve consistent improvement for both clean images and perturbed images compared to the uniform pruning policy, where the compression ratio of each layer is usually set manually. 
overall_review: Pros:
1) The paper is easy to follow and the motivation is clear.
2) Compared to other non-uniform robust-aware methods, the proposed pruning policy can achieve great improvement for both clean and perturbed image accuracy.

Cons:
1) I am not sure how novel is the method regarding to AutoML.
2) I think it will be important to see how is the performance when we naively combine adversarial training with existing automatically pruning methods.
review_summary: I lean to reject the paper but I would like to change my mind.
review_title: null

final_score: rating: 4: Marginally above the acceptance threshold (use sparsely)
review_content: summary_of_contributions: The paper claims that a *non-uniform compression* improves the final accuracy of the pruned models on the original data and adversarially perturbed data. In this regard, the authors propose a Reinforcement-Learning-based method to search for a non-uniform pruning strategy that will replace the uniform compression ratios used in other state-of-the-art methods afterward. For each layer, the RL agent estimates the compression ratio to use following the Order of Weight Magnitude criterion such that the global compression rate is constraint is satisfied. The reward for the agent is computed by evaluating the benign and robust accuracies of the pruned model obtained to reach an optimal trade-off between both.
The strategy found is implemented on two state-of-the-art robust pruning method, HYDRA and Robust-ADMM, and evaluated on the CIFAR-10 dataset in the main paper and SVHN and ImageNet datasets in the Appendix. Two types of pruning are compared, channel and weight pruning, for multiple architectures. 
overall_review: ### Positive

- The proposed idea of using a non-uniform compression rate is simple and intuitive. Indeed, some parts of the network might be more important than others. As the experiments has shown in the paper, the first layers of convolutional networks seem to be more important to keep. 
- When channel pruning, using a non-uniform strategy clearly improves the performance and robustness of the different prunning algorithm used. It can also prevent the algorithm from completely damaging the model when pruning uniformly.
- The experiments are done on multiple datasets (CIFAR-10, SVHN and ImageNet), for different global compression rates and using several architectures.

### Negative

- The most important experiments, those on the ImageNet dataset, are relegated to the Appendix when they should clearly appear in the main paper since it corresponds to the most difficult setting. Only the experiments on CIFAR-10 are presented in the main paper.
- The importance of a non-uniform strategy is overstated in the paper. On the ImageNet dataset on the weight pruning setting, which seems to be the most important setting as it is the one considered in all previous works, the effect of a non-uniform strategy is less clear.  
- The clarity of the paper could be improved with figures explaining the whole process and illustrating the algorithm to find the compression strategy to use.
review_summary: The main idea of the paper could inspire future work on non-uniform pruning algorithms, as the experiments show promising results. However, the improvement is not so clear on more difficult settings and the claim should be toned down. The clarity of the method could be improved with detailed figures.
review_title: null

final_score: rating: 4: Marginally above the acceptance threshold (use sparsely)
review_content: summary_of_contributions: This paper shows that employing non-uniform compression strategies allows conjoining model compactness and adversarial robustness. To realize this, the paper develops an RL algorithm to search for layer-specific, non-uniform compression ratios. The empirical studies are thorough and convincing. This paper is problem-oriented, and the technical novelty, honestly, is limited.

overall_review: - Positive aspects:
1. This paper leverages AutoML strategies (i.e., RL-based hyper-parameter search) to solve the network pruning problem.
2. The paper may bring insights and exhibit impact on the network pruning community.

- Negative aspects:
1. The technical novelty is limited.
2. RL-based searching may be inefficient at scale.
review_summary: The paper solves a seemingly important problem with well-studied AutoML strategies, so I am learning to accept this paper but my confidence is low.
review_title: null

Meta-review: metareview: After the discussion phase, the author's resolved (nearly) all problems regarding reproducibility (although the score in the review was not updated). In addition, one reviewer also increased the score to 5 in their reply. In view of the strengths and importance of the topic, I recommend acceptance.
confidence: 5: The area chair is absolutely certain
Decision: Accept