Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposees a novel method of HARP, which learns both compression rate and mask each layer to achieve a a high reduction rate while maintaining both the clean and adversarial accuracy of original models. Specifically, the method is delicately designed with many skills utilized to implement the non-uniform robust pruning. Experimental results validate the effectiveness of the proposed method.

strength_and_weaknesses: ### Strengths

- The research is promising and practical in the real world. This work meets the need for  compact and adversarial-robust model under safety and computation efficiency critical scenarios. 

- Out of the hypothesis that layer-specific compression strategy is equally important as deciding which connections to prune, this work firstly implements non-uniform robust pruning with well-designed optimization process and firm engineering skills. 

- A range of experiments are neatly organized to empirically prove the correctness of the starting hypothesis in adversarial robust pruning  and demonstrate the superiority of the proposed method. 

### Weaknesses

- There is not much novelty in the fundamental hypothesis in this work. Learnable non-uniform compression strategies have been already have been proven effective in pruning. As more parameters available for optimization, it's intuitively effective for more specific pruning task (e.g., adversarial robust pruning).

- In Sec 4.3, some approaches are tried to explain the superiority of the proposed methods, but it only stucks on listing the found features. The paper could yet be strengthened further by providing more detailed interpretations over the observations.
clarity,_quality,_novelty_and_reproducibility: - The design of $\mathcal{L}_{hw}$ in Eq. 2 may not support your explanations below. Considering a case that all layers' expression rate is $a_t$, then the value of $\mathcal{L}_{hw}$ will be $(L - 1)$ which still encourages lower compression rates when L > 1.

- What 's the meaning '$\langle \cdot \rangle$'  in  Eq. 5? The term inside the angle brackets $\in \mathcal{\Theta^{(l)}}$, and $\frac{\part \mathcal{L}}{\part r^{l}}$ and $r$ are scaler, so how do the '$\langle \cdot \rangle$'  map a $\Theta^{(l)}$-dimension vector into a scaler?
summary_of_the_review: Excellent implementations to achieve adversarial robust pruning based on insight from previous works. The proposed HARP empirically works with great performance. Since the starting hypothesis is already proposed, more detailed intuitive or theoretical explanations are required to make it distinct with the previous works.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a unified model compression to improve model robustness by learning the connection importance for each layer weights. In particular, the authors use a pre-trained model to learn the layer importance to meet a specific pruning budget, and gradually increases the pruning % unless a significant accuracy drop is observed.
strength_and_weaknesses: ## Strengths
=============
1. The paper is well written and organized.
2. Results are inspiring and thorough. In particular, detailed results on large scale datasets like ImageNet is good.
3. Literature survey is well up to date apart from few missing ones.

## Weakness
==============
1. Please provide comparison with DNR [1] method. DNR provides better compression-vs-accuracy trade-offs for smaller and less deeper models like ResNet18, VGG16 (as per robustbench: https://github.com/RobustBench/robustbench).

2. The paper uses pre-trained models to start compression and adversarial fine tuning, while there are existing works that did it from scratch [1].

3. The contribution section is not clear, the authors mention about experiments in the contribution. Please be more clear in stating what is the explicit contribution of this manuscript.

4. The question ***How many and which parameters to prune***, is not exactly correct in my opinion, as the ***how many*** part is governed by the target hardware's parameter support. Also, again, the learning of layer-wise pruning, that is indeed a crucial part of robust compression is being handled earlier [1]. 

5. Connection importance score has been well studied as well in the pruning literature [2-3].

Based on the above four facts, I would really recommend the authors to reconsider the motivation of the current manuscript. This also translates to lack of significant contribution for the paper to cross the border of ICLR acceptance.

6. Please detail about the training cost for HARP, including that of pre-training, to reach a specific parameter budget.

7. The robust compression details are not clear. In particular, whether the authors use traditional PGD-7 attack to create adversary or leverage faster variants like FAT [4], should be mentioned with more clarity.

8. Also, HYDRA is not the SOTA paper on robust pruning starting from a pretrained model.

[1] DNR: A Tunable Robust Pruning Framework Through Dynamic Network Rewiring of DNNs, ASP-DAC 2021.

[2] Snip: Single-shot network pruning based on connection sensitivity, ICLR 2019.

[3] Pruning neural networks at initialization: Why are we missing the mark?, ICLR 2021.

[4] Fast is better than free: Revisiting adversarial training, ICLR 2020.
clarity,_quality,_novelty_and_reproducibility: Clarity: 7/10

Quality: 5/10

Novelty: 3/10

Reproducibility: 8/10.

summary_of_the_review: The paper presents a unified robust pruning framework that can reach SOTA accuracy by honoring the layer-wise connection sensitivity.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new method to prune a robust DNN with layer-wise compression rate and score-based pruning masks. Experimental study shows that, by taking advantage of both two factors, it maintains accuracy while allows for less robustness degradation than SOTAs.
strength_and_weaknesses: Strength
- It is good to see that the non-uniform compression strategy is beneficial also to adversarial (channel) pruning.
- It shows large improvement over prior methods, robust-ADMM, HYDRA, BCS-P, and MAD.
- It contains comprehensive discussion, extensive comparisons and ablation study.

Comments
- As structural pruning is more useful and harder than weight pruning, I would suggest authors move some channel pruning results to the main text and compare with SOTAs.
- It would be better to add the pre-trained model's performance to both Table-1 and Table-2.
- In Table-2, why the variances of HYDRA with TRADES under 99% sparsity are such high? Can authors provide some insights about it?
- Will the authors release the code?
 

clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow. The technical contribution is marginal; the empirical results are surprisingly good.
summary_of_the_review: This paper proposes a method to prune a robust DNN with a non-uniform compression strategy. Interestingly the proposed pruning method outperforms existing ones both in terms of robustness and accuracy.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an algorithm for pruning of robust neural networks optimized under adversarial training objectives. The algorithm relies on simulatenously optimizing the (per-layer) compression rates and importance scores corresponding to the connections  that can be pruned, within a dynamic regularization scheme that balances the two objectives. This way the method allows a layer-wise non-uniform pruning scheme that preserves a whole-network sparsity rate at the end of optimization. Experimental results on CIFAR-10, SVHN and ImageNet show that the approach yields benefits particularly at very high sparsity rates starting from 99% and above.

strength_and_weaknesses: Strengths: The paper is well written and easy to understand. Proposed algorithm scales to ImageNet and outperforms some previous methods at that scale. It is also nice that the approach is compatible with any robust training objective similar to the line of work by HYDRA and BCS-P.

Weaknesses: Details describing adversarial robustness evaluation settings should be also present in the main manuscript rather than only the Supplementary. Current experimental comparisons need to be presented in a more rigorous fashion and also some of the evaluations at very high sparsity levels needs further analysis/clarification (see comments below).



clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is written in easily understandable fashion. Nevertheless, there should be more details/clarifications with regards to the training and attack evaluations settings in the manuscript (e.g., in table/figure captions). 

Quality & Novelty: The proposed approach appears to be motivated by the score optimization based pruning method of HYDRA, combined with the layer-wise non-uniform pruning capability of BCS-P which was shown to outperform HYDRA. While methodologically incremental in that sense, the algorithm appears to yield significant results.

Reproducibility: Sufficient, since the authors provided their code and resulting evaluation outputs in the supplement. However some of the documented results that correspond to BCS-P and MAD comparisons do not really contain the evaluations presented in Tables 2 & 3, especially the evaluations at very high sparsity 99.9%.


summary_of_the_review: Generally I am convinced that the algorithm is neat and it indeed appears to be more powerful than R-ADMM (2019) and HYDRA (2020) type of methods that are more restricted to layer-wise uniform pruning percentages. However there are some ambiguities that needs to be clarified and I list my points below:
- Overall, all figure comparisons should be also performed with a stronger attack than PGD-10 (which can be highly misleading here). Assuming that models were trained with 10-step PGD for any robust training loss as well, attack evaluations should be at least presented with 20- or 50-step PGD with 5 or 10 restarts. Going further, since the authors have involved AutoAttack evaluations, one could keep all evaluations simple by presenting all results in that setting too. For instance, can the authors present robust accuracies in Figures 1 2 and 3 under AA? 
- Attack configurations are major details that should be part of the main manuscript (at least summarized). Looking at the figure and table captions/legends, it is not possible to see with which objective the models were trained with (e.g., how many PGD steps), and what is the evaluation setting of the attack (i.e., equal number of PGD steps used during training?). For instance in Figures 2 and 3, robust training loss and attack evaluation settings are not clearly annotated.
- Authors already state in their paper that BCS-P (2021) previously proposed a non-uniform compression strategy that outperformed HYDRA and R-ADMM. This already makes BCS-P a more relevant comparison baseline for HARP to outperform at all categories, but those evaluations appear shallow (only a single VGG16 model under the TRADES objective is studied). Can the authors extend these results on page 7 with a e.g., ResNet-18 and other robust training objectives? In that vein, for instance, authors can also include BCS-P in the plots of Figure 1.
- The provided ImageNet model checkpoints by the authors that correspond to the Table 5 of the HYDRA paper, do not match with the results in the authors submission (Table 3). It appears like using the same dense model with Free-AT (60.25 clean acc), HYDRA achieves a different model at the end for e.g., 99% sparsity? Can the authors verify/discuss this?
- All results compared at 99.9% appear like other methods could not get to work at that level of sparsity at all (i.e., trainings are early terminated as all clean accuracies are at chance level). At the right halves of Table 1 and 2 all accuracies are ~10-30%. How much in depth did the authors explore these methods (such best attempt details can be described in the Supplementary)? Similarly in Figure 2a and b, BCS-P appears to yield a randomly initialized 10% performing network. How much did the authors explore this method?
- It appears like HARP necessitates a dense model to be adversarially trained in the first place, and moves on to a compute-heavy adversarial pruning phase afterwards. Given these limitations, a brief discussion of all compared methods in terms of training time and computational cost would be necessary, until one achieves the final sparse model with an associated clean/robust accuracy.
- Can the authors provide a simulation/visualization of (possibly occurring) robust overfitting during the final stages of pruning at very high sparsity, while adapting gamma?
- Figure 3 caption has a typo, HYDRA -> HARP. Also I believe comparisons to MAD would perhaps look better in a short table, since the available data points are very sparse.
- The x-axis scale in Figure 1 plots looks a bit off?
- There is a figure in page 4 without label/caption/number.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a new pruning method for adversarially robust neural networks that learns the layer-wise importance scores and compression rate. Specifically, the authors claim that the method balances the two objectives of achieving the adversarial robustness and the compactness of the resultant network by introducing a step-wise regularization weight into the training that dynamically adjusts the focus of the learning process from favoring robustness to achieving the target compression rate. The reviews are divergent, with one accept (LsE9), two marginal accepts (phem, GDVf), and one reject (j8jC). The reasons for acceptance are (i) significant experimental results of the algorithm that well scales to ImageNet, (ii) good compatibility to various adversarial training objectives, and (iii) extensive comparisons among important baselines, including those specifically pointed out by reviewers. The reasons for rejection include (i) incremental methodological advance (e.g., score optimization and layer-wise non-uniform pruning), (ii) requiring a pre-trained model, and (iii) missing comparison to a specific prior work [1]. 

[1] S. Kundu, M. Nazemi, P. A. Beerel and M. Pedram, “DNR: A Tunable Robust Pruning Framework Through Dynamic Network Rewiring of DNNs,” 2021 26th Asia and South Pacific Design Automation Conference (ASP-DAC), 2021, pp. 344-350.
justification_for_why_not_higher_score: Incremental methodological advance.
justification_for_why_not_lower_score: This paper presents significantly improved robustness performance in various settings, including an extreme case scenario (i.e., 99.9% sparsity) and positive results in a larger scale experiment (i.e., ImageNet). AC also believes that questions and concerns raised by reviewers are mostly addressed in the author’s responses during the rebuttal period, although some reviewers could not make follow-up discussions after their initial review.
Decision: Accept: poster