Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper a new adversarial training method to improve the robustness of deep learning classifiers in the field of computer vision. To do so, the authors derived a new loss function, which is the surrogate of an upper bound of the robust risk. Specifically, they point out the differences and connections between the proposed method and previous adversarial training method. Experiments on three datasets (CIFAR10, F-MINIST, SVHN) show that the proposed method outperforms other baselines in terms of clean classification accuracy and robust accuracy. Ablation studies are done to show the effect of different parts of the loss function. They also show that the proposed method can be combined with other adversarial training techniques, such as extra data, to further improve the performance. Finally, experiments on CIFAR10 show that the proposed method is helpful to improve the fairness of the classifier compared to TRADES (which is an important baseline).
strength_and_weaknesses: Strength:

- Adversarial robustness is an important security issue in the field of deep learning. The proposed method pushes the SOTA performance of adversarial training to a new level.
- The proposed loss function is derived with a theoretical support.
- Extensive experiments show the empirical advantages of the proposed method from different aspects.

Weakness:

- The novelty of the method is ok, but it is similar to previous methods like MART. The author does point out the differences between the proposed method and other methods, so should not be a big problem.
- It's good that for table 1 and table 2 the results are based on 3 runs with standard errors given, but for most results, the improvements seem marginal.
- CIFAR10, F-MNIST, SVHN are all relatively small datasets. Does the method also perform well on larger dataset?

clarity,_quality,_novelty_and_reproducibility: Clarity:

- The paper is clearly written and easy to follow.

Quality:

- The quality is good. The claims are clear and experiments are solid.

Novelty:

- Somewhat novel though similar to some previous methods.

Reproducibility:

- Code and implementation details are provided, so the reproducibility should be good. Though, I did not run the code to test.
summary_of_the_review: Overall it's a good paper with with theoretical justification and experimental support. Though, there are some weakness in terms of novelty and the limitation of the datasets used, it is a paper above the margin.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work proposes a simple, effective, and theoretically inspired regularization to enhance the robustness of DNNs agains adversarial attacks.
Extensive experimental results were carried out showing the effectiveness of the proposed approach in providing robustness enhancements. 
strength_and_weaknesses: This appear has several strengths:

- The paper is is well motivated and the added regularizer is theoretically inspired.

- The experimental analysis show the consistent improvement of the proposed method over previous art.

- The paper is well-written. Further, the contributions of this work is placed properly within the literature.

- The wide broad of the empirical results shown in this paper covers many interesting aspects such as combining the proposed approach with AWP, and increasing the fairness.


There are few weaknesses that I hope to be addressed during the discussion:

- While the paper is generally well-written, there few parts that require small adjustments. For example,
In caption of figure 1: “ We exclude MART from the figures because its performance is too bad”

- Generally, the robustness improvements that ARoW provide is marginal. Would the proposed method improve the state-of-the-art model from [A]?

[A] Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples, 2021.
clarity,_quality,_novelty_and_reproducibility: The paper clearly states its motivation, contributions, and places itself within prior art.
summary_of_the_review: There several aspects that I like about this work such as the theoretical motivation, the extensive experimental evaluation, and the writing.
However, there are two concerns that  I hope to be addressed in the discussion period.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a modification of TRADES, one of the most popular algorithms to obtain adversarially robust classifiers, to improve its performance: in particular, the regularization term to achieve robustness is weighted to penalize more the training examples which are less robust. In the experimental evaluation on several datasets, the proposed method, ARoW, is shown to achieve better robustness that existing methods, while preserving higher standard accuracy.
strength_and_weaknesses: Strengths
- The proposed modification of the TRADES loss, while small, is theoretically justified. The paper clearly presents the new scheme and its differences to existing algorithms.

- The experimental results support ARoW in comparison to existing methods. Most of the relevant baselines are included, and several ablation studies are added to analyze the effect of different components of the training algorithm e.g. label smoothing.

- The paper is well written, and it clearly presents the new method, the baselines and the experimental setup. The experiments include different architectures and datasets, and the case of using extra data for training.

Weaknesses
- The main concern is about the results reported for the baselines, especially for the case of additional data. For example, in Table 3, for the case of ResNet-18, HAT attains 56.40% and 55.44% of robust accuracy with the 500k extra images from 80M-TI and DDPM synthetic images respectively, while it is reported to get, for the same setups, 57.67% and 57.09% on [RobustBench](https://robustbench.github.io/index.html). Similarly, for WRN-28-10 with extra data, the model from [A] achieves 62.76%	of robust accuracy, higher than any method in Table 3. Then, it is not clear whether the baselines are optimally tuned.

- In general, the improvements over TRADES and HAT in terms of robustness are quite small, although consistent.

[A] https://arxiv.org/abs/2010.03593
clarity,_quality,_novelty_and_reproducibility: Clarity: the paper is well written and clearly presents the method and the results.

Quality: the proposed method is well justified, and the set of experiments is reasonable.

Novelty: the modification to the TRADES loss is quite small, but relevant.

Reproducibility: sufficient experimental details and code are provided.
summary_of_the_review: The proposed method is reasonable and shows promising results. However, clarifications about the discrepancy of the results for some baselines to the original ones are needed, as well as adding the missing baseline.

---
Update after rebuttal

Given the additional results and clarifications provided during the rebuttal, I increase the initial score to 6.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work tries to develop an upper bound of the robust risk and design a new algorithm for
adversarial training called ARoW which minimizes a surrogate version of the developed upper bound.
strength_and_weaknesses: Strength:
1. Well-written
2. Claim both theoretical and empirical contributions

Weakness:
1. **(1)** I guess Equation A.1 should be '$=$' and Throem 1 (and the proof) should be '$=$' (rather than '$\le$') for **binary classification problems**. Hence, Throem 1 is just another expression of 'R_rob = R_nat + R_bdy'. The second term in Theorm 1 is just another expression of boundary error rather than the upper bound of boundary error.     
**(2)** For **multi-classification problems**, I guess Equation A.1 is still $=$ rather than $\le$ as $x'$ includes $z(x)$. **Could authors clarify why use $\le$ rather than $=$ here and what's the meaning of Lemma 2?**      
**(3)** I also think the last line of last equation in Page 13 is $=$ (rather than $\le$) for binary classification problems, and $\le$ for **multi-classification problem** is meaningless, for me it seems like **loosing the equation and artificially creating a very loose bound** . I.e., use **($Y\ne F(z(x))$ is the sufficient and unnecessary condition of $p(Y|z(x))<0.5$ for multi-classification problem)** to bound the equation. This is the only technical part in this theorem, could we call this a new theorem and claim this a theoretical contribution (theorem 1 is even a simple expansion equation for binary case)?  
**(4) In conclusion, for me, Theorem 1 seems like an equation for binary classification problem and an artificially-created loose bound for multi-classification problem. That is,  
$R_{bdy}=\mathbb{E}[\mathbb{1}(F(X)\ne F(z(X)))\mathbb{1}(Y\ne F(z(X)))] \le \mathbb{E}[\mathbb{1}(F(X)\ne F(z(X))) \mathbb{1} (p(Y|z(x))<0.5)]$, which uses a loose and simple bound to bound the original equation. In my opinion, the $\le$ here is the only technical part in Theorem 1.   
I cannot get the hidden meaning and theoretical contribution of this bound and think it is essentially meaningless. Could authors clarify what is the specific and profound meaning of Theorem 1?**    
**Please point out my mistakes if I understand it incorrectly.**
2. In Sec. 3.2, the term 1 {pθ(Y |z(X)) < 1/2} is replaced by its convex upper bound 2(1 − pθ(Y | \hat X pgd)). This upper bound is loose, especially in a multi-classification problem (like CIFAR-10 in the experiments), this bound is meaningless as it is too loose.  
**Thus, for me, the theoretical part in this manuscript seems like a created loose bound plus a created loose bound.**
3. Weak experiments and only a little improvement, need more empirical resuts (e.g., CIFAR-100, compare with AWP-TRADES)  

clarity,_quality,_novelty_and_reproducibility: The manuscript is clear.
summary_of_the_review: I believe this paper is not good enough to publish in ICLR for the following reasons.  
1. (main) Overclaim in theoretical part. Ref weakness 1, 2. I do not think it has sufficient theoretical contribution and its theorem can support the method.
2. (secondary) Weak experiments.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a weighting mechanism to improve adversarial training. Although this is an interesting direction and there are several strengths pointed out by the reviewers (e.g., clear motivation, good related work section), during the discussion we identified several crucial weaknesses and thus decided to reject the paper: 

- The novelty is limited: 

The idea of "putting more regularization on less robust samples" has already been proposed in MART. As pointed out in the paper, the proposed formulation is almost identical to MART, except for the loss function and a small change to the weight term. However, it is not clear why these changes lead to improved performance. The authors only mentioned that their formulation is "theoretically motivated" which is not enough to explain the gain on top of MART. We suggest the authors carefully discuss these subtle differences and give intuitive explanations to clarify why the changes are important. 

Further, Theorem 1 is also not very surprising since many previous papers like TRADES have already done similar decomposition. 

- Empirical results are not strong enough: 

As pointed out by some of the reviewers, the proposed method only marginally improves over existing ones, and the improvements become less clear when using more unsupervised data. This suggests that the improvements may not exist when combined with state-of-the-art methods (e.g., unsupervised data, augmentation, ...). 
justification_for_why_not_higher_score: Limited novelty and the empirical results are not strong enough.
Decision: Reject