Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel adversarial training approach, which can increase the logit margin without losing too much clean accuracy. The authors discover that the logit margin of traditional adversarial training has two peaks. Based on this, they designed a switching mechanism and a novel margin loss function (OVR) to improve traditional adversarial training, aiming to push the right peak to the left while maintaining the left peak. Experiments show superiority over baselines.
strength_and_weaknesses: Pros:
- The paper is well written and easy to follow: start from some observations, assumptions, theorems, and the design of their approach.
- The finding about logit margin distribution is interesting. The corresponding analysis is also solid.
- The experiments demonstrate the effectiveness of the proposed method.

Cons:
- For Eq. 8, why did you choose log(1+e^-z)? The author should explain the rationale behind the choice.
- In Table 2, if we add SOVR to Trades (TSOVR), the clean accuracy decreases. It is a contradiction to the author’s maintaining clean accuracy claim. AWP results show the same phenomenon, i.e., clean accuracy decreases while applying the proposed method. 
- The authors say: “SOVR achieved better trade-off: SOVR achieved similar robust accuracies to TRADES, while it achieved better clean accuracy” However, the clean acc are 84.2 for TRADES and 83.1 for TSOVR. In addition, I disagree with the “better trade-off” for this table.
- What are the results if the proposed method encounters the logit scaling attack (Hitaj)? In addition, it is better if the author can show some potential attack ideas that can break the proposed defense. 
- Comparing Fig. 2(b) and Fig. 4, the peak shifting is a little weak. Is there any direction to improve this?
- Observing Fig. 6, it seems that the proposed method did not work well if the number of classes increased. (Cifar-100 is much worse than Cifar-10.) It is better if the authors discuss this. Also, this is a potential limitation of the proposed method.

clarity,_quality,_novelty_and_reproducibility: The paper is well-written, and the overall quality is good. The observation is novel. All the experiment details are in the appendix.


summary_of_the_review: The observation in the paper is interesting, but the experiment results are not convincing to support the claims.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper first studies the relationship of logit margin loss $\ell_{LM}$ and adversarial robustness and then proposes using one-versus-the-rest (OVR) loss to improve the logit margin loss.
strength_and_weaknesses: Strength
- The paper is well-written.
- The behavior of the logit margin loss and how OVR loss helps to improve logit margin loss is well-studied.

Weaknesses
- The idea is incremental because margin-based approaches and logit margin loss are natural and have been well studied before. Additionally, it is also well-known that OVR approach helps to boost the margin.
- The proposed method only slightly outperforms the baselines.
- The assumption in Eq. (4) is extremely infeasible and simple compared to real-world uses of the CE and logit margin losses.

clarity,_quality,_novelty_and_reproducibility: Although the paper is well-written and the concepts are deliberately empirically studied, the novelty of this idea is incremental.   The experimental results show that the proposed approach slightly improves the baselines.
summary_of_the_review: The paper is well-written. The concepts are deliberately empirically studied. However, in my opinion, the idea is incremental.

-----------------------------------------------------------------------------------------------
Thanks the authors for addressing my concerns. I still believe that the idea is incremental and the theory needs to make strong assumptions. Therefore, I am keen on my current score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work reveals that the adversarial examples used in adversarial training can be roughly divided into difficult examples and easy examples according to their logit margins. The proposed SOVR method aims to reduce the adversarial examples which have larger logit margins, then enhance the robustness of DNN models. 
strength_and_weaknesses: I'm confused about "On the other hand, logit margins of importance-aware methods concentrate near zero, and thus, importance-aware methods reduce the logit margins of easy samples. This implies that the weighted cross-entropy used in importance-aware methods is not very effective in increasing logit margins. To increase the logit margins of difficult samples, ...". First, why the observation of "importance-aware methods reduce the logit margins of easy samples" can derive "importance-aware methods is not effective in increasing logit margins"? Second, why do you want to increase the logit margins of difficult samples? Do the large logit margins indicate the examples are well learned? And the proposed method is to minimize the logit margins of "easy samples" if I didn't misunderstand.

And some questions about experiments are as follows:
1. What about TRADES perform in Figure 2? Can the histogram of TRADES also be divided into two peaks?
2. In Table 1, what about the results of TRADES?
3. In Table 2, I'm afraid that the results can't convince me that the proposed method can increase the robustness. Compare with AT+AWP and TRADES, the robust accuracy increase while the clean accuracy drops. It is well known that adversarial training methods have a trade-off between clean and robust accuracy. The performance increase of compare with SEAT is marginal.
clarity,_quality,_novelty_and_reproducibility: The clarity needs to be improved.
summary_of_the_review: Confusing description and lack of convincing empirical results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the source of the worse performance (i.e. lower adversarial robustness) of importance-aware methods compared to standard adversarial training (and variations like TRADES). It is argued that importance-aware methods yield small logits margins for all points, while naive adversarial training shows large margins for a considerable portion of training points (easy points). This has the effect of reducing robustness on the easy examples. This motivates SOVR, which uses the cross-entropy loss on the easy points and the novel OVR loss on the difficult ones. In the experimental evaluation, SOVR outperforms importance-aware methods, and can be combined with regularization methods like TRADES and AWP.
strength_and_weaknesses: Strengths
- The analysis of the logits margins is interesting and suggests an explanation for the worse performance of importance-aware methods compared to standard adversarial training.

- The proposed SOVR is a reasonable solution in light of the analysis of the behavior of the logits margins, and in the experimental evaluation is shown to be effective.

Weaknesses
- The histograms of the logits margin shown in the paper are mostly for model at the last epoch of training. In Fig. 8 in the appendix one can see that, when using the model selected by early stopping, the left peak for adversarial training is much less pronounced, suggesting that it is mainly due to overfitting. While there's still a difference in the distribution of the margins between naive adversarial training and the importance-aware methods, I think the plots for the models with early stopping should be reported in the main part to give a more complete picture of the phenomenon the paper analyzes.

- While SOVR outperforms importance-aware methods, a comparison to other methods (see e.g. https://robustbench.github.io/) for obtaining robust models is missing. The only direct comparison to TRADES is with WRN on CIFAR-10, and the two methods achieve similar results. SOVR getting similar or better results than existing methods would significantly strengthen the paper.
clarity,_quality,_novelty_and_reproducibility: Clarity: the paper is well written, and the results clearly presented.

Quality, Novelty: the analysis is interesting and the proposed SOVR shows good performance. Both are, to my knowledge, novel.

Reproducibility: sufficient details are provided.
summary_of_the_review: The paper presents an interesting analysis of importance-aware methods and a new training scheme to improve their results. A more thorough comparison to existing methods for robust classifiers would increase its relevance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper performs an empirical investigation on why existing importance-aware adversarial training (AT) methods fail and proposes a fix for this. The main empirical novelty of the paper would be that the method is the first importance-aware AT that improves on the AutoAttack benchmark compared to the standard AT. After the active discussion between the author and reviewers, two reviewers ended on the positive side, and the other two were negative about the acceptance of this paper. Despite the positive reviews, AC agrees with multiple reviewers that the technical significance of the paper is somewhat weak due to (a) the lack of convincing justification and (b) its incremental improvements over the base AT algorithms. Furthermore, given the overall marginal gains, AC thinks (c) the lack of discussion (or clarification) on the adaptive attack can be another weakness. For example, the authors should design a new type of attack for the SOVR objective (such as a logit margin-based attack) to fully convince the robustness of SOVR [1]. 

[1] Tramer et al., “On Adaptive Attacks to Adversarial Example Defenses”, NeurIPS 2020
justification_for_why_not_higher_score: Although the paper performs valuable explorations on the topic, AC thinks the reasons to reject slightly outweigh the reasons to accept at the current form.
Decision: Reject