Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:
The paper focused on the sample importance in the adversarial training. The authors firstly revealed that over-parameterized deep models on natural data may have insufficient model capacity for adversarial data, because the training loss is hard to zero for adversarial training. Then, the authors argued that limited capacity should be used for these important samples, that is, we should not treat samples equally important. They used the distance to the decision boundary to distinguish important samples and proposed geometry-aware instance-reweighted adversarial training. Experiments show the superiority over baselines. 

Pros:
- The finding on insufficient model capacity is very interesting. The following motivation for GAIRAT is intuitive and well explained. 
- The authors proposed a realized measurement to compute the distance to the decision boundary. This is inspiring for a series of decision-based work. 
- The experiments demonstrate the effectiveness of the proposed method. 

Cons:
- Treating data differently has been investigated in related work like MART and MMA. The authors should discuss the difference from these methods. 
- The capacity analysis provides a very good perspective to analyze adversarial training, however, the explanations in Figure 2 are a little bit weak. 
- The weight function of Eq. (6) lack some intuitive explanations. Why such a formula? Why choose these constants?
- PGD steps are also investigated in CAT and DAT papers. The authors should also discuss the difference to them. 
- The experiments should compare with some baselines considering the example difference, such as MART, MMA. 
- The evaluations should test some modern white-box attacks, like auto-attack, only PGD is not convincing. Besides, Black-box attacks should be tested for a complete evaluation and checking the obfuscated gradients. 

review_title: Interesting paper

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper focuses on adversarial learning. It improves the robustness while keeping the accuracy. To achieve this point, the authors find that adversarial data should have unequal importance, which naturally brings geometry-award instance-reweighted adversarial training (GAIRAT).

Pros:
1. The paper has strong novelty in philosophy level. The common belief is that robustness and accuracy hurt each other. However, this paper shows that the robustness can be improved while keeping accuracy. As far as I know, this point has never been explored before.

2. The paper is well motivated and easy to follow. First, the authors use Figure 1 to illustrate the GAIRAT, which explicitly gives larger weights on the losses of adversarial data. The authors use two toy examples in Figure 3 to explain GAIRAT more. Second, the whole logic of this paper is easy to follow. For example, after explaining motivations of GAIRAT, we can clearly see the objective function of GAIRAT and its realization.

3. The paper is sufficiently justified in experiments. For example, PGD-200 has been used to verify the robustness of GAIRAT. From my personal opinion, this result is quite strong. Moreover, the authors upgrade their method by incorporing FAT and verify the robustness of GAIR-FAT.

Cons:
1.In the top right panel of Figure 10, the SVHN experiments have a period of increasing robustness training error for GAIRAT. Could you explain this? 

2.Although authors show that model capacity is not enough in adversarial training, how large the DNN should be enough? What do you think?
review_title: Reviews for GAIRAT

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper challenges the common belief of the inherent tradeoff between robustness and accuracy.
Instead of recent methods improving accuracy while maintaining robustness, this paper proposes a geometry aware instance reweighed adversarial training (GAIRAT) method to improve robustness while maintaining accuracy. 

Pros:
1 The direction---improving robustness while maintaining accuracy---is novel and interesting. 

Specifically, several papers are challenging the inherent tradeoff, e.g., using more data [1], utilizing early stopped PGD [2], and incorporating dropout [3]. This paper still challenges the inherent tradeoff. 
However, different from [2,3] improving accuracy while maintaining robustness, this paper goes the other direction. 
To my knowledge, this is the first paper to explore this direction. 

[1] Understanding and Mitigating the Tradeoff Between Robustness and Accuracy, ICML 2020
[2] Attacks Which Do Not Kill Training Make Adversarial Learning Stronger, ICML 2020
[3] A closer Look at Accuracy vs. Robustness, NeurIPS 2020

2 This paper has made two conceptual improvements. a) This paper explicitly argues that the overparameterized networks that have enough model capacity in standard training suffer from the insufficiency in adversarial training (though many studies have already shown AT needs the large model). b) This paper argues that under limited model capacity, adversarial data should have unequal importance. Unequal data's treatment was explored in the traditional ML methods several years ago, but it is rare in deep learning at this moment. 

3 The proposed GAIRAT method is effective, indeed increasing robustness while retaining accuracy. The experiments are comprehensive over different network structures, datasets and attack methods. The experiments in the appendix provide much useful information. 


Cons:
1.The design of weight assignment function in Section 3.3 seems heuristic. Would you explain some principles on assigning instance dependent weights? 

2.In Figure 4, the GAIRAT method can relieve undesirable robust overfitting. Would you explain more about this? For example, why the robust overfitting exists in standard adversarial training? how/why your GAIRAT methods relieve it?
review_title: Paper332 AnonReviewer3

Meta-review: The paper proposes an insightful study on the robustness and accuracy of the model. It was hard to simultaneously keep the robustness and accuracy. A few works tried to improve accuracy while maintaining the robustness by investigating more data, early stopping or dropout. From a different perspective, this paper aims to improve robustness while maintaining accuracy. 

There are some interesting findings in this paper, which could deepen our understanding of adversarial training. For example, the authors conducted experiments with different sizes of the network in standard training and adversarial training. The capacity of an overparameterized network can be sufficient for standard training, but it may be far from enough to fit adversarial data, because of the smoothing effect. Hence given the limited model capacity, adversarial data all have unequal importance. Though this technique is simple and widely studied in traditional ML, it is an interesting attempt in adversarial ML and the authors provide extensive experimental results to justify its effectiveness. 

In the authors' responses, the concerns raised by the reviewers have been well addressed. The new version becomes more complete by including more results on different PGD steps and the insights on designing weight assignment function. Also, the authors gave an interesting discussion on enough model size for the adversarial training, though it is still kind of an open question. I would thus like to recommend the acceptance of this paper.  
Decision: Accept (Oral)