Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The authors investigate the robust fairness phenomenon in adversarial training and study the tradeoff between average robustness and robust fairness.  They find that with increasing perturbation radii the average robust accuracy improves over classes while the class-wise robust accuracy suffers from increased variability.  The authors propose a Fairly Adversarial Training (FAT) approach to mitigate the tradeoff between robustness and fairness in adversarial training.
strengths_and_weaknesses: The general problem is clear and seems an important problem to address.

The authors should provide more motivation for why the problem is important to explore.  An example would be helpful here.

Section 3 is difficult to read and the definitions in section 4 should come before this section.  The experiments used to generate the results in this section also need much more information in regards to the data, application, models and training/evaluation strategy for this to be self-contained and fully convincing.

More detail on explaining FAT and how it is used in section 5 is needed.  This section is not clear.

The experiment section should be expanded in the paper or appendix to examine in more detail the significance of the results.  This is not currently clear.

It is not clear how large the impact this work will have in its current state.
questions: What is the significance of the results and how can this work impact other areas?
limitations: see above
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies an important and interesting topic in adversarial training, i.e., robust fairness issue. The robust fairness issue indicates that the adversarially trained model tends to perform well in some classes and poorly in other classes, creating a disparity in the robust accuracies for different classes. This paper focuses on the influence of perturbation radii on the robust fairness problem and finds that there is a tradeoff between average robust accuracy and robust fairness. To mitigate this tradeoff, the authors present a method to correct the lack of fairness in adversarial training.
strengths_and_weaknesses: Strengths:
1. This paper empirically finds a new phenomenon that AT with a larger perturbation radius will lead to a larger variance of class-wise robust accuracy while average robust accuracy is improved. Concretely, the authors use Madry and TRADES with different neural network architectures and attack algorithms to conduct extensive experiments to prove the generality of this new phenomenon. Experimental results consistently support their finding.
2. The authors provide a theoretical explanation for their experimental results through a linear model. Then, this paper also provides evidence for the existence of a tradeoff between robust accuracy and robust fairness.
3. The authors present a method to correct the lack of fairness of the AT. Simultaneously, the authors empirically and theoretically show the effectiveness of the proposed method to obtain better robust fairness without significantly altering the robust accuracy of the model.

Weaknesses:
1. The paper is well-written overall, but it would benefit from a background section (before Section 3) that introduce some of the terminology and definitions for readers who are not familiar with the adversarial training literature.
2. There is too little space devoted to explaining experimental results, it is expected to proivide a more detailed discussion and analysis.

questions: Does the phenomenon in Table 2 still exist for other datasets?
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the relation between the variance of class-wise robust accuracy and perturbation radius in adversarial training. It finds that a tradeoff exists between average robustness and robust fairness. Theoretically, this paper analyzes this phenomenon of the tradeoff and provides an understanding. Finally, this paper proposes a variance based regularized method to mitigate the tradeoff. The idea of this paper is direct and reasonable, and the experimental results demonstrate the effectiveness of the proposed method. 
strengths_and_weaknesses: Strengths:

1.	The paper is clearly written and is easy to follow.

2.	This paper reveals fairness of robust accuracy among different classes from a theoretical point of view.

3.	Theoretical proof and analysis are sufficient.

4.	Extensive experiments are shown to demonstrate the effectiveness of the proposed framework.

Weakness:

1.	Figures 1 (a), (b) and figure 2 (a), (b) are incomplete, and some contents are not displayed well. 

3.	There are some grammatical errors, like “a understanding …” in line 34. 

4.	If I understand correctly, the “sign(t)” should take “-1” for “otherwise” rather than “0” in line 145.

5.	For the equation between line 245 and 246 in Theorem 5.1, there is an extra plus sign.

6.	I am not sure that if Φ(·) in Theorem 4.2 is the probability distribution function. In the line 419 in appendix, the authors describe it as a cumulative distribution function.

7.	There are some typos in appendix: “P(f(x) = +1|y = +1)” in equation 19 should be “P(f(x) = +1|y = -1)”, “Rnat (f,1)” in line 411 should be “Rnat (f,-1 )”.

questions: The theorems in this paper built on a simple linear model. How to extend them to DNN model with multi classification?

For the Theorem 4.5, does the situation σ_+ 〖>σ〗_- equal to the situation σ_+ 〖<σ〗_-? How the variance VCRA changes if σ_+ 〖>σ〗_-?

Can the methods in [1][2] also mitigate the variance of class-wise robust accuracy?

[1] Han Xu, Xiaorui Liu, Yaxin Li, Anil K. Jain, and Jiliang Tang. To be robust or to be fair: Towards fairness in adversarial training. In ICML, 2021.
[2] Qi Tian, Kun Kuang, Kelu Jiang, Fei Wu, and Yisen Wang. Analysis and applications of class-wise robustness in adversarial training. In KDD, 2021.

limitations: N.A
review_title: null

Meta-review: metareview: While the reviews are a bit divergent, it seems many of the concerns raised are addressed properly and some of them are confirmed with the reviewer. I believe the paper is well-written and the contribution is clear with supporting experiments and theory. Hence, I recommend the acceptance of this paper.
confidence: Less certain
Decision: Accept