To be Robust or to be Fair: Towards Fairness in Adversarial Training
Authors: Han Xu
Xiaorui Liu
Yaxin Li
Jiliang Tang
Venue:  (2021)
Keywords: Adversarial Examples
Robustness
Safety
Fairness

Abstract: Adversarial training algorithms have been proven to be reliable to improve machine learning models' robustness against adversarial examples. However, we find that adversarial training algorithms tend to introduce severe disparity of accuracy and robustness between different groups of data. For instance, PGD adversarially trained ResNet18 model on CIFAR-10 has 93% clean accuracy and 67% PGD $l_{\infty}$-8 adversarial accuracy on the class ''automobile'' but only 59% and 17% on class ''cat''. This phenomenon happens in balanced datasets and does not exist in naturally trained models when only using clean samples. In this work, we theoretically show that this phenomenon can generally happen under adversarial training algorithms which minimize DNN models' robust errors. Motivated by these findings, we propose a Fair-Robust-Learning (FRL) framework to mitigate this unfairness problem when doing adversarial defenses and experimental results validate the effectiveness of FRL.

Under review as a conference paper at ICLR 2021
TO BE ROBUST OR TO BE FAIR: TOWARDS FAIRNESS IN ADVERSARIAL TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Adversarial training algorithms have been proved to be reliable to improve ma- chine learning models' robustness against adversarial examples. However, we find that adversarial training algorithms tend to introduce severe disparity of accuracy and robustness between different groups of data. For instance, PGD adversarially trained ResNet18 model CIFAR- 10 has 93% clean accuracy and 67% PGD &o- 8 adversarial accuracy on the class "automobile" but only 59% and 17% on class "cat". This phenomenon happens in balanced datasets and does not exist in natu- rally trained models when only using clean samples. In this work, we theoretically show that this phenomenon can generally happen under adversarial training algo- rithms which minimize DNN models' robust errors. Motivated by these findings, we propose a Fair- Shir-Robust-earning (FRL) framework to mitigate this unfairness problem when doing adversarial defenses and experimental results validate the effectiveness of FRL.
INTRODUCTION
The existence of adversarial examples (Goodfellow et al.) 2014 Szegedy al. 2013) causes huge concerns when applying deep neural networks on safety-critical tasks, such as autonomous driv- ing vehicles and face identification (Morgulis al., 2019 Sharif et al., 2016). These adversarial examples are artificially crafted samples which do not change the semantic meaning of the natu- ral samples, but can misguide the model to give wrong predictions. As countermeasures against the attack from adversarial examples, adversarial training algorithms aim to train classifier that can classify the input samples correctly even when they are adversarially perturbed. Namely, they opti- mize the model to have minimum adversarial risk of that a sample can be perturbed to be wrongly classified:
min
max L(f(x 88, f = [1bbg
These adversarial training methods (Kurakin al. 2016; Madry al.) 2017 Zhang al.. 2019h) have been shown to be one type of the most effective and reliable ways to improve the model robust- ness against adversarial attacks. Although promising to improve model's robustness, recent studies show side-effects of adversarial training: it usually degrades model's clean accuracy (Tsipras al., 2018).
In our work, we find a new intriguing property about adversarial training algorithms: they usually result in large disparity of accuracy and robustness between different classes. As preliminary study in Section [ we apply natural training and PGD adversarial training (Madry et al. (017)) on the CIFAR10 dataset (Krizhevsky et al. 2009) using a ResNet18 8 (He et al. 2016) architecture. For naturally trained model, the model performance in each class is similar. However, in the adver- sarially trained model, there a severe performance discrepancy (both accuracy and robustness) of the model for data in different classes. For example, the model has high clean accuracy and robust accuracy (93% and 67% successful rate, separately) on the samples from the class "car", but much poorer performance on those "cat" images (59% and 17% successful rate). More preliminary re- sults in Section 2 further show the similar "unfair" phenomenon from other datasets and models. Meanwhile, we find that this fairness issue does not appear natural models which are trained on clean data. This fact demonstrates that adversarial training algorithms can indeed unequally help to improve model robustness for different data groups and unequally degrade their clean accuracy.
Thhe model's accuracy on the input samples that have heen adversarially perturbed.

---

Under review as a conference paper at ICLR 2021
In this work we first define this problem as the unfaimnss problem of adversarial training algo- rithms. If this phenomenon happens in real-world applications, can raise huge concems about safety or even social ethics. Imagine that an adversarially trained traffic sign recognizer has overall high robustness, but it is very inaccurate and vulnerable to perturbations for some specific signs such as stop signs. The safety of this autonomous driving car is still not guaranteed. In such case, the safety of this recognizer depends on the worst class performance. Therefore, in addition to achiev- ing overall performance, it is also essential to achieve fair accuracy and robustness among different classes, which can guarantee the worst performance. Meanwhile, this problem may also lead to the issues from social ethics perspectives, which are similar to traditional ML fairness problems (Buo-) lamwini & Gebru 2018 For example, robustly trained face identification system might provide different qualitative levels of service safety for different ethnic communities.
In this paper, we first explore the potential reason which may cause this unfair accuracy / unfair robustness problem. In particular, we aim answer the question "Will adversarial training algo- rithmss naturally cause unfairness problems, such as the disparity of cleun accuracy and adversarial robustness between different classes?" To answer this question, we first propose a conceptual ex- ample under a mixture of two spherical Gaussian distributions which resembles to the previous work Tsipras et al.) 2018) but with different variances. In this setting, we hypothesize that ad- versarial trainig tends to only use robust features for model prediction, whose dimension is much lower than the non-robust feature space. In the lower dimensional space, an optimal linear model is more sensitive to the inherent data distributional difference and be biased when making predictions.
Motivated by these empirical and theoretical findings, we then propose Fair Robust Learning (FRL) framework mitigate this unfairness issue, which is inspired from the traditional debiasing strategy to solve a series of cost-sensitive classification problems but we make specific effort to achieve the fairness goal in adversarial setting. Our main contributions can be summarized as following: (a) We discover the phenomenon of "unfairness" problem of adversarial training algorithms and implement empirical studies to present this problem can be general; (b) We build a conceptual example to theoretically investigate the main reasons that cause this unfairness problem; and (c) We propose Fair Robust Learning (FRL. framework mitigate the unfairness issue in adversarial setting.
PRELIMINARY STUDIES
CIFARIO In this section, we present our preliminary studies to show that adversarial train- ing algorithms usually present the unfairness issues, which are related the strong disparity of clean accuracy and robustness among different classes. We implement algorithms including PGD adversarial training (Madry et al. 2017) and TRADES (Zhang et al.] 2019b) on the CI- FARI0 dataset (Krizhevsky et al., 2009). In CIFARIO, we both naturally and adversarially train ResNet18 (He & Garcial (209)) models. In Figure I we present list the the model's s accuracy and robustness performance (under PGD attack by intensity 4/255 and 8/255) for each individual class.
Natural Train
PGD Adv. Train
TRADES Adv. Train
1.0
1.0
1.0
0.0
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
Clean Acc
Ceaan Acc
Clean Acc PGD-4
0.2
PGD-4 Acc
0.2
PGD-4
0.2
PGD-8 Acc
PGD-8 Acc
PGD-8 Acc
0.0
0.0
maok
0.0
Figure 1: Clean and adversarial accuracy in each class of CIFAR10 dataset, from naturally trained ResNet model (left), aGD-aaversarially trained model (middle) and TRADES (right), against ad- versarial examples under 1.0 -orm by 8/255. The trained models' robustness are evaluated by un- targeted PGD attack under 4o0 -nomm constrained by 8/255 and 4/255.
From the Figure T we can observe that - for the naturally trained models, every class has similar clean accuracy (around 90 = 5%) and adversarial accuracy (close to 0%) under the PGD attack. It suggests that naturally trained models do not have strong disparity of both clean and robustness performance among classes. However, for adversarially trained models (under PGD Adv. Training or TRADES), the disparity phenomenon becomes severe. For example, a PGD-adversarially trained model has 59.1% 1% clean accuracy and 17.4% adversarial accuracy for the samples in the class "cat",
2

---

Under review as a conference paper at ICLR 2021
which are much lower than the model's overall performance. This phenomenon demonstrates that adversarial training algorithms cannot provide the same help for the robustness for the samples in class "cat" as other classes, and unfairly degrades too much clean accuracy for "cat". We list our empirical studies under more model architectures in Table 3 and more datasets (GTRSB (Stalkkamp) ct al. 201T)) in Appendix A.2 where we can find the simila observations.
GTSRB We also investigate the fairness is- sue in German Traffic Sign Recognition Bench- mark (GTRSB) (Stallkamp et al. (01T). T. It con- sists of 43 classes of images from difterent traf- fic signs, with image sizes 32 x 32 x 3. In this dataset we also both naturally and adver- sarially train 3-Layer CNN classifier. We list the model's performance and sort the classes in the order of decreasing clean accuracy and adv. accuracy. From the Figure we can see that for the naturally trained model (left), most classes have high accuracy which is over 90%, but for adversarial training, some classes' accu- racy drops by a large margin. Meanwhile, ad- versarial training also unequally improves the model's robustness against PGD attacks given that some classes have very low adversarial accuracy. In this dataset, both natural model and robust model have clear distinguished adversairal accuracy (robustness) among classes.
1.0
Natural Model (GTSRB)
1.0
Robust Model (GTSRB)
Clean Acc. PGD-12
Claon Acc.
0.9
0.8
PCD-12 Acc.
0.6
0.6
0.4
0.2
0.2
0.0
0.0
Class
Class
Figure 2: Class-wise Clean & Adversarial Accu-
racy on GTSRB of Naturally Trained Model (left) and Adversarially Trained Model (right).
3 THEORETICAL ANALYSIS BASED ON A CONCEPTUAL EXAMPLE
From our preliminary studies, we always observe that adversarially trained models have huge per- formance disparity (clean and adversarial accuracy) between different groups. this section, we try to understand the unfairness problem via theoretical analysis based on a binary classification problem
Eexture-Gaussian distribution, which is similar to (Tsipras al. 2018). We first state the necessary notions in this paper.
Notations. In the following, we use f to denote the classification model which is a mapping f X -> y' from input data space X' and output labels V. Generally, naturally training will find the optimal f to minimize the overall clean error Ra [f) = (f) * y); and adversarially training will minimize the overall robust error (f) = Pr. (3. || I fa +d) # y). Specifically in the following binary classification problem, y (-1,, +1} and each class's clean error and robust error are denoted as the conditional probabilities: Re -1) = Pr. (f = +11y = -)), and
R.ob (f, (f,-1) 1) = (3, ||ō|| S c, ff + 8) +1yy 1)), respectively.
3.1 A BINARY CLASSIFICATION TASK
Our study is motivated by (Tsipras et al. 2018) which uncovers one key behavior of adversarial training: it excludes high-dimensional ron-ro features (which are vulnerable to attack) and only preserves power-dimensions robust features for prediction. Thus, in our case, we assume our con- ceptual dataset has the data-label pairs (x, y) sampled from distribution D follows:
if [N (W(-0,0211) if
a.e.r (-1,+1), 0 (7... V
(1)
where NN of 7) is a normal distribution with mean vector and covariance matrix o% I and same for class "-1". Following the work (Tsipras et al. 2018), we suppose that the feature space consists of two kinds of features: (a) robust features with center y and dimension m; and (b) non-robust features with center 7/ and dimension d. We assume 7) < 6 < 7, so an adversarial perturbation with intensity |o0 S can manipulate a non-robust feature to have different sign in expectation, but å cannot attack robust feature. Usually, the non-robust features' dimension d is far higher than the robust features' dimension d, i.e., (m <<
In our case, we assume that the 2 classes have a key difference between their variances: a+1 : 0-1 K : where K 1. In this theoretical example, our main hypothesis is that: the variance difference between classes will not lead to strong disparity of model performance for a naturally trained model whose prediction is based on a high dimensional feature space. However, the variance difference cause large performance gap (both accuracy and robustness) for adversarially trained
3

---

Under review as a conference paper at ICLR 2021
models which are based on low-dimensional robust features. To illustrate this fact, we will explicitly calculate the classes' clean and robust errors in the proposed distribution for both clean models and robust models.
3.2 OPTIMAL LINEAR MODEL TO MINIMIZE CLEAN ERROR
We first calculate one linear model for this data to minimize the total clean errors. Specifically, we consider linear classifier f with its optimal parameters w" and b*:
(x) = sign( (w" ,,)"",))="") b²)
where w", b- arg min Pr.(f(x) +))
where w is features' weight vector and b is the model intersection. In later parts, we use f to represent f for convenience. We call the optimized model naturally trained model because it minimizes model's clean error to get overall high clean accuracy. Typically, a naturally trained model will use both robust features and non-robust features for inference but its prediction outcome majorly depends non-robust features. Next, we show the exact form of the errors for each class in Theorem I and the proof provided in Appendix
Theorem Optimal linear classifier which minimizes overall clean ervor in D will have class- conditional clean errors:
(nn(f.-11)
Rman (f, 1) = Pr.[N(O, 1) - A v K A2 + {(K))
(3)
Rua ($, +1) = Pr.(((,,1) < -K A + A + ((K)
(4)
(nn(f++1))
klog(K) where A = I[K--Ij m2 + dn and g(K) = K--I
It has the robust evror under attack
|| S Eg:
Zmn(S-111)
Rio -1) -1) (r,(N((, S (m
ED Vmn²² + dhp
my + dn
(5)
= + (O, S (f,(f++1) + Rmb +1)
my + dy
60 Ko
(6)
vmn² + dnp
Zma(F:+1))
Note that the term A (consisting of feature dimension d, m and center y, 77) represents how expres- sive the information from D that model f can use for prediction. Thus, when the term A is large, the model has close clean errors between the classes, namely Rnar( -1) = (f. +1). It is because the q(K) term in their z-scores can be ignored when A large. can bring in major threat
the other hand for
their robust errors, typically we assume the adversarial attack by ||0||
mislead natural model /. which will result both (n 11) and Zmb (1) to be large positive
numbers, so that the robust errors of both classes Reob -1) and Roob ff +1) are also large.
3.3 OPTIMAL LINEAR MODEL. TO MINIMIZE ROBUST ERROR
During adversarial training. the desired linear classifier should minimize the total robust error, which minimizes the probability that there exists perturbation å constrained by budge || S that can let the model make mistake. Formally, we describe linear classifier after adversarial training as:
(x) = sign((w55 (w 2) + b
where Whiv b" = arg min Pr. (36 ||5|| S.c, s.t. fai + 6) + v).
w.b
*At here we use Zma(f, -1) and Zani +1) corresponding to Rma(f.- and Rmal (f. +1). Similarly, we use 7 -1) and 7 +1) to denote the 7-score of the standard normal distribution of Rno(ff,- -1) and R,, +1).
(1)+ to denote the z-score for the standard normal distribution
4

---

Under review as a conference paper at ICLR 2021
Similarly use fadv to denote falv for convenience. During adversarial training, the linear model will only preserve the weights on robust features and exclude all non-robust features as proved in Lemma 2 in the Appendix. We will also call the adversarially trained model in D a robust model, because its features cannot be easily manipulated by perturbations under ||8|| Σ We show the robust model's clean and robust errors Theorem and the proof is provided in Appendix BB.2
Theorem Optimal linear classifier which minimizes overall robust error in D will have class- conditional clean errors:
vm o
Rmaa( (fadv: -1) = Pr. 1) B VK K B +K()
c)
(8)
vm Ko c}
Rear Raavi +1) Pr.[N(0.1) K B V B² + q(K)
(9)
where 13 o(k² vm (T- e), 6y,,,, q(K) 2log(K), and robust errors under attack ||ô]| -ー-1
S 60:
Reb (Jadi; -1) = (o, S Zmme -1) + vm
(10)
Ryo (fadi: +1) = (r, 1) S (aad +1) + vm Ko
(11)
Recall that we assume the dimension of non-robust features is much higher than that of robust
features (d >> m) and y, yj and c have similar scale: c O(7) and +()). Therefore, in Eq. (8)
and Eq. 9 for model fadv's 's clean errors, the decisive term B is scale ((d/mm) A). where A
is the term in (3) In Corollary (proved in Appendix [B.3). we show that it is this relationship
between A and B that will finally bring in the "unfair" issue to both model accuracy and robustness performance.
Corollary Adversarially Trained Model on DD will have larger clean evror disparity between the 2 classes, compared a Naturally Trained model.
- 1)) for natural model f; and (Rna (futv: +1) (Rma(fatv, -1) for adversarially trained model fuvv. Since both terms are positive, show their ratio (detailed proof in Appendix):
We calculate 2 classes' clean error difference: (1))(
Ω Rumi +1) Roal (Jadv: -1), (K2-1)loge) 1))og(K)o² Reme (f, +1) (f.
O d _ )
(12)
20(72)
m
In Eq. 12). we can tell once d/m is large, the ratio Ω is also large (e.g. 1) and the adversarial training showed enlarge the classes' clean eтror disparity. The ratio Q in Eq. (12) uncovers the main factor which may cause the "unfair" phenomenon in adversarially trained models: because the robust models make prediction in feature space with dimension which is much lower than d (dimension for natural models), their clean accuracy disparity between classes can be more sensitive to the classes' distributional difference K. In this way the robust model presents strong disparity of clean accuracy.
Furthermore, for the robust errors (i.e., Eqs. (1O) and (42)) in adversariallly trained model, an ad- versarial attack under intensity en will make the test error increase only by a small margin compared to the model's clean error in Eqs. (8) and (9). It is because the their z-scores difference is de- cided by vm which is small and much lower than Because of the marginal difference between Reai (f [uvv, -1) and Rio (fatv, -1) (also for class "+1"), the model's fairness condition on robust er- rors will align to the clean errors (in Eq. (12). Empirical results real datasets also support this assumption (Figure D. As a conclusion, adversarial training can bring in unfairness issues on both clean and robust performance.
4 FAIR ROBUST LEARNING (FRL)
Faced with the unfairness problem of adversarial training shown in Section 2 and we desire to devise Fair Robust Learning (FRL) strategy, in order to train robust models that have balanced accuracy and robustness performance for each class. Formally, we aim to train a classifier f to have minimal overall robust error (Roo while stressing ff to satisfy series of faimess constraints:
minimize Reo((f)
(13)
s.t. Rus (f, i) Rau(f) S ti and Reo i) Reo $ Tu for each e Y
5

---

Under review as a conference paper at ICLR 2021
where 7i and 72 small and positive values. The constraints in Eq. 13 restrict the model's s error for each class (both clean error (f. and robust error Reo (f. 2)) should not exceed the average level (Raa()) and Roo(() by a large margin. Therefore, the model will not have specific weak points under the risk of wrong prediction adversarial attacking. Next, will discuss the detailed components of our Fair Robust Learning (FRL) algorithm to solve the problem in Eq. (3)
4.1 TRADITIONAL MODEL DEBIASING METHOD: A REDUCTIONS APPROACH
In order to solve the fair robust training problem in Eq. (13), we follow the main pipeline from traditional machine learning debiasing works such as (Agarwal et al.) 2018), which reduces the problem in Eq. (3) into series of Cost-sensirive classification problems and continuously penalizes the terms which violate the fairness constraints. We begin by introducing Lagrange multipliers Φ (Ohur (is)) (non-negative) for each constraint in Eq. (13) and form the Lagrangian:
T(S,)) = (J) +
Σ Srat (Rma (S. Rmm(() 7)) +
Loo (Rmbb Roob - -T2)
(14)
It equals to solving the max-min game between f and Φ as:
max min L(f. Ohu f
(15)
Typically, given fixed Φ. if the current model f violates some constraints in Eq. 13 (for ex-
ample Rn Rua Ti 0), we first solve the mater-maximization problem Eq. (15)
by increasing its corresponding multiplier d As result, upweight the training weight (or cost) for the clean loss (f, of all samples the class i. Then, the algorithm will solve the inner-minimization given new Φ to optimize the model f. the error of Rnar (f, is therefore heavily penalized and the model will give more priority the correct prediction for the class í. In this way the model gives more priority to mitigate violated terms in Eq. (14). During this process, the model f and Lagrangian multiplier& ó will be alternatively updated to achieve the equilibrium until finally reach an optimal model that satisfies the faimesss constraints. Based on this traditional debiasing strategy, next we will discuss the main difference of our task in the adversarial setting from this traditional approach.
4.2 DEBIAS CLEAN ERROR AND BOUNDARY ERROR SEPARATELY
One thing to note that in the Eq 13) the robust error is always strongly related clean er- rors (Zhang et al., 2019b;; Tsipras et al. 2018) (see Eq. [6]. Thus, during the debiasing process above, we could have twisted the influence (f. It means that when we upweight the cost for Reo (f, i) as introduced Eq. 15). we also implicitly upweight the cost for (f,().)). Thus, we will not get a precise update for Φ. To solve this issue, we can separate the robust error into the sum of clean error and boundary error inspired by (Zhang et al. 2019b) as:
some class i'8 clean and robust errors R ( and
Ronh i) = Pr.335, s.t. f(x + 6) # vly =1y= i}
= =Prlf(x) # vlu i) + Pr/36, f(z 5) f(z) 0
(16)
= RR 1) + Rody (f
01y=t) i) represents the probability that a sample from class lies close to the decision boundary and can be attacked. By separating the clean error and boundary error during adversarial training, we are able to independently debias the unfaireess from both clean error and boundary error. Formally, we have the training objective as:
where (f,) f(x
minimize Raa(s) + Rody (f) s
(17)
s.t. Ran(f i) Ran(f) S v2 and Rnay(f, - (f) S T2 for each i E Y
We introduce Lagrangian multipliers o = (d )irr) and solve the max-min game for Eq. 17 simi-
lar to Eq. 1 Note that if the constraints in Eq. (17 are satisfied, the fairness quality for robust error
(Roob (f. i) Rum (f. i) + Rody(f. (f, i)) of each class can also be guaranteed by тI + 72. In practice we will use surrogate loss functions (such as cross entropy) [(f(x), y) and max [((()), to [6 [8
optimize the clean and boundary errors as suggested by (Zhang et al. 2019b).

---

Under review as a conference paper at ICLR 2021
4.3 COST-SENSITING CLASSIFICATION FOR CLEAN ERRORS VS BOUNDARY ERRORS
During the debiasing training process in Eq. (17). if one class i's clean error violates the fairness inequality: Rua (f, -m (f) - > O, upweighting the cost for Raa i) can help penalize large Raa (f. 4) and mitigate the unfaimess issue suggested by (Agarwal et al. 2018). Note that refer
to this strategy as "Reweight". However, the boundary error for class x: Rhay (f. Reay
72 > 0, we claim that only upweighting its cost (or Reweight) could not succeed to fulfill the cost-sensitive classification goal in adversarial setting. Our empirical studies in Section 5 show that upweighting the boundary error for some class i cannot effectively reduce model's 's boundary error specifically for class but can bring in side-effects to degrade class i's clean accuracy. It is evident from (Ding et al. 2018) that increasing the margin e during adversarial training can effectively improve model's s robustness against attacks under current intensity c. Therefore, we hypothesize that enlarging the adversarial margin when generating adversarial examples during training specifically for the class can improve this class's robustness and reduce the large boundary error Redy (f) i). In this work, we refer to this strategy as "Re-margin", Empirical study in Table and Figure 3 validates its effectiveness.
We present the main components and process of Fair Robust Learning (FRL) in Algorithm Note that BEST ff Φ, c) denotes the adversarial training process under adjusted hyper-paramedt ó mentioned in Eq. (14) and we test the performance under the validation set which is denoted as EVAL(f.-). In Algorthhm [ in each iteration we first test our initialized or pretrained model f on the validation set to check whether it violates the unfairness constraints (step 5). Then we update Lagrangian multiplier Onat to reweight the clean loss for each class. We propose three strategies to update hyper-parament to balance the boundary loss including Reweight (option 1), Remargin (option 2) and Reweighh++Regmaggh (option 3). We follow one of the options (step 7) for boundary loss. Finally we adversarially train the model under the updated setting by 6,
Algorithm The Fair Robust Learning (FRL. Algorithm
1: Input: Fairness constraints specified by t > 0 and T2 > 0, test time attacking radius and hyper-param update rate Qr, 022
2: Output: Fairly robust neural network f
3: Randomly initialize network or initialize network with pre-trained configuration
Set &i.. = 0, Pbay 0 and ø (deat, Pody), adv. training radius e co for each E y
4: repeat
5: Rear (i, f), Rody (i, EVAL(f,
Evaluate f for each class
6:
of = Han + (Rnm (i, () TT)
D>
Update multiplier drat
7:
Phay = Payy + 022 (Raoy (i, -22)
D Option 1. Update multiplier obay
7:
Pbay Puyy's G 4 exp((22 (Rody (1, r2))
Option 2. Remargin
7:
or Bbyy = obsy + -RooR (i, f) T2)
6, = e exp(aa(RRuy
-2))
D Option 3. Reweight + Remargin > Adv. training under hyper-param di, e from current
8: S BEST(S: e)
9: until Model f satisfies all constraints
EXPERIMENT
In this section. we will present the experimental results to validate the effectiveness of the proposed framework (FRL) for building fairly robust DNN models. We implement and compare our proposed three strategies (i.e., Reweight, Remargin and Reweight-Remargin) on real-world data and discuss their possible different consequences. We also discuss the main difference of the manner of our proposed three potential debiasing strategies.
Experimental Setup& & Baselines We conduct our experiments on CIFARIO (Krizhevsky et al. 2009). For CIFAR 10, we present our main results under the model architecture PreAct Residual Network (He al. 2016). As comparison sets to show our method can improve fairness, also present the original performance from two popular adversarial training algorithms (Madry et al, 2019b). Meanwhile we add baseline debiasing method which are inherited 2018) (we directly apply it to reweight the cost of adversarial examples during
2017 Zhang et al.
from
Agarwal al.
7

---

Under review as a conference paper at ICLR 2021
adversarial training) representative to show traditional debiasing methods might not be ily applied solve unfairness issues in adversarial setting. For CIFARIO dataset, we mainly test our method's performance on ResNet18 (He & Garcia 2009)) model, and we apply the PGD at- tack (Madry al. 2017) algorithm generate adversarial examples under 42-nom under 8/255. For each of the debiasing algorithm, we set the fairness constraints тi and 72 be 5% and 7% respec- tively, for clean and boundary errors. Please refer to this link for empirical implementations.
Debiasing Performance (CIFARIO) We first check whether our proposed FRI. framework can help resolve the unfairness issue In adversarial training, Refer to our goal in Eq. (13) to achieve the fairness constraints get both balanced clean and robustmess performance. We report the trained model's 8 average clean error rate, boundary error rate and robust error rate (defined in Eq. (T)) ), as well as the worst intra-class clean, boundary and robust error rates (Table II. Thus, for an optimal equally robust model, hope each of these worst intra-class the average errors. Mcanwhile, it is also necessary that one debiasing strategy should not have too much sacrifice on the model's 's overall clean and robustness performance.
is not too large compared to
Table I: Average & worst-class clean error, boundary error and robust error for various algorithms.
Avg. Clean
Worst Clean
Avg. Bdy.
Worst Bdy.
Avg. Rob.
Worst Rob.
PGD Adv. Training
17.3
40.9
39.4
$4.4
56.9
82.6
TRADES (33 =
14.4
27.9
43.6
62.6
58.0
83.6
TRADES (8
16.9
34.9
39.1
56.6
55.5
82.2
Baseline Reweight
19.2
28.3
39.2
53.7
58.2
80.1
FRe/eeweight) (Remargg))))) ReL[/eewightt-Remrrgin
17.0
22.5
41.6
51.2
58.6
73.3
16.9
24.9
41.6
50.6
58.5
76.3
18.4
24.7
40.3
47.4
58.7
70.2
In Table for CIFAR10 dataset, present the performance of all three versions of our proposed FRL framework. Compared to the baselines, all the FRL algorithms reduce the worst intra-class clean, boundary and robust error under different degrees. FRL (Reweight) can get the best debiasing performance to achieve minimal "worst-class" clean error, but it cannot debias boundary loss well. The method (Reweight Remargin) can be the most effective way debias boundary error disparity and robust error disparity. added baseline method (Baseline Reweight) (Agarwal et al. 2018) only have minor help for clean performance fairness but cannot debias boundary error robust error. For each debiasing method of FRL, compared to vanilla PGD and TRADES, the total average performance is only degraded by a slight margin (1 ~ 2% for clean error and ~ 3% for robust error), thus the debiasing method will not sacrifice too much total performance.
Debiasing Performance (GTSRB) We also apply our FRL debiasing method to GTSRB (Stal- Ikamp et al. 201T), which consists of images from 43 various traffic signs. Since this dataset is highly unbalanced (some classes only have around 200 training images), only keep 17 classes when performing adversarial training. the robustness using PGD attack under loc norm by 12/255. In Table 2 we list the average and worst-class clean, robust and boundary emrrs for different training algorithms. The results show that the FRL method (Reweight, Reweight-Remargin) can effectively help to improve the model's worst-can worst-class clean and robust performance. Furthermore, since the original dataset is imbalanced, the debiasing method brings in side-effect improve the model's overall performance, with lower average clean error and robust error.
this dataset, we apply 4-layer CNN classifier and test
Table Average worst-class clean, boundary and robust error for various algorithms in GTSRB.
Avg. Clean
Worst Clean
Avg, Bdy.
Worst Bdy.
Avg. Rob.
Worst Rob.
PGD Adv. Training TRADES
1.2
9.1
18.3
42.3
20.6
49.5
1.6
21.5
50.4
23.1
59.7
TRADES( 2
2.1
17.5
40.3
19.6
46.3
Baseline Reweight
19.4
42.4
20.1
44.0
FeweRgee)ght)
0.7
3.5
18.3
39.4
19.0
40.0
(Re(aearrgin)
2.4
10.7
18.0
44.3
18.4
44.3
TRLLReeweight-Reemariin
0.8
4.0
16.6
34.2
17.5
38.2
https:://rive..gogl/ htpps////dhheeoooo...ooo////ooaa h3ve http://.........o..oooooooooooo.
8

---

Under review as a conference paper at ICLR 2021
Reseiggt
ReMargin
Resen 1 ReNeang
6:20
6:23
Compare Different Debiasing Strategies. In Figure B] we take a closer look at the be- havior of different proposed de- biasing strategies mentioned Section 4.3 and test whether they can succeed in solving the constrained training problem in Eq.( We present the model's maximum violation (e.g. u(i)
0.13
0:00
500
6:000
300-
6,75
WAMLA
300-
6.35
0.00
030
Goads
Training Spxds
Traning
Figure 3: Debiasing Manner for 3 FRL Options in CIFARIO.
Rost (f, - (f TI) among all classes for each training epoch that hot-started from a pre- trained (adversarially) ResNet model. If v(i) S Q for each class in } each faimess constraint is satisfied. From the figure we can tell that FRL (Reweight) method cannot adequately balance the boundary error, and always presents a trade-off relation with the clean error constraints. Introduc- ing the Remargin method will facilitate to achieve faimess for boundary errors. More details (e.g. average and worst clean robustness in training) are in Fig. 5 Appendix A.2
RELATED WORKS
Adversarial Attacks and Adversarial Training, The existence of adversarial attacks (Goodrel-
low et al. 2014 Szegedy et al. 2013) Carlini & Wagner| 2017) causes huge concerns when people
adopt machine learning models in various application domains (Xu et al. 2019: Jin et al. 2020).
As countermeasures against adversarial examples, adversarial training (robust optimization) algo-
rithms (Goodfellow et al. 2014 Madry et al. 2017: Zhang al. 2019b Shafahi al. 2019 Zhang
ct al. 2019a) are formulated as a min-max problem that directly minimize the model's 's risk on the ad- versarial samples such that the machine learning model is robust under the adversarial attacks. They are shown to be one of the most reliable strategies to improve model safety. Another mainstream of defense methods are certified defense, which aims to provide provably robust DNNs under lp norm bound (Wong & Kolter 2018 Cohen et al., 2019) and guarantee the robustness. In this work, we focus on studying the potential risk the detense algorithms from a new scope of the fairness concerns.
Fairness in Machine Learning & Imbalanced Dataset. Fairness issues recently draw much atten- tion from the community of machine learning. These issues can generally divided into two catego- rizations: (1) prediction outcome disparity: the models tend to have some unreasonable preference of prediction for some specific demographic groups ZZfarr et al., 2017); and (2) prediction quality disparity: the models tend to have much lower performance on some specific groups than oth- ers (Buolamwini & Gebru 2018). Please refer to the works (Barocas et al.; MMehrbii et al., (219)) for more comprehensive and detailed summary of fairness study in machine learning. The reasons that cause these discrimination problems might come from data distribution or the learning algo- rithms. Unlike existing works, this work is the first to study the unfairness issue in the adversarial setting. We argue that robustly trained models are likely to have different accuracy and robustness quality among different classes, and it may introduced by both data distribution and the adversar- ial training algorithm. We also mention imbalanced data learning problem (He & Garcial [2099] Lin eet al., (017) one related topic of our work. Since in work, (e.g. Figure 1). we show that the prediction performance differences are indeed existing between different classes. This phenomenon is also well-studied in imbalanced data problems or long-tail distribution learning problems (Wang let al. 2017) where some classes have much fewer training samples than others. However, our case, we show that this unfairness problem can also happen in the balanced data, so desires new scopes and methods for further study.
CONCLUSION
In this work we first theoretically and empirically uncover one side-effect of adversarial training algorithms: it can cause serious disparity for both clean accuracy and adversarial robustness between different classes of the data. As first attempt to resolve unfairness issues from adversarial training, we propose the Fair Robust Learning (FRL. framework to mitigate this issue.
REFERENCES
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A reduc- tions approach to fair classification. arXiv preprint arXiv: Seti:180000245. 2018.
Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness in machine learning.

---

Under review as a conference paper at ICLR 2021
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commer- cial gender classification. Conference on fairness, accountability and transparency, pp. 77-91, 2018.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium security and privacy (sp). pp. 39-57. IEEE, 2017.
Jeremy M Cohen, Elan Rosenfeld, and J Zico Kolter. Certified adversarial robustness via randomized smoothing. arXiv preprint arXiv: mete::20210111 2019.
Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Max-margin adversar- ial (mma) training: Direct input space margin maximization through adversarial training. arXiv preprint arXiv:
2018
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6577 2014.
Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions knowledge and data engineering, 21(9):1263-128 1284, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Wei Jin, Yaxin Li, Han Xu, Yiqi Wang, and Jiliang Tang. Adversarial attacks and defenses on graphs: A review and empirical study, arXiv preprint arXiv:2003.00653. 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. arXiv preprint #rXiv:661..01236 2016.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision, pp. 2980-2988, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 2017.
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. arXiv preprint arXiv: 1908. 1908.09638,0 09635, 2019.
Nir Morgulis, Alexander Kreines, Shachar Mendelowitz, and Yuval Weisglass. Fooling real with adversarial traffic signs. arXiv preprint 1rXiv:1907.00374 2019.
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In 2019.
Advances in Neural Information Processing Systems, pp.
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. Accessorize crime: Real and stealthy attacks on state-of-the-art face recognition. Proceedings of the 2016 acm sigsac conference on computer and communications security. pp. 1528-1540, 2016.
Johannes Stalkkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The German Traffic Sign Recognition Benchmark: A multi-class classification competition. In IEEE International Joint Conference on Neural Networks, pp. 1453-1460, 2011.
Christian Szegedy, Wojeiech Zaremba. Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 13/2,6199, 2013.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. arXiv preprint arXiv:
2018
Yu-Xiong Wang. Deva Ramanan, and Martial Hebert. Learning to model the tail. In Advances in Neural Information Processing Systems, pp. 7029-7039, 2017.
10

---

Under review as a conference paper at ICLR 2021
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning, pp. 0085558865999 PMLR, 2018.
Han Xu, Yao Ma, Haochen Liu, Debayan Deb, Hui Liu, Jiliang Tang, and Anil Jain. Adversarial attacks and defenses in images, graphs and text: A review. arXiv preprint 2019.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. Fair- ness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In Proceedings of the 26th international conference on world wide web, pp. 1171- 1180, 2017.
Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong. You only propagate once: Painless adversarial training using maximal principle. arXiv preprint arXiv:1905.00877 (3), 2019a.
Hongyang Zhang. Yaodong Yu, Jiantao Jiao. Eric P Xing, Laurent El Ghaoui, and Michael I Jordan. Theoretically principled trade-off between robustness and accuracy. arXiv preprint 19110:1901.08573, 2019b.

---

Under review as a conference paper at ICLR 2021
A APPENDIX.
A.1 OVERALL PERFORMANCE ON MORE MODELS
Table 3: Adversarial training algorithms on CIFARIO dataset (ResNet18 (above) and
ReNer34(below)) We report the average clean aceuracy and adversarial accuracy (under PGD attack by 8/ 8/255), well as the worst best clean accuracy and adv. accuracy among all classes.
Avg. Clean.
Avg. Adv.
Worst Clean
Best Clean
Worst Adv.
Best Adv.
Natural Training
93.1
0.0
87.5
97.7
0.0
0.0
PGD Adv. Training
82.7
43.1
59.1
93.6
17.4
64.9
TRADES
83.1
44.0
65.1
93.4
17.8
67.0
Natural Training
95.1
0.0
88.1
98.2
0.0
0.0
PGD Ady. Training
86.6
46.3
72.3
96.4
19.8
72.2
TRADES
85.5
56.3
67.0
95.2
27.5
79.5
A.2 THE PIENOMENON OF UNFAIR ROBUSTNESS ON GTSRB
GTSRB We also show the similar unfairness phenomenon in German Traffic Sign Recog- nition Benchmark (GTRSB) (Stallkamp et al. 2011). which consist of 43 classes images from different traffic signs, with image sizes 32 x 32 x 3. In this dataset urally and adversarially train a 3-Layer CNN classifier. We list the model's performance and sort the classes in the order of decreasing clean accuracy and adv. accuracy. From the Figure 4, we also see the natural training has similar clean accuracy between different classes, but adversarial training will enlarge their gap and result the clean performance has a heavier tail property. In this dataset, both natural model and ro- bust model have clear distinguished adversairal accuracy (robustness)). However, adversarial training even hardly provide any robustness improvement for some classes
Model (GTSRB)
Robust Model (GTSAB)
1.0
1.0
Clean Acc.
Clean Acc.
PGD-12 Acc.
PCD-12 Acc.
0.6
0.6
also both nat-
0.4
0.4
0.0
0.0
Class Index
Class Index
Figure 4: Unfairness in GTSRB
Avg. Clean.
Avg. Adv.
Worst Clean
Best Clean
Worst Adv.
Best Adv.
Natural Training PGD Adv. Training
99.5
18.9
91.7
100.0
0.0
72.5
94.5
44.4
50.0
100.0
1.3
90.0
TRADES
91.2
47.2
35.3
100.0
3.3
92.0
Table 4:
Adversarial training algorithms GTSRB dataset (on 3-layer CNN model). We report the average clean accuracy and adversarial accuracy (under PGD attack by 12/255), as well as the worst / best clean accuracy and adv. accuracy among all classes.
B PROOF THEOREMS
B. PROOF OF THEOREM
Before proving Theorem I we first establish the optimal linear classifier in natural training through Lemma I which facilitates us to prove the clean error and robust error after natural training.
Lemma (Optimal linear classifier in natural training) For the data following the distribution
in Eq. 4/). the naturally trained linear classifier
fxx sign( w0 x. + b) has the optimal weight that Wm and Wm++1 Wm++2 = Wm+b Moreover, their ratio satisfy
satisfy: W1 = W2
W1 Wm++1 = y 1) and b wi = Wo where
K2 +1 Wy 1
4
20 log K A2(K2 -
A
+
K2 11
(K2
1)2
12

---

Under review as a conference paper at ICLR 2021
Reweightt
0.153
vinsetion
0:325
Worst Viaretion for Boundary Error
0.100
maman
0.050
mmm
0.025
0.000
Training Epochs
(a) Adv. Acc
(b) Clean Ace
(c) Violation
ReMargin
0.12
3000
2714 Mo
0.07
0.00
100
Trainin
(d) Adv. Acc
(e) Clean Acc
(1) Violation
Reweight ReMargin
Wonre Viniation
Vielntion
0.300
0.075
034
0.000
0,000
Epochs
(g) Adv. Acc
(h) Clean Acc
Violation
Figure 5: Debiasing training process
Proof (Proof of [e We first prove u2 Wm+b by contradiction. We define G1 (1,2...., m] and G2 {m + I.m 2....., m d]. We make the following assumption: for the optimal w and b, we assume there exist wA < wj for i # and j E G1. Then obtain the following clean error for two classes with this classifier
and Wm++1 Wm+2
R(J,-1)=PP = Pr
Σ
WNNe +b (b,+W((-,,, w,
(18)
+(f Pri Σ
WNNe+ +b (oNk-+oow-(((tdd))))+
<
(19)
we use wy replace Wi, obtain new errors
R(f. -1) = Pr
WANe + w.N-:b-Hag-((yod.)-nyy-----.. 0}
(20)
R(f++1) Pr(
Σ knght
Wh WW <
(21)
It implies R(J, R(J,+1) < RJ, R(,-1)
R(J,+1)
Therefore, contradicts with the assumption we make and we conclude for an optimal linear clas- sifier in natural training, it must satisfies wi W2 .... = Wm- The same argument applies to i,j G2 and conclude Wm+11 Wm+2
Wmtd similarly.
13

---

Under review as a conference paper at ICLR 2021
Next, calculate the ratio between W1, #m+11
R(f) = Pr(f(()) # y}
(22)
- -1go--yy--)---y----)-------
-1 13 Prly= +1]
Prfw"x+b>> 0ly -13 + +bb> 0ly --1
0ly = =-1
= -P Σ
w/N( (-)) Σ
iEGi
jEG₂
+ Pr( iEG1
u,N(+7, 02
(0,, + 0ly +1}
jEG²
mwi7 7 - dwm++11 177
b mWii 7y - dwm + 17] .}
= Pr(W(0,
+ Pr(N(O,
mmmp m
v mnu dww K
7(-1)
Z(+1)
(23)
We derive the optimal W1 and Wm++1 by taking
BR(
0 and
BR(
0:
Owm
Dwn+1
@R())
1
1 (-2
(-
(2(-1)))
--(-1)
(- (2(+1))²)
07(+1) 0
exp
exp
Own
2T
Ow
Owi
aR())
1
1
(2(-1))²)
OZ(-1)
(- 1 (2(+1))²)
0Z(+1)
exp
(-2
exp
Owm+1
22T
@wm+1
2t
Owm+1
These imply
-
OZ(+1)
aZ(-1)
(z(11)
Own
Own
Swm+1
@wm+1
and therefore
yd bwi + drpwy Wm++1
-mmww - bwz + 1
(24)
-dd + bwr + dryw Wm++1
mmmw + +b myy 1m
Simplifying it gives wi Wm++1 =7 7-
Then, we calculate the ratio between wi and Based on the relation between Wi and Wm+11 we let W1 y and Wm+1 = T/L for some constant w. Substitute w1 and Wm+11 into Eq. (23) we have
b
m w + dnj w
R(f. = =
(25)
OW WwV + dnp
Fw my² + dnp
b
m2² + dn?
=Pr(N(0.1)
(26)
own ww2 ow/w22 + ddp
-b
mny² + dn? Ka
= = owK /wn? dip
(27)
For simplicity, we denote A = /m dnz and assume w, = 8 R()( -) O. In detail, it is
b. w.
We will find optimal b by letting
thoo
Wp
1)9
1 Wo =
exp(
exp(
(28)
Aoy 2m
2 Ao
Ko A)2) = 2log K and therefore we obiain
K KAov2.
2 KA
which gives [ we Ka
+ )
1 Wo
2 Ag
K2 +1
4
20 log K +
1
A2K
Wp
K2
(K2
1)2
AP(K2 1)*
14

---

Under review as a conference paper at ICLR 2021
Proof 2 (Proof of Theorem clean ervors
I
Substitute the optimal linear model {w, b} in Eq.
23 we have
Rma -1] Pr.(N((O, 1) S (f - 1)}:
(29)
Rmat (f, +1) Pr:(W((0.) 1) & Znar (f. (f++1)}
(30)
where
2A4
4A22
2 2log K (K² - -1)
mn -1)} (K22 - 1)0
K
(31)
(K2 - 1)2
-2KA
4A²
2 2log K (K2 1)
Zm +1])) (K2
*V
+
(K2 - 1)202
+
(32)
Substitute the optimal linear model {w,b) b) into robust error in Eq. errors:
35 and
36
have robust
Roob (J: -1) Pr.(W(0,1) S (nar(s, (aan(s-11 -1)
my + dn
6D )
(33)
Vmn² 2 + dip 0
my + dn co
Roo +1) = Pr:(N((, S Zf +1) +
Vmn² + drp² Ko }.
(34)
B.2 PROOF OF THEOREM 2
Before proving Theorem 2 we first establish the optimal linear classifier in adversarial training through Lemma which Tacilitates us to prove the clean error and robust error after adversarial training.
Lemma (Optimal linear model in adversarial training) For the data distribution assumed in Eq. 77. the adversarillly trained linear classifier f(x) = sign(w" (w x b) has the optimal weight
that satisfy: w1
Wms W Wm++1 = Wm++
wm+b = and b = wo 1 where
K2 + K2
4 m(7 - k) (K2 1)2
20 log K
wo m(7 e)²
+
m(7 - (K2 1)
Proof (Proof I Lemma 2
Rmb(f: 1) = (r. ||ا|| I {cf[(+))>00===)
= Pr. max f(r + 0) >0 Se
Pr. max
wy w; + 0+) +
Σ wW] 0).
$
i=m+
(35)
Similarly, we have
Ryb Rf +1) Pr. (3 ||o|| S,, f(x +)) < <0)0))==+++ +1)
Pr max f(x 8) > ly ++
Se
m+d
=Pr. max
Σ -
w;(N(<s (LV(),o))) + 5z)
Σ m=++1
(N((t, + +bb<), <).
So
(36)
15

---

Under review as a conference paper at ICLR 2021
To prove the theorem by contradiction, first assume that for the optimal w, there exist some wi > 0 where: E G2 {m + 1, 2..... m + d3. Then
Rs -1) 11= Pr.
max w (V((00)) 6,) + b+
max W. (N((-,,,)))) 6,) > 0}. (37)
16 Se
18.
Because > O, B is maximized when 8 = e. We obtain Rmb -1) Pr + +
> 03 2 Pr. 0}
(38)
So wi 0 gives a better robust error. We can also assume < 0 and use similar contradiction arrive the conclusion
prove W,, 0. Similar argument holds for Ryob (f,+f)) +1). Therefore,
w, =0 for all m +isis S m + The calculation of wo similar to the proof of Lemma 7 and
we omit the proof here.
Proof 4 (Proof of Theorem 2 Substitute the optimal linear model in Lemma 2 into Eq. obtain the clean error on the adversarially trained model:
23 we
Rmar (f. - Pr:((((0 2/m(7 -c)
4m(7 - c)
2 K
(K² -1)o (K3 - 1)202 + K² - /
V
(39)
Ruai +1) -)r(N(0,,) < 2/m((7 -
4m - 6)2
2 log K K
+ K
+
(40)
(K2 2 - 1)0
(K2 1)202
Zucl Zaa(Jab.t1)
Substitute the optimal linear model in Lemma on the adversarially trained model:
2 into Eq. 35 and 336 we obtain the robust error
Reb (faii: -) = Pr.(N((, 1) s Znar Zaar(aadr, -1) + vm CD
(41)
Remb (Jndis +1) = (r, < Zaae +1) + vm Ka
(42)
B.3 PROOF OF COROLLARY 1
Proof (Proof of Corollary
We try to show R' (J, -1, R(S++1) R(S,
(2((11)
R'(J++1) R'
g(x)do gg(ed) (Z (Z - (-1)
(43)
22((11)
A
R(S+1) R(, 1
(2((1)
g(z)dr= = g(() (7(+1) - - ((-))
(44)
(Z(-1)
D
where (-(-1) E S. (+(1) and Z(-1) 5 < Z(+1) according Mean Value Theorem. Here g(-) the density function of standard normal distribution.
Next, we try to show: R R(.
(f,-11
AC BD > 1. Remind that
2/m + +mm+dd(7))
4m0((12)
2 2log K K2
Z(-1)
KV (K2 1)202
(45)
o(K2_
20(7) (Vm m + d - K
Kmm m m//mdd- {(K))
(46)
(K2 - 1)0
(Vm+ K d ((K))
(47)
16

---

Under review as a conference paper at ICLR 2021
where q( {KK)= o2(K2 log K and similarly we have 20()")
((+1) x (- (-Kvm++ m + m + d q(K).
(48)
(11- < - k (K)):
(49)
m Z'(+1) x (-K / (()).
(50)
Therefore, we derive
A
2'(4) - (-1)
(51)
B
7(+1) 7(-1)
(-K + V m + (()) - k k
x
(52)
(- KVm + Vm d + (k)) mm (K))
Vm+g(8
vm
(53)
V m d ((K) vm + d
V m + + (K) + vm d
(54)
Vm+ + g(K) + vm
= -((-}}) m
d
)b)
(55)
and
C g(()
(56)
D g(c)
where ('(-1) 22./255222 ('1+) and se5
Finally,
AB
2 (
0((
@((
)$)
(57)
CD
m
17


Reviews:
[{"review_id": "owo6L2ysU0W", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors study adversarially trained classifiers and observe that the accuracy discrepancy between classes is larger than that of standard models. They then propose a theoretical model where this phenomenon provably arises. Finally, they propose a method to reduce the (standard and robust) accuracy discrepancy between classes, by adapting existing methods from the non-adversarial setting.\n\nI found the paper interesting. The original observation is intriguing and rigorously reproduced on a synthetic dataset. Moreover, the method proposed does seem to improve the class disparity of these models.\n\nHowever, I have one major concern: it is not clear that the phenomenon observed is a property of adversarial training. A different explanation could be that robust models have lower accuracy than standard ones, with their robust accuracy being even lower. Should the increase in error be multiplicative (which is the most likely scenario) then it would potentially explain the main observation of the paper without taking into account adversarially training at all. Specifically:\n- Figure 1: Even the standard model accuracy fluctuates across classes. There are clearly classes that have at least 3 times the error rates of others. The situation for the robust model does not seem particularly different, there are also classes with similar multiplicative ratios in their error, the difference just looks larger because the error rate itself is larger. Measuring the ratio between different error rates might shed some light here.\n- Theoretical setting: Looking at equation (12), consider a similar case where the standard error of the robust model is, say, 4 times higher than that of the standard one. In this case, $\\Omega$ would be 4 purely based on the accuracy of the model.\n\nTo summarize, based on the existing arguments in the paper, we cannot tell apart the scenario where robust training causes class disparity due to an inherent property of the method or simply because it increases the model's error rate. Should the latter be the case, the contribution of the paper would be significantly smaller since the underlying phenomenon would have little to do with adversarial training (similar for the proposed method).\n\nAnother issue that was not clear to me is whether the test set is used during training to compute the weighting of the different loss terms (line 5 of algorithm 1). From what I understand, this is the same test set used to report performance, in which case the methodology would be fundamentally flawed and a separate test set (completely unseen) would be needed.\n\nOverall, I do not believe that the paper is ready for publication but I would be willing to update my review based on further discussion.\n\nOther comments (not affecting score):\n- Intro: \"tradition debiasing...\" -> \"traditional debiasing...\"\n- Figure 1: I assume models are trained against an 8/255 adversary, is this correct?\n- Eq 14: The fourth error rate should be R_rob, right?\n- Section 4.3: broken references\n- Section 5: \"fairly robust\" can be interpreted as \"somewhat robust\" which is not the intended meaning of \"fairly\" here\n\n====== POST-RESPONSE UPDATE ======\n\nI appreciate the authors' response and the additional experimental results provided. While I do think that they are a step in the right direction (I hence slightly increased my score to a 5), they still do not address my concerns. Specifically:\n- **Empirical results.** Taking a closer look at Table 1* of the response and the results on randomized smoothing (provided in other responses), I do agree that not all errors increase by the exact same multiplicative factor. At the same time, the discrepancy does not seem to be particularly large---i.e., the ratios have a mean of 2.5 with a standard deviation of <0.5. Clearly, given that this is a real-world dataset, it is natural to expect that the effect of adversarial training is not perfectly linear across all classes.\n- **Theoretical results.** After reading the response, reading the discussion with Reviewer4, and going through the paper again, I am still not convinced that the increase in class disparity cannot be attributed to a large extent to the gap in clean accuracy introduced by robust training. Perhaps there is a cleaner way of formulating there results or highlighting the key components of the analysis that resolves this. However, unfortunately, I still do not find the analysis convincing in its current state.\n\nOverall, I believe that this point is quite nuanced and the existing empirical and theoretical analysis is not sufficient to draw a confident conclusion. Given how this point is at the core of the paper's contribution, I still recommend rejection.", "extra_scores": {}, "extra_text": {}}, {"review_id": "mQrkN9K0pE2", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper introduces a fairness perspective on accuracy performance among distinct classes in the context of adversarial training. It makes an observation that adversarial training algorithms (Madry et al. 2017, Zhang et al. 2019) yield biased performances on CIFAR 10. It also offers a theoretical study under a Gaussian mixture setting that respects Eq. (1). Three versions of fair robust algorithms are proposed and evaluated on CIFAR 10.\n\nStrength:\n\nThe paper makes an interesting observation on a fairness perspective in adversarial training.\nIt provides a theoretical insight that tries to explain the observation in a certain setting Eq. (1) in which low-dimensional robust features play a dominant role in adversarial training.\nWeakness:\n\nWhile the observation is new and interesting, this reviewer wonders whether it happens “fundamentally” in any adversarial training setting. The theoretical study definitely offers a deeper understanding, yet it is analyzed under a particular setting Eq. (1) which I am not sure if it represents most practically-relevant scenarios. Also, all the experimental results are provided for the two algorithms (Madry et al. 2017, Zhang et al. 2019) under only one dataset CIFAR 10. As the authors mentioned in Section 6.2, this may be due to particular algorithms and data distribution. If that is the case, it would be good to see that the observation may not happen in other settings. Otherwise (if the observation is fundamental), a thorough theoretical study together with extensive experiments that cover more diverse datasets are desired to be presented.\n\nTheoretical analysis: I can barely follow Theorems 1/2 and their implications stated in the paper. But it was not intuitively clear to me why the unfairness issue occurs in the considered setting. Any elaboration on intuition and insight might help.\n\nThere are many grammatical errors and typos.", "extra_scores": {}, "extra_text": {}}, {"review_id": "lehMp4-SA1k", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Thank you for your response, which cleared up some of my questions so I increased my score to a 5. I would have liked more analytical / empirical evidence to substantiate some of the claims made in response to the questions numbered 4, 5, 12, and 13 in the comments provided by the authors. My overall opinion remains unchanged but I encourage the authors to continue this line of work on build on these results and incorporate the feedback provided by the other reviewers as well.\n\n###\n\nThis work makes an interesting observation that techniques designed to maximize robustness to adversarial examples may have negative consequences on the model's performance. Typically, this has been studied primarily with an emphasis on accuracy. Here the authors look at fairness instead.\n\n- The first sentence of the introduction makes strong claim about the implications of adversarial examples without providing appropriate references or supporting the claim with evidence.\n- Adversarial examples do not have to be \"imperceptible\", the main requirement is that the perturbation introduced does not affect semantics of the input.\n- Adversarial training has been subsumed into verifiable approaches to defending against p-norm bounded adversaries (e.g., randomized smoothing, etc.). Could you comment on why these approaches were left out of the experiments here? They would be able to prove robustness is achieved uniformly for each training example.\n- If we know that adversarial training impacts accuracy (as recently re-demonstrated by Tsipras et al.), could you comment on why it would have been expected that fairness would not be impacted given that the training set is balanced among classes and certain pairs of classes are closer to one-another under the p-norm than other pairs of classes? \n- The presentation is somewhat confusing given that it entangles the notions of fairness and safety. This seems likely to lead to confusion from the readers. I would recommend splitting the work into two distinct studies: one of the worst-case performance of the classifiers in the absence of an adversary (i.e., safety) and the other being on fairness itself. Unless the authors wish to demonstrate explicitly the connection between safety and certain definitions of fairness, in which case this connection should be presented more upfront in the paper.\n- Typo/grammar in \"In the lower dimensional space, an optimal linear model is more sensitive to the inherent data distributional difference and be biased when making predictions.\"\n- Could you motivate the choice of dataset here? Why is CIFAR10 a good dataset to study fairness?\n- Figure 1: I would recommend using another type of graph. Typically series like this are used to represent data that is sequentially meaningful whereas here a bar plot or something along these lines would probably convey your message better.\n- The paper does not explicitly state which fairness definition is used (for instance in Section 2) while this is important here. In particular, the definition considered seems to be based on parity but this is not explicitly stated there. Would the approach considered here apply to other definitions?\n- It is not clear what is the applicability of the analysis based on robust/non-robust features is beyond the toy dataset considered here. Could you motivate better why this toy dataset matters in the context you are considering here?\n- Grammar error in the statement of Theorem 1\n- Grammar in \"should not be too larger\" (page 5)\n- Could you motivate the choice of different thresholds on the natural model error and adversarial model error?\n- Missing Section reference on page 6\n- Section 4.3 discusses the intricacies and possible tensions between natural and robust accuracies and fairness. The re-margin strategy proposed seems to be at odds with recent results demonstrating that improving robustness to p-norm perturbations conflicts with generalization (see Tramer et al. in ICML20). Have you thought about the suitability of the p-norm constraint placed on adversaries in the context of your approach? Does this create an artificial tension with fairness for the same reason that it creates an artificial tension with accuracy (because the p-norm is not aligned with class semantics).\n- Related to this, Table 1 suggests that there is increased tension with natural accuracy when employing FRL compared to the baselines considered. Is that due to the point mentioned above or to what is discussed in Section 4.2?\n- The experimental evaluation is limited. It is hard to draw conclusions from one dataset. In particular, I would encourage the authors to consider datasets which are not balanced, which are from other domains than images, and which have a natural motivation for achieving \"fairness\" due to the task being solved. \n- Experiments should also include multiple runs and report variance.\n- The discussion of related work is a bit \"thin\".", "extra_scores": {}, "extra_text": {}}, {"review_id": "1yz66cT_t7", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper begins with the empirical observation that adversarially trained models often exhibit a large different in clean (and robust) accuracies across different classes. This is an important observation, which I have not seen in published work (though I believe is generally understood by many practitioners with the robustness community), and is an important contribution.\n\nThe paper then proceeds to use a theoretical example (adapted from the model in Tsipras et al 2018) where adversarial training increases the accuracy difference across classes. More on this later. The paper proposes an algorithm, Fair Robust Learning (FRL), to address this issue. The starting point is a standard Lagrangian-based approach to approximately ensure constraints that the performance for each class should be close to the overall performance. The FRL algorithm then proposes 2 modifications. First, rather than applying the constraint on the robust errors directly, they decompose the robust error into clean error and a robustness term (similar to TRADES) and approximately maintain constraints on both terms. Second, for difficult classes, they propose increasing the adversarial radius, instead of/in addition to only increasing the corresponding Lagrange multiplier. The experiments demonstrate that their FRL approach improves the worst-class clean and robust errors, relative to both standard adversarial training and the baseline Lagrangian approach, without losing much in average-class errors.\n\nOverall, I believe the paper is fairly well-executed and investigates an important topic. The motivating empirical observations are an important contribution. The proposed approach is natural and well-motivated, and the experiments show improvements in the worst-class errors, as should be expected conceptually. The empirical explorations of different variations on the core idea are also valuable.\n\nThere were some points in the paper which I believe could be improved.\nFor me, the theoretical example does not add much insight regarding why this effect occurs. \n- Regarding Theorem 1, the comment says that \"when the term A is large, the model has close clean errors between the 2 classes, namely $R_{nat}(f, -1) ~ R_{nat}(f, +1)$. Assuming A^2 >> q(K), the terms simplify to:\n\n$$R_{nat}(f, -1) = P[N(0, 1) \\leq A(1- \\sqrt{K})]$$\n$$R_{nat}(f, +1) = P[N(0, 1) \\leq A(1- K)]$$\n\nwhich means we have a $\\sqrt{K}$ factor difference between the two errors. In this case, it seems that even before adversarial training, we see a difference in class errors between the classes.\n\n- Additionally, in Theorem 2, using a similar approximation B^2 >> q(K), we'd have\n\n$$R_{nat}(f_{adv}, -1) = P[N(0, 1) \\leq B(1- \\sqrt{K})]$$\n$$R_{nat}(f_{adv}, +1) = P[N(0, 1) \\leq B(1- K)]$$\n\nwhere the two terms differ by a $\\sqrt{K}$ factor again, only the leading constant is different.\n\n- The claim in Equation 12 that \"the ratio $\\Omega$ is large (e.g. >1)\" seems to be the result the section builds to, but it seems it is more naturally explained by the fact that if there is already a difference in clean errors, since adversarial training will increase these errors (Tsipras et al 2018), then adversarial training will also scale up the difference in these class-wise errors.\n\nIt's possible that this explanation is actually what's happening, and it doesn't seem clearly inconsistent with the empirical data, since in Figure 1 (left), there are already class-wise differences, which generally match the shape of the class-wise differences after adversarial training. But it does seem very different from the story the paper is trying to tell, and it would be good to figure out which one it is (or acknowledge both).\n\nMoving on to the experiments:\n\n- The TRADES baseline has robust accuracy almost 10% lower than the standard TRADES model - can you comment on this?\n- Section 4.3 comments \"we claim that only upweighting its cost (or Reweight) could not succeed to fulfill the cost-sensitive classification goal in adversarial setting\" - could you explain why this is? (after all, reweighting is the basic approach used here too)\n- How does the Re-Margin approach compare to the Reweight approach, when the reweighting also involves sweeping over the training $\\epsilon$?\n- It's nice to see the code. Hopefully this can also be released along with the paper if accepted.\n\nOther notes:\n\n- It would be nice to connect this to the broader fairness literature. This is unfortunately not my area of expertise, but difference in performances between classes (or demographic groups) is a very common metric, and it would be nice to relate this work more closely to work there. \"Distributionally Robust Neural Networks for Group Shifts\" (Sagawa et al 2020) is a (somewhat arbitrary) example of work which seems related, but I'm sure there are many more. It would be great to connect to this, especially papers which address why/when difference in performance across classes is expected.\n\nI give the paper a 6 overall, though could adjust my rating in either direction depending on the author feedback.\n\n\n________________\n\nEDIT: Score changed from 6 to 5 during discussion, see comments below.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper examines adversarially trained robust models, and finds that accuracy disparity is higher than for standard models. The authors introduce a method they call Fair Robust Learning using Lagrange multipliers to minimize overall robust error while constraining the accuracy discrepancy between classes.  

In discussion, consensus was reached that the observations and approach are interesting but the paper is not yet ready for publication. The main concern is that it is not clear if the class accuracy disparity is due to adversarial training, or simply due to lower accuracy in general. Please see reviews and public discussion for further details.
Decision: Reject