Towards Fair Classification against Poisoning Attacks
Authors: Han Xu
Xiaorui Liu
Yuxuan Wan
Jiliang Tang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Poisoning Attacks
Fairness
Robustness

Abstract: Fair classification aims to stress the classification models to achieve the equality (treatment or prediction quality) among different sensitive groups. However, fair classification can be under the risk of poisoning attacks which deliberately insert malicious training samples to manipulate the trained classifiers' performance. In this work, we study the poisoning scenario where the attacker can insert a small fraction of samples into training data, with arbitrary sensitive attributes as well as other predictive features. We demonstrate that the fairly trained classifiers can be greatly vulnerable to such poisoning attacks, with much worse accuracy & fairness trade-off, even when we apply some of the most effective defenses (originally proposed to defend traditional classification tasks). As countermeasures to defend fair classification tasks, we propose a general and theoretically guaranteed framework which accommodates traditional defense methods to fair classification against poisoning attacks. Through extensive experiments, the results validate that the proposed defense framework obtains better robustness in terms of accuracy and fairness than baseline methods.

Under review as conference paper at ICLR 2023
TOWARDS FAIR CLASSIFICATION AGAINST POISONING AT- TACKS
Anonymous authors Paper under double-blin blind review
ABSTRACT
Fair classification aims to stress the classification models to achieve the equality (treatment or prediction quality) among different sensitive groups. However, fair classification can be under the risk of poisoning attacks that deliberately insert malicious training samples to manipulate the trained classifiers' performance. In this work, we study the poisoning scenario where the attacker can insert a small fraction of samples into training data, with arbitrary sensitive attributes well as other predictive features. We demonstrate that the fairly trained classifiers can be greatly vulnerable to such poisoning attacks, with much worse accuracy & fairness trade-off, even when we apply some of the most effective defenses (originally proposed to defend traditional classification tasks). As countermeasures to defend fair classification tasks, we propose a general and theoretically guaranteed framework which accommodates traditional defense methods to fair classification against poisoning attacks. Through extensive experiments, the results validate that the proposed defense framework obtains better robustness in terms of accuracy and fairness than representative bascline methods.
1 INTRODUCTION
Data poisoning attacks (Biggio et al., 2012] Chen et al., 2017 Steinhardt et al. 2017] have brought huge safety concerns for machine learning systems that are trained on data collected from public resources (Konečný et al., 2016 Weller & Romney| 1988). For example, the studies (Biggio et al. 2012. Mei & Zhu 2015) Burkard & Lagesse| 2017 Steinbaat et al. 2017) have shown that an attacker can inject only small fraction of take data into the training pool of a classification model and intensely degrade its accuracy. As countermeasures against data poisoning attacks, there are defense methods (Steinhardt et al.) 2017 Diakonikolas et al. 2019) which can successfully identify the poisoning samples and sanitize the training dataset.
Recently, in addition model safety, people have also paid significant attention to fairness. They stress that machine learning models should provide the "equalized" treatment or "equalized" prediction quality among groups of population (Hardt et al. 2016; Agarwal et al.. 2018 Donini al., 2018 Zafar et al. 2017). Since the fair classification problems are human-society related, it is highly possible that training data is provided by humans, which can cause high accessibility for adversarial attackers to inject malicious data. Therefore, fair classification algorithms are also prone to be threatened by poisoning attacks. Since fair classification problems have distinct optimization objectives & optimization processes from traditional classification, a natural question is: Can we protect fair classification from data poisoning attacks? In other words, are existing defenses sufficient to defend fair classification models?
To answer these questions, we first conduct a preliminary study on Adult Census Dataset to explore whether existing defenses can protect fair classification algorithms (see Section 3.2). In this work, we focus on representa- tive defense methods including k-NN Defense (Koh et al.] (021) and SEVER (Diakonikolas et al. 2019). To fully exploit their vulnerability to poisoning attacks, we introduce a new attacking algorithm F-Artack, where the attacker aims to cause the failure of fair classification. In detail, by injecting the poisoning samples, the attacker aims to mislead the trained classifier such that cannot achieve good accuracy, or not satisfy the faimess constraints. From the preliminary results, we find that both k-NN defense and SEVER will have an obvious accuracy or fairness degradation after attacking. Moreover, we also compare F-Attack with one of the strongest poisoning attacks Min-Max Attack Steinhardt et al. (2017) (which is devised for traditional classification). The result demonstrates that our proposed F-Attack has a better attacking effect compared to Min-Max Attack. In

---

Under review as conference paper at ICLR 2023
conclusion, our preliminary study highlights the vulnerability of fair classification against poisoning attacks, especially against F-Attack.
In this paper, we further propose defense framework, Robust Fair Classification (RFC), to improve the robustness of fair classification against poisoning attacks. Different from existing defenses, our method aims scout abnormal samples from each individual sensitive subgroup in each class. To achieve this goal, RFC first applies the similar strategy as the works (Diakonikolas et al.) 2017 2019), to find abnormal data samples which significantly deviate from the distribution of other (clean) samples. Based on theoretical analysis, we verify that RFC can exclude more poisoning samples than clean samples in each step. Moreover, to further avoid removing too many clean samples, we introduce an Sertin--Dta-Saniin we remove possible poisoning set from a single subgroup of single class and test the retrained models' performance on clean validation set. This helps us locate the subgroup which contains the most poisoning samples. Through extensive experiments on two benchmark datasets, Adult Census Dataset and COMPAS, we validate the effectiveness of our defense. Our key contributions are summarized as:
process: in each iteration,
We devise strong attack method to poison fair classification and demonstrate the vulnerability of fair classification under the protection of traditional defenses to poisoning attacks.
We propose an efficient, and principled framework, Robust Fair Classification (RFC). Extensive experiments and theoretical analysis demonstrate the effectiveness and reliability of the proposed framework.
2 PROBLEM STATEMENT AND NOTATIONS
In this section, we formally define the setting of our studied problem and necessary notations.
Fair Classification. In this paper, we focus on the classification problems which incorporate group-level fairness criteria. First, let x S Rd be random vector denoting the moon-sensitive) features, with label y E y = (Yi, Yn with m classes, sensitive attribute E 2 = 1 Z2, ..... Z with k groups. Let f(X, w) represent a classifier with parameters E W. Then, a fair classification problem can be defined as:
min 18 [[[[(( w).v) w) s.t. ()( S V
(1)
where the function [[[(-)] is the expected loss on test distribution, and F the unfaimess tolerance. The constraint function g (w) E h(w, xr 2) = j represents the desired faimees metrie for each group Z. For example, in binary classification problems, we use fII w) > 0 to indicate positive classification outcome. Then, (w x, y) (fr w) > 0) [[] w) > 0) refers to equalized positive rates in the equalized treatment
criterion (Mehrabi et al. 2021). Similarly, Nww, x. v) = (z, w) > 0ly = +1) TR. (J((z, w)) > 0ly = +1)
is for equalizing true/ / false positive rates the equalized odds (Hardt et al. 2016). Given any training dataset D, we define the empirical loss function L(D, w) as the average loss value of the model, and gj (D, w) is the empirical fairness constraint function.
In our paper, we assume that the clean training samples sampled from the true distribution D following the density P(x, 2). We also use Du. to denote the distribution of (clean) samples given by y = Yu and = Zu, which has density P(I, y, 21y = Ya, 2 = Z.).
Poisoning A Attack (c-poisoning model). In our paper, we consider the poisoning attack following the scenario.
Given fair classification task, the poisoned dataset is generated follows: first, n clean samples De [(T, yi, 2:)]_=1 are drawn from DD to form the clean training set. Then, an adversary is allowed to insert an e fraction of De with arbitrary choices of Dp [1 yi, 2 We can define such poisoned training set De U Dp as e-poisoning model.
3 FAIR CLASSIFICATION IS VULNERABLE TO POISONING ATTACKS
In this section, we first introduce the algorithm of our proposed attack F-Attack under the e-poisoning model. Then, we conduct empirical studies evaluate the robustness of fair classification algorithms (and popular defenses) against F-Attack and baseline attacks.
2

---

Under review conference paper at ICLR 2023
3.1 F-ATTACK: POISONING ATTACKS FOR FAIR CLASSIFICATION
Given a specific fair classification task, we consider that the attacker aims to contaminate the training set, such that applying existing algorithms cannot successfully fulfill the fair classification goal. Note that for fair classification tasks, both accuracy and fairness are the desired properties and they always have strong tension in practice Menon & Williamson (2018). Therefore, in our attack, we consider misleading the training algorithms such that at least one of the two criteria is unsatisfied. Formally, we define the attacker's objective as E2, where the attacker inserts a poisoning set Dr with size en the feasible injection space F to achieve:
B. B[((S(o,, w²) s.t. w" a min L(Dc U UD, w). DPCF
max
(2)
weH
It means for the classifier w* that trained on Dc Dp and has a low empirical loss L(Dc U Dp, w"), it falls in the space H fair. it will have large expected loss (on the test distribution D). Here, Hfair is the space of models (with limited norm) that satisfy the faimess criteria clean distribution D. To have closer look Eq. (2. we discuss case by case. Suppose we obtain w" by fair classification on the set De U.p, there are cases:
1. =² F.Jai:: The faimess criteria (on the test set) is not satisfied.
2. w² E Effair: Since w" is trained on De U Dp to have low L(D, U w))"", ), w" will have high test error.
For each case, the model w' will either have unsatisfactory accuracy or unsatisfactory fairness. Next, we simplify the objective and constraints in (q. to transform it into a solvable problem. We first conduct relaxations of the objective in (2) (similar to the works (Steinhardt et al. 2017 Koh et al.) 2021)):
E ((ICf(X, w),
L(Do. w) L(De,w) w) + EL(Dp, w) = (1 +1++))4 LDD U UD,,w)
Specifically, approximation (i) holds if clean training data De has sufficient samples and is close to the test distribution D, and model w is appropriately regularized. The upper bound (ii) holds because of the non non-negativity of loss values. The upper bound (ii) can be tight the fraction of poisoning samples small. Thus, we transfer (2) to a bi-level optimization problem between w and Dp. If the model J(.) and loss I(.) are convex, we can further swap them to get a max form as:
max min Dr Wf
L(DoUDp.W)-
min max L(DcUDp, WEHfate DrCF
Our proposed F-Attack is to solve Eq. which is shown in Algorithm It solves a saddle point problem to alternatively find the worst attack points (x. y, 2) w.r.t the current model and then update the model in the direction of the attack point. In detail, in Step (1), given the current model w. solve the inner maximization problem to maximize L(De U Dp, w). It is equal to finding sample (x, y, 2) with the maximal loss:
max DrCF
L(De Dp, w) = L(Do, w) + max [[(xx, w).y). (a.g,))FF
In Step (2), we update w to minimize L(Dc Up, Note that in Step (2), we should also constrain the model to fall into the fair model space H. fair- Thus, when update in Step (2), we also penalize the faimess violation of w. Here, we calculate the fairness violation as (gy( Do, w) - -) (with weight parameter À >> 0) on the clean set De approximate the fairness violation on real data D.
Algorithm 1: Algorithm of F-Attack
Input :Clean data Dc, number of poisoned samples en, feasible set F, Fairness constraint functions g;(-) and tolerance T, (j 1,.... 3 À 0, m > 0, warm-up steps Néurn-
Output: :A poisoning set D':
for 1.... Aburn + en do
1.Solve inner maximization: (I, = arg max (r,,)))) ((f(z, w). v) and let Dp en x {(x,8,2))
2. Solve outer minimization: w n V.. U Dp. w) (9, w) +)+)
if > Nburn then
Dj = Dj. U ((x,y,2).
end
end
3

---

[|2, where Under review as conference paper at ICLR 2023
3.2 PRELIMINARY STUDY ON ADULT CENSUS DATASET
Adult Census Dataset. In this subsection, we conduct an ex- periment on Adult Census Dataset Kohavi et al. (196)), to test whether F-Attack can poison fair classification methods and whether existing defense methods can resist F-Attack. Here, we focus on the fairness criteria: Equalized True Positive Rate (TPR) Hardt et al. (2016) between the genders, and we apply the constrained optimization method Donini et al. (2018) to train linear classifiers to fulfill the fair classification objective. It is worth mentioning that, this dataset contains many categorical features, such as marital-status, occupation, etc. For simplicity, we pre-process the dataset by transforming categorical features into a continuous space that is spanned by the first 15 princi- ple directions of training (categorical) data. More details of the pre- pre-ppoceesing procedure can be found in Appendix AA.2
Ye 2m0
Y=
Y=1,2=0
V=1.2=3
(L.
Figure 1: PCA visualization of Clean Samples and Poisoning Samples
Defense Methods. Besides naïve fair classification, we mainly consider two representative data-sanitization defenses, which are existing popular methods to defend against poisoning attacks:
k-NN Defense (Koh et al. 2027). This method removes the samples that are far from their k nearest
neighbors. In detail, the k-NN defense calculates the "abnormal" score as qi ||2; 1 x (k:2)
(R.3) is the k-th nearest neighbor to sample vi in class In this paper, we set k: 5. 5.
SEVER. Diakonikolas ef al. (2019) This method aims to find abnormal samples by tracing abnormal gradients. In each iteration, we first train fair classifier ʃ (with fixed r) and calculate the gradient of loss w.r.t the weight w for each training sample (Iz, 3.). and get the normalized gradient matrix
Q = Vul(f(I,, w), (.) 1 Eリ= Vul(f(zz, w). v5) i=1,... SEVER flags the samples with large
"abnormal" score qi = v)2 as abnormal samples, where a is the top right singular vector of Q. Intuitively, the "abnormal" samples make a great contribution to the variation of the gradient matrix Q which suggests their gradients can significantly deviate from the gradients of other samples.
Results. In our experiments, we insert 10% poisoning samples to the training set, and define the feasible injection set F to be ((r, z) || (icc-/0 Myll S d). where d is a fixed radius. This will constrain the inserted samples not too far from the center of their labeled class, to evade potential defense. Since F is not related to sensitive attribute z, during F-Attack, we generate poisoning samples with a fixed 2 to be 0 (female) or I (male). During fairness training. we train multiple models with various hyperparameters to control the unfairness tolerance on the training set (following (Lamy et al., (019)). Then, we report the test performance when it has the best validation performance (which considers both accuracy and fairness, see Section 4 Tabic]] we report the performance for the defense methods. From the result, we can see: all training methods have a significant performance degradation under F-Attack. For example, under F-Attack (z = 0), the SEVER defense has z 4% accuracy drop and 2% fairness drop. This suggests that defenses such SEVER and k-NN can be greatly vulnerable to poisoning attacks in fair classification. Moreover, we compare F-Attack with a baseline attack method Min-Max (Steinhardt et al., 2017,Kok, et al., al., (202I), which is one of the strongest attacks for traditional classification. It also solves Fq. but does not constrain w E Hffiir- From Table [ we can see that Min-Max has worse attacking performance than F-Attack, by causing slighter performance degradation. This result highlights the threat of F-Attack to fair classification.
Eq. (9) for more details). In
Table I: F-Attack vs. Min-Max on Adult Census Dataset
No Attack
Min-Max (z= 0)
Min-Max(z== 1)
F-Attack (z = 0)
F-Attack (z I)
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
No Defense.
0.811
0.962
0,801
0.958
0.799
0.851
0.768
0.956
0.793
0.803
k-NN.
0.794
0.952
0.681
0.936
0.680
0.904
0.655
0.951
0.695
0.895
SEVER.
0.812
0.969
0.798
0.958
0.797
0.967
0.773
0.942
0.772
0.943
We report the "goodness of faimess" as 1 - Unfair, i.c., 1 - [TPR(>
(2 = 0) - (= = 1) in Table
4

---

Under review as conference paper at ICLR 2023
Discussion. To have deeper understanding on the behavior of F-Attack, in Figure 1 we visualize the clean samples and poisoning samples (via F-Attack and Min-Max Attack) in a 2-dim projected space (via PCA). From the figure, can see that: compared to Min-Max Attack (red), the samples obtained by F-Attack (yellow) have a smaller distance to the clean samples in their labeled class same feasible injection set FF. It is because F-Attack aims to find samples with maximal loss (Eq,44) for fair classifiers, so the generated samples do not have the maximal loss for traditional classifiers, Thus, the poisoning samples from F-Attacks are closer to their labeled class. This fact helps explain why F-Attack is more insidious than Min-Max Attack under the detection of traditional defenses, such as SEVER. In Appendix [A.4] we further conduct experiments on Adult Census Dataset focused on a Neural Network based classification model, which demonstrates that the idea of F-Attack has potential to can be adapted to DNN models.
1, although they are constrained in the
4 ROBUST FAIR CLASSIFICATION (RFC)
Motivated by studies in Section new defenses are desired to protect fair classification against poisoning attacks, especially against F-Attack. In this section, we first introduce a novel defense framework called Robust Fair Classification (RFC), and we provide a theoretical study to further understand the mechanism of RFC. In Section S we conduct empirical studies to validate the robustness of RFC in practice.
4.1 ROBUST FAIR CLASSIFICATION (RFC)
Based on the discussion in Section 3 F-Attack can evade traditional defenses such as SEVER and k-NN Defense, because the generated poisoning samples are close to the clean data of their labeled class. However, we assume that they may deviate from the distribution of clean samples in their labeled subgroup (the data distribution D given y and z). Refer to the Figure 2 which shows the location of poisoning samples generated via F-Atlack (z = 0) and clean samples in subgroup (y = 1, 0) in the 2D projected space. It suggests that the poisoning samples greatly contaminate the information of distri- bution P(x,, =1y = 1, 2 0) in the training data. Thus, the injected poisoniend samples will not only confuse the original prediction task from x to y. but also greatly disturb the faimess constraints (I. This observation motivates us to propose a new defense method that can scout abnormal samples from each individual subgroup in each class. Next, will introduce the details of our proposed defense RFC.
Figure 2: PCA visualization in (1, 0)
Centered Data Matrix& & Poisoning Score. Our method shares a similar high-level idea as (Diakonikolas al. 2017 2019), to find data points that systematically deviate from the distribution of other (clean) samples. Specifically, in our method, given (poisoned) dataset D, we repeatedly scout the poisoning samples from each subgroup ((x, y, z Yu, z Zu), where we use (u, v) to denote the index of each subgroup and class. In the later parts, we use n to denote the samples in D(z, y, ely = Yu, 2 = Zz) for simplicity, Then, we define:
Qu,
(s.....))DD..
to be the centered data matrix of training samples Du. and nu,v is the size of the set Du.
For each Qu.,, the top right singular vector Va." of Qu.e is the direction which explains the variation of the data distribution in (Y., Z.). Similar to the studies in Diakonikolas et al. 20117 2019), we conjecture: the poisoning samples are deviated from clean samples, so they will take the major responsibility for the variation of
the data matrix Qu.7. In this way, they will have high alignments with the direction of yu.". Thus, we define the Poisoning Score q²t"" for each training sample (I,, Vos 2.) in the D as:
q² (x,) Q. (V%.OTT.
(6)
Notably, the poisoned samples are likely have the same or opposite direction with the top right singular vector yu." but the poisoning samples should share the same direction. Thus, in our method, we define two Proposed
5

---

Under review conference paper at ICLR 2023
Poisoning Sets for each (u, v), so that one of the two sets is likely to have poisoning samples:
" (Igs (x,)) 74.9" (x;) > 0}; P (a; q q"' (24) > 7- 0};
(7)
In Eq. we set the N (or y-) to be the q-th (q- 90) percentile of the given all Poisoning Scores (or negative poisoning Scores) in D so that each proposed poisoning set only contains small portion of D".", In practice, we will repeatedly test whether removing the proposed poisoning sets can help improve the retrained model's performance (both accuracy and fairness) on a clean validation set. It helps to decide whether the proposed poisoning set contains poisoning samples. In Section we will further conduct a theoretical analysis to show that the poisoning samples are more likely to have higher poisoning scores.
Fair Classification by Excluding Poisoning Set. Finally, given a (poisoned) training set D as well as the proposed poisoning sets, we can keep retraining fair classifiers by excluding proposed poisoning sets:
W+ arg min L(f(X, w) Y; D P+") s.t. gj(D) P :w) S Ty V
"EW
(8)
arg min L(S(X, w), Y; D P! s.t. gj(D P= 3 Tj Vj
wEW
In practice, our proposed RFC method repeatedly proposes potential poisoning sets for each individual subgroup in each class D until finding the best poisoning set among all choices. The introduction of the procedure of RFC, which is an Online Data Sanitization process. Specifically, during each iteration of RFC, for each D".", we first calculate the poisoning scores and the proposed poisoning sets (Step (1)) Then, we remove the proposed poisoning set from the dataset D, and conduct fair classification on D (Step (3)). In Step (4), we evaluate the retrained classifier on a clean separated validation set and find the best-proposed poisoning set which results in the highest validation performance. Notably, we measure the validation performance by considering both the accuracy and fairness criteria, by defining:
the detailed
ValScore = Pr.(f(z, v)
Σ (0, w) - +)++ JEZ
where F is a unfairness tolerance threshold and À is positive number (we set X = 3 in this paper). The second term penalizes the models if some subgroups' unfairness violation is over т. Finally, we remove the proposed poisoning set which results in the highest ValScore and conduct the next round of searching (Step (6)).
Algorithm 2: Robust Fair Classification (RFC) An Online Data Sanitization Algorithm
Input :An c-poisoning model with dataset D = {(2,, Ui 4)) im Iterations T of RFC.
Output A fair classifier
while < T do
for Yu y, Zu E Z do
1. Get the Centered Data Matrix Q".7" and Poisoning Score q; u.v following Fq. and Eq, (
2. Get the Proposed Poisoning Sets: P":"" and p"." following Eq.
3. Conduct fair classification by removing Proposed Poisoning Ser, via Eq. 8 and get w
4. Record the performance get ValScore on a separated (clean) validation set for each w4."
end
5. Get the best proposed poisoning set P which achieves the highest ValScore across all u, and Pl.".
6. Removing the best proposed poisoning set from D and set = D \ P
end
Remarkably, it is also worth mentioning that the framework RFC is also possible to be extended to various model architectures, such as Deep Neural Networks (DNNs), for robust fair classification. For example, we can apply the Energy-based Out-of Distribution Detection Liu et al. (2020) to find the abnormal samples from each D Then, we follow a similar manner RFC to propose poisoning sets and conduct fair classification. We will leave the study in DNNs for future exploration.
4.2 THEORETICAL ANALYSIS
In this subsection, we conduct theoretical analysis further help understand the behavior of RFC, especially understand the role of calculating poisoning scores in finding poisoning samples. In particular, we consider a
6

---

Under review as conference paper at ICLR 2023
simple theoretical setting where the clean samples from each group D=." follow a distinct Gaussian distribution Du. ~ N (nth," -".",, with center and covariance matrix "u""E""" ). In the following theorem, we will show: when there are (poisoning) samples that deviate from the clean samples of D".", by having a center ja which is far from the center of clean samples, they will have larger squared poisoning scores (Eq(6) than clean samples. Thus, the proposed poisoning sets (Eq.(7) are likely to contain more poisoning samples than clean samples. For simplicity, we use D and [(, 2) to denote the clean distribution of a given group D".".
Theorem 4.1. Suppose that a set of "clean" samples Spood with size n are i.id sampled from distribution [(, Σ) where 3 o²I. There is a set of "bad" samples Stad with size np =n/K, K > 1 and center 11 /ull2 d, d = 7 a. Then, the average squared poisoning scores of clean samples and bad samples have a relationship:
K - K+1 72 (K - 1K+
BieSoad [s?(x:) - EiEsyoodd [r]( 2
Theorem suggests that the difference between the average (squared) poisoning scores of Soud and Syood is (K+1)²
controlled by y and the sample ratio K. Since K 1, if > K-1 (which suggests the poisoning samples are sufficiently far from clean distribution), we can get the conclusion that the difference is positive. Thus, removing samples with the highest positive (or lowest negative) poisoning scores (as (7) will help to eliminate more poisoning samples than clean samples. If -2 is small, the poisoning samples are close the true distribution, which will cause the poisoning samples to have limited influence on the model performance. The detailed proof of Theorem 4.1 is deferred to Appendix [A.I] In our algorithm of RFC, we alternatively check each proposed poisoning set and see whether removing it helps improve the retrained models' performance. This will also avoid removing too many clean samples. In Appendix A.5] we provide extra discussion on the precision of the detection of RFC, which demonstrates that RFC will not remove too many clean samples in practice.
5
EXPERIMENT
5.1
EXPERIMENTAL SETUP
In this section, we conduct comprehensive experiments to validate the effectiveness of our proposed attack and defense, two benchmark datasets, Adult Census Dataset and COMPAS Dataset. In this part, we only consider Equalized True Positive Rate (TPR) between different sensitive subgroups, which is optimized via the fair classification method (Donini et al., 2018). When applying (Donini et al. 2018), we train multiple models with various hyperparameters to control the unfairness tolerance the training set (following (Lamy et al. 2019)). Then, we report the test performance when it has the best validation performance (which considers both accuracy and fairness, Section 4 Eq. 9)) In Appendix [A.3] we provide additional results for different type of faimess "Equalized Treatment", and a different fair classification method (Zafar et al. 2017). The implementation can be
found at http:htttt/// http:////nnoooouuuu 4open. sci wwciiencerr/ee att_ttttackkeo tacctack-44 4017/ 7/
A Attacks:
We consider that the training set can be contaminated by: Label Flipping (Paudice et al. 2018). and Sensitive Attribute Flipping (Wang al. 2020). We also consider the attack methods, Min-Max and F-Attack, which are introduced in Section 3 Notably, y, for each method, we assume the poisoning samples are constrained in the sample feasible injection set F {(=,, y, 2) ||x- pyll S d). which limit the poisoning samples' distance to the class center. Thus, for Min-Max and F-Attack, we assign the generated samples to have a pre-defined sensitive attribute z 0 or 1. Furthermore, we introduce an additional attack method "F-Attack"" which has the same algorithm with F-Attack but have different feasible injection set: F = {ar, V, z) ||x /yy=zll S d), where My. is the center of the group Da. Because this feasible injection set is related the sensitive attribute z, we don't 't need to pre-define z during F-attack*. Remarkably, this attack aims to test the robustness of RFC, because the major goal of RFC is to find samples in each group D which are far from My... Thus, F-Attack is possible to evade RFC by constraining the poisoning samples' distance Myy.. In Appendix A.3 we also report the performance of all attacks & defenses under different choices of radius d.
Baseline Defenses. To validate the effécctiveness of RFC, include baseline defense methods: (1) the naive method which does not apply any defense strategies; (2) SEVER (Diakonikolas et al., 2019), which are representative defenses for traditional classification tasks. We apply (Donini al. 2018)) on the sanitized dataset by SEVER. In addition, we also include (3) Roh et al. (2020). which is a method to defend fair classification methods against label flipping attacks. It leverages adversarial training strategy Zhang et al. (2018); and (4) the method (Wang et al. 2020) applies Distributional Robust Opitmization (DRO) to improve robustness when labels
7

---

Under review conference paper at ICLR 2023
Table 2: RFC Baseline Methods' Performance against Poisoning Attacks on Adult Census Dataset.
No Attack
Label Flip(10%)
Label Flip(20%)
Attr.
(10)
Attr. 2lip(20%
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
No Defense.
0.817
0.963
0.805
0,966
0.800
0.968
0.812
0.945
0.804
0.951
SEVER.
0.812
0.960
0.805
0,967
0.803
0.960
0.813
0.937
0.803
0,944
Wang-
0.809
0.958
0.793
0,970
0.779
0.961
0.809
0,945
0.813
0.935
Roh.
0.805
0.948
0.798
0,937
0.788
0.939
0.800
0,950
0.795
0,943
RFC.
0.811
0.959
0.807
0.973
0.796
0.966
0.805
0.967
0.802
0.965
(e= 10%)
Min-Max(z = 0)
Min-Max(z 1)
F-Attack( (z 0)
F-Attack = 1)
F-Aftuck
Acc.
Fair.
Acc.
Fuir.
Acc.
Fair.
Acc.
Fuir.
Acc.
Fair.
No Defense.
0,801
0.969
0.797
0.864
0.769
0.966
0.796
0.804
0.799
0.799
SEVER.
0.800
0.960
0.795
0.954
0.778
0.945
0.773
0.937
0.772
0.956
Wang-
0.782
0.976
0.783
0.968
0.779
0.956
0.791
0.948
0.792
0.945
Roh.
0.780
0.963
0.782
0,961
0.765
0.959
0.766
0.956
0.776
0.944
RFC.
0.802
0.967
0.811
0.946
0.803
0.950
0.808
0.952
0.809
0.951
(c 15%)
Min-Max(Z = 0)
Min-Ma((Z I)
F-Attack(z
F-Attack(2 = 1)
F-Attack
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
No Defense.
0.792
0.961
0.787
0.837
0.686
0.953
0.775
0.779
0.788
0.760
SEVER.
0.792
0.959
0.783
0.978
0.765
0.952
0.755
0,909
0.765
0.927
Wang-
0.777
0.970
0.779
0,965
0.738
0.955
0.762
0.960
0.711
0.955
Roh.
0.726
0.948
0.748
0,957
0.722
0.962
0.764
0,929
0.774
0.929
RFC.
0.801
0.954
0.796
0,941
0.800
0.963
0.795
0.947
0.802
0,945
and sensitive attributes are contaminated. For baseline methods, we report their performance with the choice of hyperparameter that achieves the optimal ValScore (Eq. 9 on clean validation set.
5.2 EXPERIMENTAL RESULTS
Adult Census Dataset. We first show the results in Adult Census Dataset in Table To further guarantee that the comparison between different defenses is fair, we use balanced clean training dataset where each class has an equal number of samples since the baseline methods such as SEVER can be affected by class imbalance. Under this dataset, we set the desired fairness criteria to be [TPR(= = 0) TPR(= = 1)] < 0.05. In Table 2 we also mark the cases (with brown color) when the algorithms output models with much poorer fairness than the desired fairness. From Table 2 we can see that RFC can achieve good accuracy & fairness among different types of dataset contamination. Especially, under strong attacks such as F-Attack, the accuracy and fairness are only slightly degraded after injecting poisoning samples. However, the baseline methods such as Wang al. (2020) and Roh et al. (2020), will have a clear performance (especially accuracy) degradation under F-Attack. Notably, the attack method F-Attack", has similar attacking performance as F-Attack (= 1). It is because under this dataset, F-Attack* also generates samples that have =
COMPAS Dataset. In COMPAS dataset (Brennan et al. 2009), we consider the same type of fairness criteria, which is Equalized TPR. In this dataset, we consider that the equity is desired among races, which are "Caucasian (z = 0), African-American that in Adult Census Dataset. In this dataset, the number of samples in the group "Hispanic" is much smaller than the other two groups. Thus, we only consider inject poisoning samples to 2 or 1. Table we report the performance of our studied attacks and defense, and we use max [PPR(: = j) - TPR| measure the "goodness" of fairness, where TPR is the averaged TPR in the whole dataset. During training, we set the desired fairness criteria to be maxjez TPR(< = - TPR < 0.15. From the result in Table we can see that RFC is the only method that consistently preserve the model accuracy and faimness after there are poisoning samples injected into the dataset.
1) and Hispanic (z = 2)", and follow the similar preprocessing procedure as
6 RELATED WORKS
Poisoning Attacks. In this section, we introduce related work and discuss how this work differs from prior studies. Data poisoning attacks (Biggio et al. (2012) refer the scenario that models are threatened by adversaries
who insert malicious training samples, in order to take control of the trained model behavior (Li et al. 2020
8

---

Under review conference paper at ICLR 2023
Table 3: RFC Baseline Methods' Performance against Poisoning Attacks on COMPAS Dataset.
No Attack
Label
(10p(10%)
Label Flip(20%)
Attr.
(10)
Attr. Flip(20%
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair
No Defense.
0.656
0.865
0.655
0.866
0.664
0.836
0.677
0.839
0.665
0.847
SEVER.
0.650
0.854
0.677
0.833
0.674
0.835
0.674
0.835
0.667
0.835
Wang-
0.662
0.847
0.650
0,869
0.631
0.863
0.643
0.861
0.659
0.862
Roh.
0.654
0.851
0.646
0.891
0.663
0.823
0.621
0.834
0.615
0.845
RFC.
0.661
0.850
0.676
0.859
0.682
0.841
0.685
0.850
0.667
0.856
(e= 10%)
Min-Max(z = 0)
Min-Max(z 1)
F-Attack( (z 0)
F-Attack = 1)
F-Attuck
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair
No Defense.
0.649
0.845
0.645
0.837
0.665
0.832
0.630
0.860
0.661
0.826
SEVER.
0.634
0.865
0.630
0.859
0.634
0.845
0.627
0.840
0.664
0.826
Wang-
0.648
0.851
0.620
0.855
0.639
0.865
0.624
0,871
0.644
0.868
Roh.
0.644
0,880
0.631
0.865
0.659
0.865
0.633
0.825
0.640
0.858
RFC.
0.656
0.863
0.661
0.834
0.668
0.853
0.667
0.866
0.673
0.845
(c 15%)
Min-Max(z = 0)
Min-Ma((2 I)
F-Attack(z
F-Atlack(2 = 1)
F-Attack
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair
No Defense.
0.676
0.802
0.644
0.863
0.665
0.802
0.630
0,860
0.642
0.831
SEVER.
0.621
0.880
0.620
0.880
0.585
0.890
0.606
0.887
0.663
0.845
Wang-
0.645
0.865
0.631
0,852
0.606
0.842
0.611
0.829
0.624
0.841
Roh.
0.655
0.847
0.628
0,877
0.625
0.865
0.631
0.849
0.621
0.877
RFC.
0.676
0.852
0.659
0,843
0.653
0.848
0.645
0.847
0.672
0.841
Shafahi et al.) 2018). In this work, we concentrate on the untargeted poisoning attacks (Biggio et al. 2012
Koh al. 2021) where the attacker aims to degrade the overall performance of the trained model. To defend
against poisoning attacks, well-established methods (Wilcox 2011 Rubinstein et al., 2009 Steinhardt et al., 2017 Diakonikolas et 2019 Tao et al., [221]] Wang et al.j 2021b) are proposed to efficienlly and effectively defend against poisoning attacks in various scenarios. This paper is within the scope of linear classification problems and we leave the studies in DNN models for future work.
Fair Classification. Fairness issues have recently drawn much attention from the community of machine learning.
Fairness issues for common classification problems can be generally divided into two categories: (1) Equalized treatment Zafar al. (2017) (or "Statistical Rate"); and (2) Equalized prediction quality (Hardt et al. 2016). For
classification models to satisfy these fairness criteria, popular methods including (Zafar et al. [2117 et al.
2018 Agarwal et al. 2018) solve constrained optimization problems, and (Zhang et al. 2018) apply adversarial training (Madry et al. 2017) method.
Comparison to Prior Works. There recent works that try to test the robustness of fair classification methods by manipulating their training set. They also proposed possible strategies to defend the perturbations. For example, the works (Wang et al., 2020; Lamy et al. 2019; Celis et al.] 2021a b) consider injecting naturally adversarially generated noise only on sensitive attributes. Another line of researches Roh et al. 22020;:1 [Wang et a202 considers the vulnerability of fairness training (coordinated) label-flipping attacks (Paudice al.
2018 As countermeasures to defend against their proposed perturbations, representative works such as (Roh et al. 2020) proposed an adversarial training framework (Zhang al.) (018). train the model distinguish clean samples and poisoning samples, while preserving the model faimess. The work (Wang et al., (2020) solves robust optimization problems by assigning soft sensitive attributes. our work, in terms of attack, we consider a stronger attacker because he/she can insert sophisticatedly calculated features and sensitive attributes, to fully exploit the vulnerability of fairness training methods.
7 CONCLUSION
In this work, we study the problem of poisoning attacks on fair classification problems. We propose a strong attack method that can evade the defense of most existing methods. Then, propose effective strategy greatly improve the robustness of fair classification methods. In the future, we aim to examine if our findings can be generalized other machine learning tasks, and other machine learning models, such as Deep Neural Networks (DNNs).
9

---

Under review conference paper at ICLR 2023
REFERENCES
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A reductions approach fair classification. In International Conference on Machine Learning, pp. 60 60-69. PMLR, 2018.
Alex Beutel, Jilin Chen, Zhe Zhao, and Ed H Chi. Data decisions and theoretical implications when adversarially learning fair representations. arXiv preprint 4rXiv:1707.00075 2017.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. arXiv preprint arXiv: 220666389, 2012.
Tim Brennan, William Dicterich, and Beate Ehret. Evaluating the predictive validity of the compas risk and needs assessment system. Criminal Justice and behavior, 36(1):21 2009.
Cody Burkard and Brent Lagesse. Analysis of causative attacks against svms learning from data streams. In Proceedings of the 3rd ACM on International Workshop Security And Privacy Analytics, pp. 31-36, 2017.
L Elisa Celis, Lingxiao Huang, Vijay Keswani, and Nisheeth K Vishnoi. Fair classification with noisy protected attributes: A framework with provable guarantees. In International Conference on Machine Learning, pp. 1349-1361. PMLR, 2021a.
L Elisa Celis, Anay Mehrotra, and Nisheeth K Vishnoi. Fair classification with adversarial perturbations. arXiv preprint arXiv:2106.05964 2021b.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks deep learning systems using data poisoning. arXiv preprint arXiv:/712.05526, 2017.
Mass Diakonikolas, Gautam Kamath, Daniel M Kane, Jerry Li, Ankur Moitra, and Alistair Stewart. Being robust (in high dimensions) can be practical. In International Conference on Machine Learning, pp. 999-1008.
PMLR, 2017.
Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Jacob Steinhardt, and Alistair Stewart. Sever: A robust meta-algoria meta-algorithm for stochastic optimization. In International Conference on Machine Learning, pp- 1696-1606. PMLR, 2019.
Michele Donini, Luca Oneto, Shai Ben-David, John Shawe-Taylor, and Massimiliano Pontil. Empirical risk minimization under fairness constraints. arXiv preprint 1rXiv:1802.08626, 2018.
Harrison Edwards and Amos Storkey, Censoring representations with an adversary. arXiv preprint arXiv: 15500597, 2015.
lan Goodfellow, Jean Pouget-Abadie Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144 2020.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems,
2016.
Pang Wei Koh, Jacob Steinhardt, and Percy Liang. Stronger data poisoning attacks break data sanitization defenses. Machine Learning, pp. 1-47, 2021.
Ron Kohavi et al. Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid. In Kdd, volume 96, pp. 202-207, 1996.
Jakub Konečný, Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492. 2016.
Alexandre Louis Lamy, Ziyuan Zhong, Aditya Krishna Menon, and Nakul Verma. Noise-tolerant fair classifica- tion. arXiv preprint 1901v:1901.188.2 2019.
Yiming Li, Baoyuan Wu, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: A survey, arXiv preprint orXiv:2007.08745 2020.
10

---

Under review conference paper at ICLR 2023
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems, 53:21464-21475, 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 06083, 2017.
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6): 2021.
Shike Mei and Xiaojin Zhu. Using machine teaching to identify optimal training-set attacks on machine learners. In Twenty-Niuth AAAI Conference on Artificial Intelligence, 2015.
Aditya Krishna Menon and Robert C Williamson. The cost of faimness in binary classification. In Conference on Fairness, Accountability and Transparency, pp. 107-118. PMLR, 2018.
Andrea Paudice, Luis Muñoz-GGnzzlee Andras Gyorgy, and Emil C Lupu. Detection of adversarial training examples in poisoning attacks through anomaly detection. arXiv preprint arXiv:1802.03041. 2018.
Yuji Roh, Kangwook Lee, Steven Whang, and Changho Suh. Fr-train: A mutual information based approach to fair and robust training. In International Conference on Machine Learning, pp. 8147-8157. PMLR, 2020.
Benjamin IP Rubinstein, Blaine Nelson, Ling Huang, Anthony D Joseph, Shing-hon Lau, Satish Rao, Nina Taft, and J Doug Tygar. Antidote: understanding and defending against poisoning of anomaly detectors. In Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement, pp. 1-14, 2009.
Nikolaos Sarafianos, Xiang Xu, and loannis A Kakadiaris. Adversarial representation learning for text-to-image matching. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5814-5824, 2019.
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. arXiv preprint arXiv: 100.009904 2018.
Jacob Steinhardt, Pang Wei Koh, and Percy Liang. Certified defenses for data poisoning attacks. In Proceedings of the 3 International Conference Neural Information Processing Systems, pp. 3520-355 3532, 2017.
Lue Tao, Lei Feng, Jinfeng Yi, Sheng-Jun Huang, and Songcan Chen. Better safe than sorry: Preventing delusive adversaries with adversarial training. Advances in Neural Information Processing Systems, 34, 2021.
Jialu Wang, Yang Liu, and Caleb Levy. Fair classification with group-dependent label noise. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 526-536, 2021a.
Serena Wang, Wenshuo Guo, Harikrishna Narasimhan, Andrew Cotter, Maya Gupta, and Michael I Jordan. Robust optimization for fairness with noisy protected groups. arXiv preprint arXiv:2002.09343 2020.
Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai- Wei Chang, and Vicente Ordonez. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5310-5319, 2019.
Yunjuan Wang, Poorya Mianjy, and Raman Arora. Robust learning for data poisoning attacks. In International Conference on Machine Learning, pp. 10859-10869. PMLR, 2021b.
Susan C Weller and A Kimball Romney. Systematic data collection, volume 10. Sage publications, 1988.
Rand R Wilcox. Introduction to robust estimation and hypothesis testing. Academic press, 2011.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. Fairness constraints:
Mechanisms for fair classification. In Artificial Intelligence and Statistics, pp. 962-970. PMLR, 2017.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International conference on machine learning, pp. 325-333. PMLR, 2013.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. pp. 335-340, 2018
11

---

Under review conference paper at ICLR 2023
A APPENDIX
A.1 PROOF OF THEOREM
In this part, we provide the detailed proof of Theorem 4.1 in Section
Theorem A.1 (Recall Theorem 1. Suppose a set of "clean" " samples Syood with size n are i.idd sampled distribution Nu, E), where Σ I. There is a set of "bad" sumples Soad with size up = n/ K, K >> and center 1| 14122 d, d =. Then, the average squared poisoning scores of clean samples and bad samples
have the relationship:
BieSbad [x:) ExeS [x:)) 2
K KK-1 K [+1 (K -
(K -
-KK+))
Proof. We denote ggood the set of clean samples, Shod is the of bad samples, and the union of clean samples and bad samples form the whole set S. In the later part, to distinguish between the centers of each set, we use Mg Up and Hg to denote the center of clean samples, bad samples and the whole set.
First, it is easy to know that Ma. Up and μ4a are in a same line. Thus, given fgg T2p K 1, we have the relationship of center distance: d || - upll (1 +1++ K)ll - u.ll ((K 1)/ ())K) Ms ||. In the following, we will study the squared poisoning score in the whole group S. Given any unit vector V is the top right singular vector of the centered data matrix of S:
E.es [(V (x - )s))²] Eies [(V (x - Mp) + V. (up - ps))²
= [(V (Ir p=)) FAes 22(V . (x (p)(Mp - -,) v²] Ries [0V (V-(1pp 1s))
= E.es [(V -x- mb)) (V (1p-1u)))
Note that is the top right singular vector of the centered data matrix X,- Mu. we choose v' = (Mp-Ms)/11c0 ps)ll. which is the unit vector that has the same direction with (up - Hs). We get:
Eies [(V (xr - )=)) 2 Bies [(v' (x - -m)))² - |1 uall
For the first term in the right hand side of the inequality above:
(K + 1)- Bies
[[v' (xr - --))) 2 K Eied [(t' (x pp))
because the poisoning scores are all positive. Then, we have:
(K + Bies [(t' (x (p))² K BiieS [(v' (x Up))²]]
K EreSyood [(v' (x My)) e' (ug -pp))²]]
K (X, 1b) no) n.th + (wo No)( 1p)) -
2 K (0+ 1|Ha- - 1²)
The first term is larger than 0 because of the semi-definite property of the matrix (X, 1g (g) p² the second term is because o' has the same direction with (Ap (s)) (because Mp, Hg and Ms are in the same line). Therefore, we get the average poisoning score of the whole set S:
K
K d
Bies [(V (x n.))
ž K +
1lMy
1p - 11mp uall
(1+ KY²
In the following, we will calculate the average squared poisoning score in the good set Spood.
Exesyond [(V- (x pp))) = V (X, /4 (X, /²y)²
va + V- (Mg n=)(X, 1T)²² va
< 0 + ||, 11 < 0² K +1 d)²
Based on previous calculation about the average score of whole set and good set, we can get the average squared poisoning score in the bad set Spodt
Bxe [(V- (x (b))) = (1 + K)E [(V (e - -,))
K KEEss good [(V (x [p)))
k²
d² Ko²
(K 1)²
12

---

Under review as conference paper at ICLR 2023
Table 4: RFC Baseline Methods' Performance on Adult Census Dataset, for Equalized Treatment
No Attack
Label
Flip(10%)
Label 2lip((0%)
Attr.
Flpp(10%)
Attr. 200)(
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair
No Defense.
0.795
0.808
0.787
0.823
0.784
0.834
0.797
0,800
0.802
0.800
SEVER.
0.772
0.829
0.754
0,823
0.757
0.801
0.785
0.795
0.781
0.790
RFC.
0.781
0.803
0.791
0.820
0.799
0.830
0.788
0.806
0.783
0.823
(E = 10%)
Min-Max(z = 0)
Min-Max(2 = 1)
F-Attack(z = 0)
F-Attack(2 1)
F-Autack*
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
No Defense.
0.779
0.834
0.787
0,810
0.781
0.795
0.785
0.823
0.790
0.822
SEVER.
0.766
0.803
0.757
0,800
0.770
0.788
0.764
0.788
0.769
0.803
RFC.
0.778
0.834
0.788
0.804
0.781
0.812
0.794
0.799
0.793
0.803
[e = 15%)
Min-Max(Z = 0)
Max-Max(Zz:= = I)
F-Attack(2 = 0)
F-Attack(z = 1)
F-Attack*
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair:
Acc.
Fair.
No Defense.
0.782
0.797
0.783
0.792
0.766
0.787
0.777
0.822
0.770
0.801
SEVER.
0.775
0.769
0.766
0.792
0.741
0,803
0.755
0.773
0.766
0.770
RFC.
0.786
0.806
0.794
0.812
0.776
0,815
0.790
0.822
0.786
0.823
and the difference between two averaged scores:
BieSsod
BigS good 2 k
- (ls + 1)o² =
(K-1
K + 772
- (K - -(K1))) -2)(
k+1
A.2 MORE EXPERIMENTAL DETAILS
In this part, we provide additional experimental details such as the pre-process procedure.
Adult Cenesus Dataset. In this dataset, we have 5 numerical features "age, education-num, capital-loss and capital gain"", and we also use categorical features such "workclass, education, marital-stataus, occupation, relationship, race". For simplicity, we first transform the categoircal features into dummy variables and conduct Principle Component Analysis project them to the space, which is spanned by the first 15 principle components. After projection, we normalize all 20 features by centering and standadizing. Under this dataseet, during the attacking of Min-Max and F-Attack, we assign the radius of the feasible injection set to be d = 9.0. Appendix A.3]
we provide the empirical results for more choices of d, i.e., d = 6.0.
COMPAS Dataset. this dataset, we have the numerical features: "age, age_cat, juy huw_fel_cooee fel__cuut,,,,ttteeeooooootr juv misd_coum www.subsseeeennn...e
juv oother_count, priors_coume count,
decile_score, c_jail oojai We use "c_jail.out c_jail_in" in" to get the number of days in jail and exclude c. c_jail_out, c.jail_in. in. We also have categorical features Scharge-degree and sex, and we use PCA to find the first two principle directions. Then, standardize each feature. Under this dataset, during the attacking of Min-Max and F-Attack, we assign the radius of the feasible injection set to be d = 9.0. In Appendix A.3]
provide the empirical results for more choices of d, i.c., d = 6.0.
A.3 MORE EXPERIMENTAL RESULTS
In this part, we provide additional empirical results to validate the effectiveness of our attack and defense method. In detail, we consider more settings about: (1) Different Type of Fairness Criteria, such as "Equalized Treatment"; (2) Different fair classification method, such as (Zalar et al., 2017), (3) Different choice of the radius of feasible injection space d. such as d 6.0. Notably, in the all experiments in the main paper, we set d with fixed value d = 9.0. In this part, we provide another option when we choose a smaller d = 6.0. It is because larger d, i.e d > 9.0 will make the poisoning samples easier to be detected by most defense methods. For fairness criteria under "Equalized Treatment" (Equalized Positive Rate (PR)) in Adult Census Dataset, we set the desired fairness criteria to be PR(2 = 0)- PR(: = 1)] < 0.2. For fairness criteria under "Equalized TPR" in Adult Census Dataset, set the unfairness criteria to be (TPR(> 0) TPR(2 1)] 0.05. For faimess criteria under "Equalized TPR" and "Equalized Treatment" in COMPAS Dataset, we set the desired fairness criteria to
be max (PR(2 2 = j) TPR < 0.15 and maxjez |PR(2 = j) PR| < 0.15.
13

---

Under review conference paper at ICLR 2023
Table 5: RFC Baseline Methods' Performance on Adult Census Dataset using
Zafar et al.
2017
No Attack
Label 11ip(10%)
Label 2lip((0%)
Attr.
11p((0%)
Attr.
(l0)(20%)
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Fair.
Acc.
Fair.
No Defense.
0.815
0.944
0.813
0,962
0.804
0.954
0.814
0.942
0.811
0.931
SEVER.
0.812
0.952
0.806
0,954
0.808
0.957
0.814
0,942
0.811
0.953
RFC.
0.814
0.964
0.808
0.956
0.801
0.951
0.802
0,963
0.802
0.944
(E = 10%)
Min-Max(z = 0)
Min-Max(z = 1)
F-Attack(2 (z = 0)
F-Attack(2 1)
F-Autack*
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
No Defense.
0.801
0.969
0.797
0.864
0.769
0.966
0.796
0.804
0.799
0.799
SEVER.
0,800
0,960
0.802
0.954
0.788
0.945
0.786
0,937
0.779
0.956
RFC.
0.802
0.967
0.811
0.946
0.803
0.950
0.808
0.952
0.809
0.951
(e = 15%)
Min-Max(Z = 0)
Min-MMa(ZZ== I)
F-Attack(2 = 0)
F-Attack(z = 1)
F-Attack*
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair:
Acc.
Fair.
No Defense.
0.784
0.953
0.797
0.841
0.700
0.943
0.755
0.821
0.798
0.800
SEVER.
0.802
0.959
0.793
0.978
0.775
0,952
0.778
0,909
0.775
0.927
RFC.
0.799
0.944
0.799
0.952
0.796
0,950
0.801
0.951
0.803
0,946
Table 6: RFC & Baseline Methods' Performance Adult Census Dataset when d = 6
(c 10%)
Min-Maxie = 0)
Min-Max(z = 1)
F-Attack(z 0)
F-Attack(Z 1)
F-Attack*
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
No Defense.
0.796
0.970
0.803
0,955
0.762
0.977
0.803
0.778
0.811
0.723
SEVER.
0.762
0.955
0.786
0,956
0.744
0.902
0.743
0.944
0.762
0.735
RFC.
0,800
0.969
0.805
0.966
0.812
0.945
0.785
0.963
0.783
0.961
(6= 15%)
Mix-Max(z = 0)
Min-Max(z = 1)
F-Attack = 0)
F-Attack(z = 1)
F-Attack*
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair:
No Delense.
0.802
0.945
0.800
0.945
0.759
0.946
0,800
0.672
0.789
0.721
SEVER.
0.758
0.960
0.743
0.980
0.724
0.961
0.726
0.855
0.724
0.756
RFC.
0.805
0.955
0.801
0.967
0.775
0.953
0,801
0.955
0.774
0.945
Table 7: RFC& & Baseline Methods' Performance on COMPAS dataset for Equalized Treatment
No Attack
Label Flip( (00%)
Label Flip(20%)
Attr.
Flip(10%)
Attr. Flip(20%)
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair
No Defense.
0.671
0.845
0.663
0.854
0.655
0.845
0.659
0.873
0.670
0.845
SEVER.
0.665
0.831
0.665
0.849
0.667
0.836
0.674
0.842
0.672
0.834
RFC.
0.671
0.830
0.669
0.845
0.672
0.846
0.677
0.850
0.672
0.844
[E 10%)
Min-Max(z = 0)
Min-Max(2 = 1)
F-Attack(z = 0)
F-Attack(z 1)
F-Autack*
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
No Defense.
0.680
0.843
0.662
0,845
0.666
0.862
0.658
0,854
0.677
0.843
SEVER.
0.662
0.836
0.659
0,839
0.646
0.850
0.648
0,825
0.671
0.842
RFC.
0.677
0.838
0.676
0.841
0.675
0.852
0.674
0,848
0.668
0.845
(E = 15%)
Min-Max(z = 0)
Min-Max(2 = 1)
F-Attack(z = 0)
F-Attack(z 1)
F-Autack*
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
No Defense.
0.629
0.866
0.668
0.845
0.669
0.820
0.662
0,855
0.675
0.855
SEVER.
0.633
0.848
0.631
0.855
0.613
0.850
0.613
0,866
0.678
0.845
RFC.
0.659
0.845
0.674
0.846
0.675
0.835
0.671
0.846
0.680
0.845
A.4 GENERALIZING F-ATTACK TO ONN-BASED FAIR LEARNING
In the paper, we have mainly discussed linear models and convex optimization problems. In this part, we argue that method and conclusions are general and representative, which can easily extend to Deep Neural Networks (DNNs). To show this point, we focus on a representative and popular fair learning method in DNN, called Learning Fair Representations (LFR) (Zemel et al.] 2013 Edwards & Storkey 2015 Beutel al. 2017 Wang et al. 2019). Then, we propose a modified version of F-Attack, called F-Attack DNN (FAD), which inherits the
14

---

Under review as conference paper at ICLR 2023
Table 8: RFC Baseline Methods' Performance on COMPAS Dataset using
Zafar et al. 2017
No Attack
Label 10ip(10%)
Label 2lip((0%)
Attr.
Fli((1%%)
Attr. Flip(20%)
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair
No Defense.
0.651
0.844
0.642
0.865
0.644
0.827
0.651
0.855
0.662
0.840
SEVER.
0.647
0.846
0.6770
0.841
0.657
0.830
0.644
0.851
0.642
0.830
RFC.
0,656
0.852
0.668
0.841
0.679
0.850
0.677
0.846
0.661
0.852
(E = 10%)
Min-Max(z = 0)
Min-Max(2 = 1)
F-Attack(z = 0)
F-Attack(2 1)
F-Autac+
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair
No Defense.
0.655
0.840
0.645
0.842
0.646
0.832
0.628
0.851
0.654
0.816
SEVER.
0.645
0.860
0.640
0.861
0.628
0.840
0.619
0,852
0.659
0.831
RFC.
0.655
0.858
0.662
0.854
0.663
0.850
0.660
0.851
0.659
0.852
(e = 15%)
Min-Max(Z = 0)
Max-Max(Z.== = 1)
F-Attack(z = 0)
F-Attack(z = 1)
F-Attack*
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair:
Acc.
Fair
No Defense.
0,659
0.820
0.631
0.851
0.649
0.800
0.621
0.851
0.658
0.815
SEVER.
0.641
0.866
0.638
0.851
0.596
0,865
0.611
0,870
0.643
0.831
RFC.
0.671
0.861
0.659
0.855
0.650
0,851
0.640
0.857
0.659
0.850
Table 9: RFC& & Baseline Methods' Performance COMPAS dataset, when d = -
(c 10%)
Min Max(z 0)
Min-Max(z = 1)
F-Attack(z 0)
F-Attack(z = 1)
F-Attack*
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair:
No Defense.
0.650
0.865
0.581
0.838
0,648
0.800
0.651
0,820
0.631
0.815
SEVER.
0.668
0.853
0.643
0.788
0.652
0,800
0.656
0.802
0.644
0.823
RFC.
0.670
0.850
0.651
0.835
0.658
0.820
0.663
0.822
0.665
0.835
(e ===10 15%)
Min-Max(2 = 0)
Min-Max(z = 1)
F-Attack = 0)
F-Attack(z = 1)
F-Attack*
Acc.
Fair.
Ace.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fuir.
No Defense.
0.621
0.854
0.617
0.859
0.515
0.980
0.614
0.890
0.640
0.801
SEVER.
0.625
0.819
0.641
0.801
0.624
0.770
0.645
0.853
0.635
0.815
RFC.
0.660
0.833
0.675
0.857
0.672
0.832
0.664
0.832
0.660
0.847
idea of F-Attack but accommodate to attack LFR. Finally, we show the experimental results to demonstrate the success of our proposed attack method.
Learning Fair Representation (LFR). The goal of learning fair rep- resentations is remove sensitive information from the learned repre- sentations, so that the sensitive information will not interfere the model prediction. In this paper, we focus on the method (Edwards& & Storkey 2015) to achieve the goal of LFR. It is an adversarial framework that learns the target classification task without the ability to predict sensi- tive information. Many fairness learning methods in DNN tasks follow the similar strategy to learn fair representations for images (Wang] Figure 3: Learning Fair Representation
Extracted Feature
((h(x))
Label y
Inputs
h(x)
[(h(x))
Sensitive Attribut
et al.) 2019) and texts (Sarafianos et al. 2019)) Figure 3 illustrates the overview of the framework. In detail, feature extractor h(-) first extracts latent information form the input. Then, two prediction heads f.).) and f-) are attached to the extracted feature space to predict the label and sensitive attribute respectively. To ensure there is no sensitive information in the learned representation ht.), the training objective of LFR can be expressed as:
min max E (((((t)))) - a ((([(((((()))))
(10)
f J.
Here, the function f(-) = ()) is the overall classification model. This objective briefly resembles the idea of GAN (Goodfellow et al. 2020): In the inner maximization, f=(-) is updated to minimize ((f=(h)))), which aims to explore helptul information from h(()) to predict the sensitive attribute 2; In the outer minimization, the model f() and feature extractor /(-) are updated break fa(''s ability to predict z, while preserve the ability
to predict y. In this way, the model is trained to eliminate sensitive information from the extracted features.
15

---

Under review as conference paper at ICLR 2023
F-Attack-DNN (FAD). Based on the design of LFR, propose F-Attack-DNN to generate poisoning samples to degrade the performance of LFR. It shares the similar overall optimization objective as in Eq in Section which solves min-max problem:
min max SCH DrCP
L(Dc Dp, w)
(11)
To solve this problem for DNN-based model f. we propose the following algorithm Algo. 3 called F-Attack- DNN (FAD). Specifically, in the inner maximization, FAD first finds the sample which maximize loss value under current model f. which similar (4) in Section 3 Notably, to solve this optimization problem for
ENN-based model, we can apply well-known methods such as Projected Gradient Descent (Madry et al.] (017).
Then, Step (2) and Step (3) imitate the (re)-nraining process of LFR to update the model f. In detail, Step (2) updates the classifier fa to predict sensitive attribute from the learned feature h(r);; Step (3) solves the outer minimization for Eq. This will ensure f fall into the space Hffair and f has a minimized training error.
Algorithm 3: Algorithm of F-Attack-DNN (FAD)
Input :Clean data Dc, number of poisoned samples en,
feasible set F, a > 0, 17 > 0, warm-up steps Tiuurn
Output :A poisoning set D2-
for = 1...., 7lburn + on do
1. Solve inner maximization: (z, arg max (2., (f(x,, w), v) v) and let Dp = on ((x, y, 2)
2. Update the discriminator: f: = f= 1) Mouu L(.,.))DDUDD V (II=(h(=)))))
3. Solve outer minimization: f= - Pep El.g, V (((f(z),)) a- (ff(h(()),
if > Tbbrn then Dp = Dp U {(x,1,2)
end
end
Notably, the only difference between F-Attack (Algo. and F-Attack-DND (Algo. 3 is about how the outer minimization problem is solved, which is how the fair classification model f updated. In another word, for various fair learning schemes, we can adaptively modify the objective of F-Attack to accommodate the targeted fair learning methods. Next, we will show experimental results to demonstrate the effectiveness of FAD.
Experimental Results. To demonstrate the effectiveness of our proposed attack, we again focus on the Adult Census Dataset and consider the fairness criteria as Equalized True Positive Rate (TPR) between genders. Under this dataset, consider two layer Multi-Layer Perceptron (MLP) for feature extraction and classification. Then, the sensitive attribute discriminator f-()) is attached onto the pen-ultimate layer of MLP. In our experiments, we insert various fraction (from 0% 40%) of poisoning samples the training set, and define the feasible injection set F to be {(r,, z) ||x [|xx-/a Mbyll I d}. where d = 9. In the Table 10 we report the models' goodness of accuracy and fairness, for the models which are obtained via LFR without any defenses. Notably, we choose to report the average model performance (for 10 runs) on the checkpoint when it achieves best validation performance (see Section Eq D. From the result, we find that FAD can achieve much stronger attacking effectiveness than other baseline attacks, such as label flip, attribute flip and Min-Max Attack (MM) (Steinhardt et al.) 2017). Here, MM represents the attack algorithm similar to Algo 3 but without step and the term about fe in Step 3. This experimental result implies the generality of F-Attack and its potential to extend to various fairness training schemes. We will leave the study on fair learning for other DNN applications, such as image domain and text domain for future investigation.
A.5 THE PRECISION OF THE DETECTION OF RFC
To have deeper understanding of the defense method RFC, we check the caleulated poisoning scores for all training samples in a poisoned Adult Census Dataset. In Figure we visualize the distribution of poisoning scores for clean samples, well poisoning samples from 4 versions of F-Attack. Specifically, we report the scores for each subgroup separately and we use red dots to denote the poisoning samples. From this figure, it is easy to see that the poisoning samples always appear at the tail in their labeled subgroup with (almost) lowest poisoning scores. Note that during our proposed defense method RFC, at each iteration, it only removes a small
16

---

Under review conference paper at ICLR 2023
Table 10: F-Artack- -NN (FAD) vs. Baseline Attacks on Adult Census Dataset
Label.
Flip
Attr.
Flip
MM
=
MM
1)
FAD
FAD (Z
1)
Ratio.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
Acc.
Fair.
0%
0.797
0.960
0.799
0.966
0.797
0.960
0.799
0.960
0.797
0.960
0.797
0..66
10%
0.781
0.972
0.795
0.960
0.774
0.970
0.778
0.960
0.757
0.936
0.775
0.954
20%
0.783
0.972
0.798
0.958
0.776
0.960
0.740
0.963
0.731
0.945
0.713
0.952
30%
0.766
0.967
0.795
0.958
0.724
0.977
0.692
0.967
0.676
0.947
0.644
0.965
40%
0.741
0.971
0.797
0.936
0.701
0.971
0.668
0.987
0.655
0.924
0.647
0.961
Clean
Clean
Poisening
Poisoning
palconing Scare. Attaack [2 0)
00
06-
F-Attack
00
-62
-644
polsoning
7-42--1
Y-12-0
1-12-3
1-412-9
V-12-0
Y-1--1
(a) F-Attack (Z 0 6)
(b) F-Autack (Z 1,d=6)
Clean Poisening
Glenn Paisoning
58
paisoning score F Attack IZ
06
F-Attack
0.4
04
02-
02
0.-
00-
-02
-04
-16
Y=-12-0
Y-41
2-1
Y-12-0
V-12-
V=1.2-0
7-1222
(c) F-Attcck (Z 0.
(d) F-Attack (7 1, =d
Figure Puisoning Score calculated by RFC for samples from each subgroup.
portion of samples from particular subgroup with highest or lowest poisoning score. As result, RFC will not remove too many clean samples until they can find the poisoning samples. Besides, one interesting fact is: under this Adult Census Dataset, F-Attack prefers to add poisoning samples with y 1.
17


Reviews:
[{"review_id": "qR-oqjV7ye", "reviewer": "Reviewer_Bdkz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors first describe an attack to poison fair classifiers, and then introduce a defense (filtering based) to mitigate poisoning attacks on fair classifiers.", "summary_of_the_review": "While I think the paper is well presented, and thorough, the combination of the attack scenario being somewhat niche, and the attack constraints being rudimentary, I don't think the work is of sufficient interest at this point to warrant publication at a top conference. \n\nI am, however, willing to change my score in response to new information/reasoning from the authors.", "strength_and_weaknesses": "## Strengths:\n1. The paper is well rounded and thorough - the authors describe the setting and the problem well, introduce an attack, and then describe a defense to this attack.\n2. The results, at least in the setting described in the paper (linear classifier, adult census dataset) improve upon existing techniques. \n\n## Weaknesses:\n1. Significance - I'm not convinced this work is significant enough to warrant publication at a top tier conference. While fair classification is of great interest, and data poisoning attacks at well, a defense against data poisoning for fair classification networks seems a bit too niche/lacking general interest.\n\n2. The attack seems like a straightforward extension of the technique in [1] with a straightforward addition of penalty to encourage fairness in the inner optimization problem for poisoning. Granted, the proposed defense does seem novel.\n\n3. The setting for this paper seems weak. The poisoning attacks allow for techniques like label flips, and it seems like a simple linear network is used for the victim model (avoiding any questions of white-box vs. black-box attacks). This is in contrast to recent, more sophisticated poisoning attacks that are in \"clean-label\" setting, and/or against deep networks.\n\n\n### Random tidbits:\n- I believe the equation in between eqns 2, 3 should have an $\\leq$, and not $=$ in the last part. \n\n\n[1] Muñoz-González, Luis, et al. \"Towards poisoning of deep learning algorithms with back-gradient optimization.\" Proceedings of the 10th ACM workshop on artificial intelligence and security. 2017.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "pLDTBogrl8", "reviewer": "Reviewer_Qsnj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied data poisoning attacks against fair classification algorithms. The objective of poisoning a fair learner is approximated by a minimax optimization problem. When the loss function is convex, one can switch the min-max operator and use gradient descent to construct the optimal poisoning dataset. On the basic fair learner and a few robust variations such as k-NN based defense and Sever-based defense algorithms, the authors demonstrate that their poisoning algorithm can reduce the accuracy and the fairness metric of the victim learners. Motivated by that, the paper designed a defense mechanism called RFC, which employs similar ideas as prior works, but adapted to the fair classification setting. The paper analyzed the relationship between the poisoning scores of clean and poisoned samples. Extensive experiments on abundant datasets show that the proposed defense can enhance the resilience of fair learner against poisoning attacks.", "summary_of_the_review": "I worked in related areas.", "strength_and_weaknesses": "Strength:\n\n(1). The paper performed extensive experiments and detailed analysis on the experimental results, which clearly conveyed two critical points: i) fair classification learners, although equipped with existing defense strategies such as k-NN and Sever, are not robust enough against data poisoning attacks. ii) the RFC defense mechanism proposed in this paper indeed helps mitigate the effect of data poisoning attacks on fair learners and outperforms existing defense methods.\n\n(2). The paper is easy to read. I can quickly understand the topic and the key research points after reading the problem statements. The presentation and paper structure is also nice and clear.\n\nWeaknesses:\n\n(1). The paper is technically weak from a scientific point of view. While the paper performed extensive experiments and also designed interesting attack and defense algorithms, I find it hard to justify that their approaches contain very novel techniques compared to prior works. Their attack algorithm is pretty straightforward and looks like adapted from existing research outcomes. The minimax formulation also applies to traditional classification victim learners. The only difference here is that in this paper, the min operator is restricted to models that satisfy the fairness constraint. Therefore, I don't see why the attack algorithm is technically novel. Similarly, as the authors pointed out, the defense here also borrows some ideas from prior works. Overall, I am a bit concerned about not having enough novelty and technical contribution for this paper.\n\n(2). The theoretical results do not appear appealing to me because it does not state directly to the point that how much improvement in the accuracy or fairness is guaranteed by the defense. Right now, the results are somewhat weak because it's about the poisoning score. In contrast, the more interesting theoretical problem is how much accuracy is preserved when using the RFC defense compared to without using RFC. I am wondering if the theoretical part of this paper can be improved in that way.\n\n(3). In the objective approximation equation (i.e., the equation between (2) and (3)), the last equality does not make sense to me. Instead of an equality, I believe that should be an inequality. I am wondering if that's a typo or did I miss anything?\n\n(4). The scope of this paper is limited to classification only, Furthermore, it's restricted to convex loss functions. As a result, the results in this paper are not directly applicable to DNN-based classification learners. I believe the paper can easily generalize their results, e.g., by designing different poisoning algorithms for DNN-based learners. The minimax formulation no longer works but from a practical perspective, I don't really think the minimax formulation is super critical to this paper. There are a variety of poisoning attacks against DNN, and these attacks can be easily adapted to the fair classification setting. Please correct me if I am wrong.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "465THrIMmQ", "reviewer": "Reviewer_MXgE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies data poisoning attacks against fair classification algorithms, and proposes a new poisoning attack, which is a variant of the existing min-max attack. In addition, a robust fair classification defense is proposed, which is an extension of the Sever defense into fair classification problems. Empirical results show that the proposed attack is more effective than the existing min-max attack against different baseline defenses and the proposed defense is also robust against the newly proposed attack.  ", "summary_of_the_review": "The idea of studying fair classification under poisoning attacks is interesting. However, the core techniques in this paper are mostly based on existing poisoning attacks and defenses for traditional classification and hence, have limited technical novelty. In addition, the proposed attacks also have limited effectiveness empirically, which further weakens the results. Considering all these factors, a weak reject is recommended. ", "strength_and_weaknesses": "Strength:\n1. vulnerability of fair classification under data poisoning is interesting. \n2. the proposed attack and defense performs better than the existing attacks and defenses.\n\nWeakness:\n1. the major concern is in the significance of this work: the F-attack is a variant of the existing min-max attack, where the major difference is how the model weight is updated after the poisoning point is selected. The proposed defense is also a variant of the Sever defense, with finer analysis of the training data at particular subgroups. Therefore, the technical novelty of the paper is limited.\n2. the considered attacks are quite weak and may not even be a real threat. The fraction of poisoning points is >10%, which is considered quite a high ratio in practical applications. However, even with such a high poisoning ratio, the attack effectiveness is limited. Taking the poisoned model's test accuracy as an example, we expect a strong poisoning attack to induce much higher test error in comparison to the poisoning ratio $\\epsilon$, while results in this paper induce test error smaller than $\\epsilon$. Therefore, one may conclude that current fair classifications are not under severe threat from poisoning attacks. \n3. As for the evaluation, one relevant baseline is missing (Jo et al., 2022). The min-max attack may also be improved by leveraging target models, which is done in Koh et al., (2021) and also demonstrated similarly in Suya et al., (2021).  \n\nJo et al., \"Breaking Fair Binary Classification with Optimal Flipping Attacks\", ISIT 2022. \nSuya et al., \"Model-Targeted Poisoning Attacks with Provable Convergence\", ICML 2021. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject