Towards Poisoning Fair Representations
Authors: Tianci Liu
Haoyu Wang
Feijie Wu
Hengtong Zhang
Pan Li
Lu Su
Jing Gao
Venue: ICLR 2024 poster (2024)
Keywords: data poisoning attack
fair representation learning
fairness
Primary area: societal considerations including fairness, safety, privacy

Abstract: Fair machine learning seeks to mitigate model prediction bias against certain demographic subgroups such as elder and female. 
Recently, fair representation learning (FRL) trained by deep neural networks has demonstrated superior performance, whereby representations containing no demographic information are inferred from the data and then used as the input to classification or other downstream tasks. 
Despite the development of FRL methods, their vulnerability under data poisoning attack, a popular protocol to benchmark model robustness under adversarial scenarios, is under-explored. Data poisoning attacks have been developed for classical fair machine learning methods which incorporate fairness constraints into shallow-model classifiers.
Nonetheless, these attacks fall short in FRL due to notably different fairness goals and model architectures. 
This work proposes the first data poisoning framework attacking FRL. We induce the model to output unfair representations that contain as much demographic information as possible by injecting carefully crafted poisoning samples into the training data.
This attack entails a prohibitive bilevel optimization, wherefore an effective approximated solution is proposed. A theoretical analysis on the needed number of poisoning samples is derived and sheds light on defending against the attack. Experiments on benchmark fairness datasets and state-of-the-art fair representation learning models demonstrate the superiority of our attack.

Published as a conference paper at ICLR 2024
TOWARDS POISONING FAIR REPRESENTATIONS
Tianci Liu', Haoyu Wang', Feijie Wu', ,Hengtong Zhang', Pan Li", Lu Su', Jing Gao !Purdue University "Tencent AI Lab *Georgia Institute of Technology 1(1iu3351 wang5346, wul 977, lusu,, pur edu Phazhang. work@gmail com ppanli@gatech.. edu
ABSTRACT
Fair machine learning seeks mitigate model prediction bias against certain de- mographic subgroups such as elder and female. Recently, fair representation learning (FRL) trained by deep neural networks has demonstrated superior per- formance, whereby representations containing inferred from the data and then used as the input to classification or other down- stream tasks. Despite the development of FRL methods, their vulnerability under data poisoning attack, a popular protocol to benchmark model robustness under adversarial scenarios, is under-explored. Data poisoning attacks have been de- veloped for classical fair machine learning methods which incorporate fairesss constraints into shallow-model classifiers. Nonetheless, these attacks fall short in FRL due to notably different fairness goals and model architectures. This work proposes the first data poisoning framework attacking FRL. We induce the model to output unfair representations that contain much demographic information as possible by injecting carefully crafted poisoning samples into the training data. This attack entails prohibitive bilevel optimization, wherefore an effective ap- proximated solution is proposed. A theoretical analysis on the needed number of poisoning samples is derived and sheds light on defending against the attack. Ex- periments on benchmark fairness datasets and state-of of-the-art fair representation learning models demonstrate the superiority of our attack.
demographic information are
INTRODUCTION
Machine learning algorithms have been thriving in high-stake applications such credit risk anal- ysis. Despite their impressive performance, many algorithms suffered from the so-called fairness issue, i.e., they were shown biased against under-represented demographic subgroups such fe- males in decision making (Barocas & Selbst, 2016; Buolamwini & Gebru, 2018). As remedies, fair machine learning has accumulated a vast literature that proposes various fairness notions and attempts to achieve them (Pedreshi et al., 2008; Dwork et al., 2011; Hardt et al., 2016). For shallow models such as logistic regression and support vector machine, fairness notions are often defined upon scalar model predictions and have been well-studied, see Kamishima et al. (2011), I): Zafar et al. (2017) and reference therein for instances. We refer to this family of work as classical fair machine learning. Recently, fair representation learning (FRL.) with deep neural networks (DNNs) has at- tracted great attention (Xie al., 2017; Madras et al., 2018; Creager et al., 2019; Sarhan et al., 2020)- FRL learns high- dinensiinnsii representations for downstream tasks that contain minimal information of sensitive features (i.c., the memberships of demographic subgroups). These information-batt fairness notions equip FRL with higher transferability than classical methods (Creager al., 2019).
Despite the success of fair machine learning methods, not much is known about their vulnerabil- ity under data poisoning attacks until very recent studies (Chang al., 2020; Solans et al., 2021; Mehrabi et al., 2021). Data poisoning attacks aim to maliciously control a model's behaviour to achieve some attack goal by injecting poisoning samples to its training data and are widely used to benchmark the robustness of machine learning models in adversarial scenarios (Bard & Falk, 1982; Biggio et al.. 2012). Recently researchers successfully attacked classical fair machine learning meth- ods such as fair logistic regression (Mchrabi et al., 2021) and exacerbated bias in model predictions, thereby hurting fairness. But it is still an open question whether FRL suffers from a similar threat'.
See App for more detailed discussion on attacking fair representations versus downstream classifiers.

---

Published as a conference paper at ICLR 2024
Notwithstanding, devising poisoning attack to degrade fairness of representations proves to be a non-trivial task. The difficulty is from two aspects. First, high-dimensional representations are more complicated evaluate fairesss than scalar predictions in classical fair machine learning. This makes existing attack goals for fairness degradation against the latter fall short to apply. Sec- ondly, fair representations implicitly depend on the training data, and non-convex DNNs make this dependency hard to control by previous optimization- or heuristic-based attacks on classical fair ma- chine learning. Optimization-base
attacks (Solans et al., 2021) need the victim model to be simple enough (e.g., convex) to analyze. Heuristics such as label flipping (Mehrabi et al., 2021) do not directly optimize the attack goal, thereby often requiring great effort to design good manipulations to success.
We propose the first data poisoning attack that directly targets on fairness of high-dimensional representations as shown in Figure 1. Our at- tack is optimization-based with new attack goal. Specifically, following the common prin- ciple behind FRL that ideal fair representations should contain no information of sensitive fea- tures (Moyer al., 2018; Zhao et al., 2019; Creager et al., 2019), we devise our attack goal to maximize the mutual information (MI) be- tween the Iearned representations and sensitive features. The attack goal entails a bilevel opti- mization, whose challenges lie in two-folds
Training data
Roosoning
Target dain
x
Victim
Attaok Goal: MI >
Figure 1: Framework of data poisoning attack (in red) on FRL. Irrelevant components such as class labels are omitted. The attacker poisons train- ing data to contaminate the victim training (solid lines), resulting in unfair representations for tar- get data (dotted lines) such that its MI to sensitive feature a is maximized. The MI is supposed to be small before the attack.
Firstly, MI does not admit analytic form, mak- ing its maximization challenging. Instead. a variational lower bound of MI (Barber & Agakov, 2003) inspires to use the negative loss of the optimal classifier that predicts sensitive features from representations as a proxy. To avoid the complcxity of training such a classifier in the inner loop of the optimization, we further connect the classification loss with principled notion of data separability. FLD scores give us an analytic measure of such and simplify the optimization substantially. Notably, FLD score as valid data separability measure does not rely on the mixture- of of-Gaussian assumption (Fisher, 1936), and our empirical results also show that this assumption is not essential for our purpose in practice. To our best knowledge, we propose the first attack goal based on MI explicitly towards fair machine learning that is highly principled and flexible. We also analyze its connection to demographic parity, one of the most popular fairness notions. This is one of the most significant contributions of our work.
Secondly, representations are learned by DNNs, whose dependency on poisoning samples is impos- sible to track exactly. Consequently, one cannot identify training on what poisoning samples the vic- tim will produce the most by matching the upper- and lower-level gradients following Geiping et al. (2020): The gradient of the victim from FLD score gives the desired update direction; when it matches the gradients of the victim on poisoning samples, training the victim on poisoning samples will solve the attack goal approximately. To improve stcalthiness of the attack on fairness degradation in our scenarios, we design better constraints how much poisoning samples can deviate from clean samples. In spe- cific, we define a more proper valid range based on the data type, and an elastic-net penalty (Chen et al., 2018) to encourage such deviation to be sparse. Meanwhile, based attacks (Geiping et al.,
representations. We solve this problem approximately
derive the first theoretical
minimal number of poisoning samples required by gradient-malchin 2020) succeed under regular conditions. This bound is crucial for practical altacks, as using more poisoning samples will increase the chance of being detected (Geiping et al., 2020; Koh et al., 2022). This theoretical analysis is another contribution of this paper and can be of independent interest to users of all
based attacks.
Extensive experimental results in Section demonstrate high effectiveness of our attacks on four representative FRL methods using as few as 5% of training data for poisoning. In the remaining part of this paper, we review related works in Section 5, and conclude the work in Section 6.
2

---

Published as a conference paper at ICLR 2024
2 PROPOSED METHOD
We propose the first white-box clean-label data poisoning attack on deep learning- barning-based fair repre- sentation learning (FRL) to degrade fairness. We follow previous attacks classical fair machine learning (Chang et al., 2020; Mehrabi et al., 2021) and suppose a worst-case threat model, which has full knowledge and control of the victim model, as well as the access part of the training samples.
2.1 PRELIMINARIES
We denote dataset consisting of N datapoints by D == fz,, or y4 ) where I; E R represents multivariate nonsensitive feature, a, E {0, is binary sensitive feature and yr is binary class label. As adopted in literature (e.g.. Moyer et al. (2018); Reddy et al. (2021)), we assume z has continuous values that can be fed into neural networks. A FRL model parameterized by 0 has an encoder h to learn representation from the nonsensitive feature by =0) b( 0) and is trained on DD to minimize some fairness-aware loss L(D;0). A data poisoning attack aims to maliciously control the training of this victim model by perturbing few training samples to minimize some attack loss (0). The perturbed training samples are referred to as poisoning samples.
Mutual information (Mi)- fairness aims to minimize (a, z) between sensitive feature and representation z. It makes fair representations highly transferable on a wide range of downstream classification tasks and has become the de facto principle in FRL (Moyer et al., 2018; Creager et al., 2019; Zhao et al., 2019). Formally, the data processing inequality (Cover, 1999) states that [a, 2) 2 [a, s())) > holds for any classifier g acting z In addition, [(a, g())) = is equivalent to demographic parity (DP, Zemel et al. (2013)), one of the most popular group fairness notions. At a colloquial level, if representations from different demographic groups are similar to each other, then any classifier acting on the representations will be agnostics to the sensitive feature and fair thereof. This condition remains valid without access to y when DP cannot be evaluated.
2.2 POISONING FAIR REPRESENTATIONS FORMULATION
Motivated by the importance of MI-based faimess in FRI,, we attack the faireess on some target data Di, by maximizing [aa, z). This involves a bilevel optimization problem. Given a victim and its lower-level loss L(D; 0), the training data D Do U Dat consists of P poisoning samples Dpo and clean samples Dat, the attacker wants to minimize - (a, z) over target data Dt by learning perturbations = { to add on Dpo through solving
minsEC (i = (0) s.t. (A) argming L(D (A) U Dati) 0).
(1)
Our clean-label attack leaves original label y unpoisoned and only perturbs nonsensitive feature x, i.e,, Dro = {x,p + 5p. Op, Up } under constraint C which will be detailed shortly.
Connection to attacking group fairness. Our attack is principled and jeopardizes DP. We use well- defined metric variation of information (VI, Kraskov et al. (2005)). For and a, their VI distance is VI(y, a) ( (1) a) where H-)) is the entropy. Applying triangle inequality g(2),
H
y, and a gives us 7 [g(2), a) 2 [g](z),) () y) H (v) = [[]]z),)) y) - H (y a). By maximizing [(z,a) a) that upper bounds [g((z), a). a successful attack diminishes the guarantee for the MI-based fairness. When H (y a) < [[g(z), y). fitting to predict y from = that maximizes I (g(2), y) will force [(g(2), a) increase, thereby exacerbating the DP violation. Notably, H a) depicts how dependent and are and is expected small when an fairness issue exists (Hardt et al., 2016). We provide empirical success of attacking DP with our attack goal in Appendix E.3.
Unfortunately, Eq. (1) is intractable with difficulty lying in finding the most fairness-violatinn rep- resentations and the method for acquiring them. Mathematically, the first problem involves I(a, 2) which lacks an analytic expression. This makes its computation non-trivial, let alone its maximiza- tion. The second entails a feasible set in the lower-level optimization that is NP-hard to identify due to the non-convexity of deep neural networks. We solve Eq. (1) approximately follows.
2.3 UPPER-LEVEL APPROXIMATIONS FISHER'S 'S LINEAR DISCRIMINANT (FLD)
The lack of analytic form for I(a, necessitates some approximations. Our first step is to lower bound MI by negative binary-cross-entropy (BCE) loss which is easier to estimate. For any classi-
3

---

Published as a conference paper at ICLR 2024
fier g that predicts from x, let q(a x)
be the distribution learned by the optimal g", we have T(a, 2) = (oo,.) log p(a)))
(a)
2 Ep
[(a|z)] p(a)
log
= Ep Dog g(a|z)] + Ep (og
p(a))((al)
where the inequality (a) holds from omitting some non-negative KL terms and is tight when g recovers the true distribution paa 2) as shown in Barber & Agakov (2003). On the other hand, (a) 0, the first term, negative BCE loss of g², is lower bound for T(a,2), and is a measure of how fair the representations are (Feng et al., 2019; Gupta et al., 2021).
since Ep [- log p(a]]
We dub the BCE loss of g" the optimal BCE loss.
However, substituting the MI maximization from Eq. (1) with minimizing the optimal BCE loss does not make it solvable: 9 depends on z's for Dn and how to update them minimize the BCE loss of 9 is unknown. This requires differentiate through the whole optimization procedure.
To walk around this challenge, we note that the optimal BCE loss of g measures how difficult to separate z for Da with = 0 or 1. While this difficulty is hard to tackle directly, it can be approximated by the data separability: If the two classes of data are more separable, one can expect the classification simpler and the optimal BCE loss lower. Motivated by this, we instead maximize Fisher's ' linear discriminant (FLD) score, a
data separability measure.
Specifically, suppose the two classes of representations have mean μa, μr and covariance S S' respectively. FLD maps them
1 1-dimensional space via linear transformation v which induces separation Se (v 1ra w 1e') ) (v Sov o' S'v).. Its maximal value over v. termed as FLD score, is
(nr 1 T + - (n n²) when v x (S" + -1)--((( - This equation allows us compute its gradient with respect to z for Dma which gives the direction to update these representations in order to make it less fair. For stability we regularize s by
(n 1²) ) (S + S =l) - -m'),
and resort to solving the following bilevel optimization:
minsec -s() s.t. 6 (A) argming (A)
(3)
In Appendix we extend our attack to multi-class sensitive feature scenarios. Remark 2.1. Maximizing T(2, a.) is a general framework poison FRL and admits other proxies such as sliced mutual information (Chen et al., 2022), kernel canonical correlation analysis (Akaho, 2007), and non- nan-aaramttric dependence measures (Székely et al., 2007). In this work, we use FLD because of its conceptual simplicity, interpretability, and good empirical performance. As one may recognize, when p(z a 1) and p(= a = 0) are Gaussian with equal variance, FLD optimal (Hamsici & Martinez, 2008) whose BCE loss attains the tight lower bound of [(2,a) up to constant (a). In this case, our method provably optimize the lower bound of [(z, a) whereas other proxies may not due to the lack of direct connections to mutual information. While the Gaussianity may not hold in general, FLD score as a measure of data separability is still valid (Fisher, 1936), and we verify its efficacy for our goal in Appendix E.2, where we show that FLD score is highly informative for the empirical minimal BCE loss of a logistic regression.
2.4 LOWER-LEVEL APPROXIMATION ELASTIC-NET GRADMATCH (ENG)
Bilevel optimization in Eq. (3) enjoys a tractable upper-level loss but its lower-level optimization remains challenging due to the use of deep neural networks in FRL and needs further approxima-
tion. To this end, we fix parameter 0 in a pre-trained victim model and treat its lower-level gradi- ent on poisoning samples LD (Do (A); 8) as a function of A. An attack is launched by aligning (DL and (o) by maximizing their cosine similarity
B(0, A) (Ves(0), (oL(Dpo
(4)
a V (PP:
with respect to A. When the two directions are matched, gradient descents the lower-level loss L from poisoning samples will decrease the upper-level loss - as well.
The concept of gradient matching in poisoning attacks was initially introduced in GradMatch by Geiping et al. (2020) for image classification. To enhance the stealthiness of such attacks in fair- ness degradation scenarios, we impose new constraints the learning perturbations. First,
4

---

Published as a conference paper at ICLR 2024
hard constraint C maandates that poisoning samples must reside within the clean data domain, i.e., ming, Zem Σ 6pm + Pm I maXa,ED; Pem for all feature dimensions (1 S m S M) and poisoning samples (1 S p S P), this allows using dimension-specifie ranges. Second, the soft con- straint employs the elastic-net penalty (Zou & Hastie, 2005; Chen et al., 2018) to promote sparsity in each 6p (i.e, only few dimensions perturbed). Combining these constraints, we formulate below optimization problem aimed at poisoning fair representations:
minsec -B(0, + Σ (Arll5plla + A211óp11)-
(5)
In execution, rescale the three terms before tuning X's to avoid bearing with the magnitude differ- ttaaaagggeheeshldinng algorithm
ence. Non-differentiable L1 norms are tackled with the iterative
(ISTA, Beck & Teboulle (2009)) 85+11 Projc (Sx, [6] + a* (B(O, A) ^211o, [ ) where Sx, is the element-wise projected shrinkage-thresholdiin function and Projc is the projection onto C. We refer to our attack by Elastic-Net GradMatch (ENG) and summarize it in Algorithm
Computation complexity. Denote the number and dimension of poisoning samples by P and M, iteration numbers of running the attack by T, dimension of by D. The computation complexity of ENG and GradMatch (Geiping et al., 2020) are O(TM + TDPM) and OPTDPM), respectively. The additional computation cost (TTM) due to the one-step ISTA in ENG is marginal.
ANALYSIS ON MINIMAL NUMBER OF POISONING SAMPLES
non-trivial minimize the number of poisoning samples that deteriorate the victim model's performance. An insufficient amount may not impact lower-level optimization. while large amount makes easy to detect the attack, leading to a direct failure (Huang al., 2020; Koh et al., 2022). Our analysis is built upon the convergence of upper-level loss (()) = =0). Our conclusion relies on the following assumptions.
Assumption 3.1 (smooth upper- and lower-level losses). There exists C 0 such that upper-level loss (8) and lower-level loss [.)) are C'smooth, i.e., their gradients are C-L-ipcchizz continuous.
Assumption 3.2 (attack well-trained victims only). Before attack, the victim is well-trained so its gradient each clean sample performs like random noise with mean and finite norm a.
Assumption 3.3 (well-matched gradients). After being attacked, gradient (oL evaluated any poisoning sample is an unbiased estimator of the gradients (O) with bounded error norm, i.c. for any poisoning sample p. (L, (OU(()) Cp where E[(p] = and E|[|cpl|| a.
Assumption 3.1 is a fairly weak condition that has been widely used (Colson et al., 2007; Mei & Zhu, 2015; Sinha et al., 2017), the rest two are introduced by us and are well-suited to the context of our proposed framework. Assumption 3.2 is valid because 0 undergoes thorough training with the clean samples. Assumption 3.3 presumes that given constant gradient (O), we can construct an unbiased estimator with /' poisoning samples E[ Σp= VoL,((0 = (O). With these assumptions, we obtain the following theorem.
Theorem 3.4. Suppose that Assumption 3.1, 3.2, and 3.3 hold. Let and N be the number of poisoning and total training samples, respectively. Set the learning rate and the batch size to n. Then, the ratio of poisoning data P/N should satisfy
P N
2n VoU + aC
aCo²
2 c
(6)
2
such that the upper-level loss U(0) is asymptotic to an optimal model. Here e is a small constant (e.g., (0-4) sufficient descent, and 0 is a well-trained model on the clean data.
Deferring the proof to Appendix C, we summarize the underlying idea. We assume the pretrained victim converged with respect to L(L(O) Besides, after applying ENG, a poisoning sample can (in expectation) induce gradient equal to (oU(() so training the victim model on it will optimize the upper-level loss (@)). However, clean samples may obfuscate this goal as their lower-level gradients do not depend on the upper-level loss. To counteract with this effect, minimal portion of poisoning
5

---

Published as a conference paper at ICLR 2024
samples are needed to dominate the training. In practice, learning rate is often small compared with C, batch size n is large, and [() is much greater than 0. Therefore, the minimal portion bound is expected to be smaller than 1.
Practical Signifieance. Theorem 3.4 sheds light on the difficulty of ENG and other BrradMatch-basee attacks, whereon defending strategy can be built. If batch size n. is large, the attack simpler and needs fewer poisoning samples. So reducing n should help defense and we evaluate its perfor- mance in Appendix E.7. In addition, term x² is affected by the lower-level gradients, whose increase will require more poisoning samples. This helps explain why adding noises to gradients can defend against GradMatch as ver- ified in Geiping al. (2020).
Algorithm 1 Craft Poisoning Samples with ENG Attack
1: Input: clean data Pat poisoning data ppo, target data Du: victim 0 and its lower-level loss L; number of pre- training epochs E and attack iterations T.
2: Fix Dro unperturbed. Pretrain victim on Dpo U Da for
E epochs and obtain 0F
3: Compute upper-level gradient (DU 0E).
4: Randomly initialize A = {5p.P = I,..., P) in C.
for t I,..., do
6: Compute lower-level gradient (DL(Dpo(A)): as a function of A'.
7: Compute ENG loss in (5) and update A' with ISTA.
8: end for
9: return Dpo when using A
EXPERIMENTS
We evaluate the proposed attack on four FRL models trained on two faimess benchmark datasets and show its effectiveness through extensive experiments. Ablation studies and practical insights are also given to help understand how the attack succeeds in poisoning victims.
4.1 SETUP
Attack Goals. We consider three variants of Eq. (2) to maximize: (a) FLD follows Eq. (2) with c = 10 4 to stabilize the covariance estimation. (b) sFLD takes the same form as FLD but does not propapro through covariance terms when computing - vo)( (c) EUC replaces the covariance terms with an identity matrix?.
Attacks. An attack to maximize score X using ENG is referred to ENG-X, for instance, ENG- FLD is to maximize FLD. Conceptually, ENG-EUC is suitable for small Du as it omits covariance matrix, which can be unstable to estimate. ENG-FLD should be favored when Dm is large as it is based on the exact FLD score 8 and should measure separability more accurately, which is expected to benefit solving Eq. (5) as well. ENG-sFLDD strikes balance in between by using only part of covariance information.
FRL Victims. We select four representative FRL models as our victims. CFAIR and CFAIR-EO (Zhao el al., 2019) are based on adversarial learning and seek to achieve different fairness notions. Non-adversarial learning based ICVAE-S and ICVAE-US (Moyer et al., 2018) differ in whether y is accessible. We follow the official codes to implement the victims as detailed in Appendix D.
Datasets. We train victims on two benchmark datasets from UCI repository that are extensively studied in fair machine learning, which are mr-processed following Zhao et al. (2019); Reddy et al. (2021). Adult (Kohavi, 1996) contains 48,842 samples of US census data with 112 features and the objective is to predict whether an individual's yearly income is greater than $50K dollars or not. The sensitive feature is gender. German (Dua & Graff, 2017) consists of 1,000 samples of personal financial data with 62 features and the objective is to predict whether or not client has good credit score. The sensitive feature is binarized age as in Moyer et al. (2018) and we adjust the threshold to increase its correlation with label In Appendix E.6 we study multi-class sensitive feature race. On both datasets, we leave 20% of total samples out as Dmn More results on COMPAS (Dieterich et al., 2016) and Drug Consumption Datasets (Dua & Graff, 2017) are presented in Appendix. E.8.
"This score is equivalent to the squared Euclidean distance between μr and μt' and gets its name thereof. In Appendix E we discuss the practicability of perturbing pre-processed versus raw data We increase the lower threshold of defining advantaged group from 25 to 30.

---

Published as a conference paper at ICLR 2024
Evaluation. We treat decrease in the BCE loss of a logistic regression predicting from mea- sure of increase in [(z, a). To verify how group faimess and representation utility can be affected, present the exacerbation of DP violation and accuracy of predicting y from x in Appendix E.3 and Appendix E.4 respectively. Representations are extracted after training the victims on poisoned Adult and German dataset for 20 and 50 more epochs respectively considering their size difference.
Baselines. We compare ENG-based attacks with four variants of anchor attack (AA), a recent heuris- tic generic poisoning attack on classical fair machine learning (Mehrabi al., 2021). RAA-y and RAA-a randomly picks one training sample from the subgroup with (y = 1, a 0) and one with (y = 0, a 1) after each epoch, then makes copies of the two chosen samples with flipped y or respectively. NRAA-y and NRAA-a replaces the random selection in RAA counterparts with pick- ing from each subgroup the training sample that has the most neighbors within a pre-specified radius and has not been selected yet. (N)RAA-Y were proposed in Mehrabi et al. (2021), and we imple- ment AN)RAA-a following the same logic. Note that (N)RAA-y are not clean-label, (N)RAA-a are but they directly modify the sensitive feature. In contrast, our attacks only modify the nonsensitive feature. Moreover, all baselines are allowed to use any training sample to poison, while ours can only perturb given randomly selected set of poisoning samples. These differences put our proposed attacks in an unfavorable situation under direct comparison. Nevertheless, ours can still outperform the four baselines by a large margin.
4.2 COMPARISON BETWEEN ENG AND (N)RAA.
We compare three ENG-based attacks with penalty coefficients A = 0.0025, À2 = 0.005 (Eq- (5)) against four AA attacks under different settings where 5% to 15% of training data are used for poisoning. Performance of an attack is measured by the decrease of BCE loss, higher is better, and corresponding DP violations are reported in Appendix E.3.
Figure 2 shows results averaged over 5 replications. Three ENG-based attacks achieved notable performance (on both BCE loss and DP violations) in various settings. In contrast, AA encountered severe failures, for instance, when attacking CFAIR trained on German dataset, only RAA-a suc- ceeded with all the three budgets. Such failures cannot be fully attributed to the budgets: (N)rAA-y succeeded with budget 10% but failed with budget 15%. (N)RAA-a occasionally achieved the best performance because of much stronger capacities. Nonetheless, the proposed ENG-based attacks beat AA baselines in terms of better and more reliable performance by a large margin in most case. When comparing the three proposed attacks with each other, their performance difference matched our previous analysis, e.g., ENG-FLD gave the best result on larger Adult dataset. These results clearly establish the efficacy of our attack.
Vidim CAAI
Victan CHAIS-0
Victi ICWR2 5
Vicim ICWRE
0:2
6.
1.5-
5.75
0375
0.02
342
0075
DNC/LC
0.02
032
0025
DVG
0.
2
2iv
61
635
5.th
00
6m
01
515
Fadiin of Poloning Semplo
Fation Prbonng Sempla
Parton Poturing Serpto
Pedberl Paicenng Candils
(a) Adult dataset
Victon
Willn
0C6
0.00
MMU
5.50
054
(MG-EUC
-220
-0.31
000
050
DNC
-100
4,32
23
lorton Poboning Sampla
Forticn roooting sampla
Portzs of Pebooing
Fartic Foboninng Semple
(b) German dataset
Figure 2: ENG-based attacks reduce BCE loss more than AA baselines with less portion of poison- ing samples. Results are averaged over 5 independent replications and bands show standard errors.
7

---

Published as a conference paper at ICLR 2024
4.3 PERFORMANCE OF ELASTIC-NO PENALTY
Next, we sprovide sensitivity analysis of ENG-based attacks against the choices of Å and 22- We set A 2 and vary λ following Chen al. (2018). Results are again averaged over replications. Figure 3a exhibits how elastic-net penalty affects the performance of ENG-FLD under different poisoning budgets where victims are trained on Adult dataset. More results are deferred to Appendix E.5 due to page length but conclusions here hold in general. All three attacks are relatively robust to small and intermediate level of elastic-net penalty. Moreover, improvement from applying the elastic-net penalty is observed (e.g., column in Fig 3a). This implies that the penalty can actually help stabilize the optimization.
We further study how elastic-net penalty regularizes the learned perturbations by computing Lı and L2 norms. We only present L, norms of perturbations using ENG-EUC attack on Adult dataset as an illustration and defer others to Appendix E.5 after observing similar trends. According to Figure 3b, L1 norms were significantly shrank by clastic-net penalty with mild degradation on the attack performance. For instance, elastic-net penalty with A 0.0025 effectively reduced the Lı norm of perturbations by a third without hurting the performance of attacking CFAIR. When a more stealthy attack is wanted, A 0.01 was able to launch satisfactory attack with only one half budget of perturbation norm. These results clearly show the efficacy of our proposed ENG-based attacks.
Given these merits of elastic-net penalty, one may ask if it be used in other attacks such AA. In Appendix E.5 we discuss the difficulty of doing so and highlight its affinity to our attack by nature.
Victim CHAET
Vistin CPAIREDD
Victm ICUAE-5
Victm e (CVRE US
005
0.04
3.
0.00
5373
060
0.2
5322
0.
0.05
63
315
3.0
0.3
3.5
0.5
0,1
0.55
06
8.5
(a) Decrease of BCE loss insensitive to under small intermediate level of elastic-net penalty.
20
121
ois Fortin Polosiig Samplaa
2/.
Porton "olooiing Sempli
Fartoo of Fabiiinggg Sumplic
1C Pouneeo Sampiix
(b) La norm of learned perturbations is effectively restricted by elastic-net penalty.
Figure 3: Decrease of BCE loss and I, norm perturbations learned by ENG-FI.D attack. Victims trained Adult dataset and results are averaged over 5 replications.
4.4 ROBUST FEATURES OF VICTIMS
ENG-based attacks can do feature selections, and we consider unselected features as robust ones for the victim, in the sense that they did not help in attacking. We end this section with a case study on this aspect. Table shows the percentage of selected features and corresponding robust features when attacking ICVAE-S and ICVAE-US trained Adult dataset with ENG-FLD. From the table, elastic-net penalty successfully reduced the perturbed features by 20%, and robust features for ICVAE-US and ICVAE-S are largely overlapped. These results helped understand how ENG- FLD was launched by walking around these robust features when attacking the victims. Note that this robust features identification
applicable for AA-based attacks.
RELATED WORK
Fair representation learning (FRL) is family of algorithms to learn fair representations from nonsensitive features such that any downstream tasks (c.g., classification) acting on these represen- tations will be fair. Towards this goal, different approaches removing sensitive information from
8

---

Published as a conference paper at ICLR 2024
learned representations have been proposed. To name few, Zemel et al. (2013) set representations to be multinomial and used fairness violations based on the representations as penalties, and Madras et al. (2018) derived different bounds for these violations used for adversarial regularization. Mutual information and other measures between distributions have also been used as penalties to encourage independence between the representation and sensitive features, either in an adversarial (Xie al., 2017; Creager et al., 2019) or non-adversarial way (Louizos et al., 2015; Moyer et al., 2018; Sarhan al., 2020; Wang et al., 2021). Recently, Zhao al. (2019) proposed to learn different encoders on different label groups with theoretical guarantees and achieved state-of-the-ar performance in an adversarial manner Reddy et al. (2021). Moyer al. (2018), on the other hand, provided a non- adversarial based solution and also got promising results. We select representative methods from adversarial and non-adversarial regimes to test our attack.
Data poisoning attack aims to achieve some attack goal with a victim model by controlling its training via injecting poi- soned samples. Early works showed that simple heuristics such as label flipping (Barreno et al., 2010; Paudice et al., 2019) can succeed in attack. However, these poi-
Table I: Percentage of selected features and top ro- bust features from Adult dataset, P/N denotes the portion of poisoning samples, attacker is ENG-FLD.
A (A2 2A1)
P/N
Top Robust Features
0,0025
0.005
0.01
soning samples often look unnatural and are easy to detect (Papernot & McDaniel, 2018; Paudice et al., 2019). Consequently, clean- label attacks that only modify the poison- ing samples' features but not labels are pre- ferred (Shafahi et al., 2018). Another drawback of heuristic attacks is the lack of performance guarantee as they do not directly solve the attack goals. In practice they may perform less effective.
ICWES
0.05
0.1 0.99
0.99
0.95
0.96
0.89
0.89
0.73
0.75
workelass, marital-stane statts
occapatton, native-country
0.15
0.99
0.97
0.89
0.77
0.05
0.95
0.89
0.84
0.79
native-country.
0.1 0.95
0.91
0.85
occapation, workclass, 0.79 education
0.79
0.15
0.96
0.91
0.85
Bilevel optimization is widely used for data poisoning attacks (Bard & Falk, 1982; Biggio et al., 2012: Geiping et al., 2020; Jagielski et al., 2021; Koh et al., 2022). For convex victims such as logistic regression and support vector machine, the lower-level optimization is characterized by the KKT condition. This reduces the bilevel optimization to constrained optimization that can be solved exactly (Mei & Zhu, 2015). For other victims, unfortunately, when their optimal solutions NP-hard to identify, are the bilevel optimization problems (Colson et al., 2007; Sinha et al., 2017) and inexact solutions are needed. When the second-order derivatives of the lower-level loss is cheap, using influence function to identify influential samples for the victim training and poisoning them can produce strong attacks (Koh & Liang, 2017). These attacks have been successfully applied to classical fair machine learning (Chang et al., 2020: Solans et al., 2021; Mchrabi et al., 2021), but the non-convexity of neural networks and expensive influence function computations make them unsuitable for poisoning FRL. Approximate solutions for attacking deep learning models have been proposed recently. For instance, inspired by model-agnostic meta-learning (MAML, Finn et al. (2017)), MetaPoison (Huang et al., 2020) back-propagted through a few unrolled gradient descent steps to capture dependency between the upper- and lower-level optimizations. GradMatch (Geiping et al., 2020) matched the gradient of upper- and lower-level losses and achieved state-of-the-a performance. However, it is unclear how to apply them to poison FRL. In this work, we propose the first work towards this goal and reduce it to an approximate optimization that can be handled.
CONCLUSION AND FUTURE WORKS
We develop the first data poisoning attack against FRL methods. Driven by MI-based fairness in FRL, we propose new MI maximization attack goal and reveal its connection to existing fairness notion such as demographic parity. We derive an effective approximate solution to achieve this attack goal. Our attack outperforms baselines by a large margin and raises an alert of the vulnerability of existing FRL. methods. We also theoretically analyze the difficulty of launching such an attack and establish an early success of principled defense. Motivated by promising results on tabular data, which is the primary focus of many FRL methods (Moyer et al., 2018; Zhao et al., 2019), we plan to extend our attack to fair machine learning on large-scale image and text datasets that also relies on deep neural networks, and delve into attacking these methods in the future. In addition, Jagielski et al. (2021) showed that an attack can be much more effective towards certain subpopulations and impossible defend against, and we plan explore this for further improvement of our attack.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work is supported in part by the US National Science Foundation under grant NSF IIS-2226 108. Any opinions, findings, and conclusions or recommendations expressed this material those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
REFERENCES
Shotaro Akaho. A kemel method for canonical correlation analysis, 2007.
David Barber and Felix Agakov. The im algorithm: variational approach information maxi- mization. Advances in neural information processing systems, 16(320):201,000
Jonathan F Bard and James E Falk. An explicit solution to the multi-level programming problem. Computers & Operations Research, 911:777-10,, 1982.
Solon Barocas and Andrew Selbst. Big data's 's disparate impact. Calif. Rev., 104:671, 2016.
Marco Barreno, Blaine Nelson, Anthony D Joseph, and J Doug Tygar. The security of machine learning. Machine Learning. 81:121-148. 2010.
Amir Beck and Mare Teboulle. fast iterative Shrinkage-thresholding algorithm for linear inverse problems. SIAM journal imaging sciences, 2(1):183-200) 2009.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector ma-
chines. arXiv preprint arXiv:/206.6338 2012.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commer- cial gender classification. Conference on fairness, accountability and transparency, pp. 77-91. PMLR, 2018.
Hongyan Chang, Ta Duy Nguyen, Sasi Kumar Murakonda, Ehsan Kazemi, and Reza Shokri. On adversarial bias and the robustness of fair machine learning. arXiv preprint 20Xv:2000.08869, 2020.
Pin- Yu Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsich. Ead: elastic-net attacks to deep neural networks via adversarial examples. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Yanzhi Chen, Weihao Sun, Yingzhen Li, and Adrian Weller. Scalable infomin learning. In Al- ice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances Neu- ral Information Processing Systems, 2022. URL https: http://pppneeeeeeeeeeeee Ojakr9ofova
Benoît Colson, Patrice Marcotte, and Gilles Savard. An overview of bilevel optimization. Annals of operations research, 153:235-256, 2007.
Thomas M Cover. Elements of information theory. John Wiley Sons, 1999.
Elliot Creager, David Madras, Jörn-Henrik Jacobsen, Marissa A. Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentanglement, 2019.
William Dieterich, Christina Mendoza, and Tim Brennan. Compas risk scales: Demonstrating ac- curacy equity and predictive parity, Northpointe Inc, 7(4):1-36, 2016.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL htthttt://arcciiv.. ics.uci, wew.uu/...ddm...
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Rich Zemel. Faimesss through awareness, 2011.
Rui Feng, Yang Yang. Yuehan Lyu, Chenhao Tan, Yizhou Sun, and Chunping Wang. Learning fair representations via adversarial framework. arXiv preprint arXiv: 1904. 13341, 2019.
10

---

Published as a conference paper at ICLR 2024
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learing for fast adaptation of deep networks. In Interaational conference on machine learning, pp. 1126-1135. PMLR, 2017.
Ronald A Fisher. The use of multiple measurements in taxonomic problems. Annals of eugenics, (2):179-188, 1936.
Jonas Geiping, Liam Fowl, W Ronny Huang, Wojeiech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' brew: Industrial scale data poisoning via gradient matching. arXiv preprint arXiv:2009.022: 02276, 2020.
Umang Gupta, Aaron M Ferber, Bistra Dilkina, and Greg Ver Steeg. Controllable guarantees for fair outcomes via contrastive information estimation. Proceedings of the AAAI Conference on Artificial Intelligence, pp. 7610-7619, 2021.
Onur C Hamsici and Aleix M Martinez. Bayes optimality in linear discriminant analysis. IEEE transactions on pattern analysis and machine intelligence, 30(4):647-657, 2008.
Moritz Hardt, Eric Price, and Nathan Srebro. Equality opportunity in supervised learning, 2016.
W Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein. Metapoison: Prac- tical general purpose clean-label data poisoning. Advances in Neural Information Processing Systems, 33: 12000-1091, 2020.
Vijay S. Iyengar. Transforming data to satisfy privacy constraints. Proceedings of the Eighth ACM SIGKDD International Conference Knowledge Discovery and Data Mining (KDD '02'), pp. 279-288, 2002.
Matthew Jagielski, Giorgio Severi, Niklas Pousette Harger, and Alina Oprea. Subpopulation data poisoning attacks. In Proceedings of the 2021 ACM SIGSAC Conference Computer and Com- munications Security, pp. 1104-3122 2021.
Zhimeng Jiang, Xiaotian Han, Chao Fan, Fan Yang, Ali Mostafavi, and Xia Hu. Generalized de- mographic parity for group fairness. In International Conference Learning Representations, 2022.
Toshihiro Kamishima, Shotaro Akaho, and Jun Sakuma. Fairness-aware learning through regular- ization approach. In 2011 IEEE Ith International Conference Data Mining Workshops, pp- 643-650. IEEE, 2011.
Pang Wei Koh and Perey Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885-1894, PMLR, 2017.
Pang Wei Koh, Jacob Steinhardt, and Percy Liang. Stronger data poisoning attacks break data sanitization defenses. Machine Learning, pp.
Ron Kohavi. Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid. In Proceed- ings of the Second International Conference on Knowledge Discovery and Data Mining, KDD'96 96, pp. 202-207. AAAI Press, 1996.
Alexander Kraskov, Harald Stögbauer, Ralph Andrzejak, and Peter Grassberger. Hicrarchical clustering using mutual information. Europhysics Letters, 70(2):278, 2005.
Tianci Liu, Haoyu Wang, Yaqing Wang, Xiaoqian Wang, Lu Su, and Jing Gao. Simfair: A unified framework for fairness- aware multi-labe classification. Proceedings of the AAAI Conference on Artificial Intelligence, 37312):14338-1444 2023.
Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel. The variational fair autoencoder. arXiv preprint 1r 2015.
David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning adversarially fair and transferable representations, 2018. URL https: arxiv orr/abs// 1802 06009.
Ninareh Mehrabi, Muhammad Naveed, Fred Morstatter, and Aram Galstyan. Exacerbating algo- rithmic bias through fairness attacks. Proceedings of the AAAI Conference on Artificial Intelli- gence, pp. 8930-8938, 2021.
11

---

Published as a conference paper at ICLR 2024
Shike Mei and Xiaojin Zhu. Using machine teaching identify optimal training-set attacks machine learners. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Daniel Moyer, Shuyang Gao, Rob Brekelmans, Aram Galstyan, and Greg Ver Steeg. Invariant representations without adversarial training. Advances in Neural Information Processing Systems, 31, 2018.
Nicolas Papernot and Patrick McDaniel. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint
2018.
Andrea Paudice, Luis
and Emil C Lupu. Label sanitization against label flipping poisoning attacks. In ECML PKDD 2018 Workshops: Nemesis 2018, UrbReas 2018, SoGood 2018, /WAISe 2018, and Green Data Mining 2018, Dublin, Ireland, pp. 5-15, 2019.
Dino Pedreshi, Salvatore Ruggieri, and Franco Turini. Discrimination-awen aware data mining. In Pro- ceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 360-568, 2008
Ram Mohan Rao, S Murali Krishna, and AP Siva Kumar. Privacy preservation techniques in big data analytics: survey. Journal of Big Data, 5:1-12, 2018.
Charan Reddy, Deepak Sharma, Soroush Mehri, Adriana Komero-Soriang Samira Shabanian, and Sina Honari. Benchmarking bias mitigation algorithms in representation learning through fair- ness metrics. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL https:/ //pp//pppeeviieeeeeee/ee net forum? 2id= OTneQUEWPKu.
Mhd Hasan Sarhan, Nassir Navab, Abouzar Eslami, and Shadi Albarqouni. Fairness by learning orthogonal disentangled representations. In Computer Vision-ECCV 2020: 16th European Con- ference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX 16, pp. 746-761. Springer, 2020.
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. Advances in neural information processing systems, 31, 2018.
Ankur Sinha, Pekka Malo, and Kalyanmoy Deb. A review on bilevel optimization: From classical to evolutionary approaches and applications. IEEE Transactions on Evolutionary Computation, 22(2):276-29) 2017.
David Solans, Battista Biggio, and Carlos Castillo. Poisoning attacks algorithmic faimesss. In Machine Learning and Knowledge Discovery Databases: European Conference, ECML 2020, Ghent, Belgium, September 14-18, 2020, Proceedings, Part pp. 162-177. 177. Sprinnger 2021.
Gábor J Székely, Maria L Rizzo, and Nail K Bakirov. Measuring and testing dependence by corre- lation of distances. The Annals of Statistics, 35(6):2769-2794, 2007.
Haoyu Wang, Hengtong Zhang, Yaqing Wang, and Jing Gao. Fair classification under strict un- awareness. Proceedings of the 2021 SIAM International Conference on Data Mining (SDM), pp. 199-207. SIAM, 2021.
Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig. Controllable invariance through adversarial feature learning. Advances in neural information processing systems, 30, 2017.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. Fair- ness constraints: Mechanisms for fair classification. In Artificial Intelligence and Statistics, pp. 962-970. PMLR, 2017.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International conference on machine learning, pp. 325-333. PMLR, 2013.
Han Zhao, Amanda Coston, Tameem Adel, and Geoffrey J Gordon. Conditional learning of fair representations. arXiv preprint arXiv: 1910.7761,, 2019.
12

---

Published as a conference paper at ICLR 2024
Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the royal statistical society: series B (statistical methodology), 67(2):301-32,, 2005.
13

---

Published as a conference paper at ICLR 2024
A MOTIVATION OF ATTACKING FAIR REPRESENTATIONS
Attacking mutual information at representation level, as formalized in Section 2, aligns better with fair representation learning (FRL) and is more universal than attacking classifiers. For example, a bank may want to obtain representation 2 for each user that can be used to determine their eligibility for existing and upcoming financial products such as credit cards without concerning about fairness again (Zhao et al., 2019). Here each financial product is associated with (unique) label, and determining the eligibility entails a classification task. this case, there are two challenges for delivering classification-based attack. First, one has to determine and justify which classifier to use and why consider the fairness of this specific classification task. Second, for any upcoming financial product, its label does not exist and one cannot obtain classifier (we need this label to train the classifier), let along attacking it. In contrast, a representation-Ievel attack can overcome the two challenges in single shot. As discussed in Section 2, for any classifier g acting on z, by maximizing the mutual information T(z, a) between z and sensitive feature a, T(g(z), a) will be maximized long as the fairness concern exists. This implies launching attack on all labels simultaneously, including the ones where classifiers cannot be trained.
B EXTENSION TO MULTI-CLASS SENSITIVE FEATURE
To attack a FRL method trained on multi-class sensitive feature a E [K], We first define äg 1(a k) to mark whether the sample belongs to the k-th sensitive group. Then we immediately have [(2,0) a) 2 [[2, āz. thanks to the data processing inequality. This implies that
T(z a) Z
T(z
k=1
Note that in RHS each term is the mutual information between and binarized â and is lower bounded by the BCE loss, therefore we can approximate each one with a FLD score readily. This idea is similar to transforming a K-class classification to K one-vs-all binary classifications. We report empirical results in Appendix E.6.
Further simplification is viable by choosing some specific i and attacking I(2, āx) only. This allows one to launch an attack when K large. However, is noteworthy that handling a sensitive feature that involves many groups (large K) is general challenge for fair machine learning. The difficulty twofold. First, it involves more complicated constraints, making the problem harder to optimize. Second, by categorizing data into more fine- fine-grained sensitive groups, each group will have fewer samples and the algorithm may suffer from unstable estimation issues (Jiang et al., 2022; Liu et al., 2023). As a result, when the number of sensitive groups is large, fair machine learning methods often bin them into few larger groups in pre- pre-processing (Zemel et al., 2013; Moyer et al., 2018; Zhao et al., 2019; Reddy et al., 2021).
C OMITTED PROOF OF THEOREM 3.4
We start with restating Theorem 3.4.
Theorem C.I. Suppose that Assumption 3.1, 3.2, and 3.3 hold. Let P and N be the number of poisoning and total training samples, respectively. Set the learning rate to a, and assign the batch size with n. Then, the ratio of poisoning data (Le., P/N) should satisfy
P N
+ aC 2n VoU
aCo²
2 c
2
such that the upper-level loss (()) (i.e., negative FLD score) is asymptotic to optimal model. Here e is a small constant (e.g., 10 4) for sufficient descent, and is a well- trained model on the clean data.
14

---

Published as a conference paper at ICLR 2024
Proof. Without loss of generality, assume that the victim is trained by mini-batched stochastic gradient descent, i.e., given current parameter goli, the new value is updated by
gnew gold
V VeLi (
7
where (() denotes the loss on the i-th training sample. Let denote the number of poisoning samples selected in the current batch, we have that follows Hypergometric distribution. Given
p Up poisoning samples in the current batch, we collect all randomness in the minibatch gradient as
T2 Dpp
old
VoL (Oold).
Ej
i=1
According to Assumption 3.3 and 3.2 have Eleah! and
Rp
Rp
e [2]
=
E
Σ
4
+
V VoL.
Ri
Σ
Eill
(o +0
no?
as crossing terms have mean zero. Moreover, we can express
V (0
(oLi
+
Poisoning samples
Clean samples
np
Uip
18p
Σ
G
(Holl
i=1
= np (Ooll))))
cold
This implies that further given Assumption 3.2, the minibatch gradient is a biased estimator of the gradient of the ueper-level loss. To see this, by the law of total expectation
n V (ooid
E
V
Looldd)
1 = 7g
71
E E [pV. Vo (Buld))) cold p [-=]]]]
E (VVO((
VOU(Or [E(p]
n P
N (OU()))))
Under Assumption 3.1, descent lemma gives us
C gold) = 110pew gold
( < U (gold) + (VeU( gnew
a²c
= =(Odd a (Vo (Ookd)
(ool +
V
VoL, (00l
2n
15

---

Published as a conference paper at ICLR 2024
Take expectation both side with respect to the mini mini-batch get
a²C 2n
11
) S U((ool) - av (VVo VOU()))))
E1
aP
a²C
U((0ol))
N VoU
(
E
Σ
Vo
(fold)) |1
2n²
and
V وo:; (0old) 21 = E E
0 (ool)) )1( 1p np
E
E [Il U (ghh)
10 p mpp
E [ p
(old)) 11 || 1| + 2p V oU (grld) ) e
np]
E'p" V /0 ( 2| + E[I |2] + 0
n² V V (Bol))
( || + no?
Plugging in back to equation (8) we have
aP
a²C. (n ( +
E(U( ) - ((glld)
N
2n
aP a²c
Voli + a²Co²
= =U((ol))
N 2
2n
Therefore, sufficient descent such that
(fuew) < ((0ll)) - call VoU (0r
for some 2 0 can guaranteed by
a& a²C (Boldy)) a²o2²
2 ca
)OU(())))
2n
Rearrange
aCo²
+ aC
ct
N
2n
[oU((o)))
2
This completes our proof.
D MORE IMPLEMENTATION DETAILS
We provide details about victims' architectures and training. Our model architectures follow official implementations in Moyer et al. (2018); Zhao et al. (2019).
On Adult dataset, we use
CFAIR:
Encoder: linear, representation E 60
Discriminators: one hidden layer with width 50, using ReLU activation.
Classifier: linear.
Training: AdaDelta optimizer with learning rate 0.1, batchsize 512, epochs 50.
CFAIR-EO:
Encoder:
linear, representation E 60
Discriminators: hidden layer with width 50, using ReL.U activation.
16

---

Published as a conference paper at ICLR 2024
Classifier: linear.
Training: AdaDelta optimizer with learning rate 0.1, batchsize 512, epochs 50.
ICVAE-US:
Encoder:
one hidden layer with width 64, output representation RR:00 using Tanh activation.
Decoder: one hidden layer with width 64, using Tanh activation.
Classifier: one hidden layer with width 64, using Tanh activation.
Training: Adam optimizer with learning rate 0.001, batchsize 512, epochs 50.
ICVAE-S:
Encoder: one hidden layer with width 64, output representation = E R230 using Tanh activation.
Decoder: one hidden layer with width 64, using Tanh activation.
Classifier: one hidden layer with width 64. using Tanh activation.
Training: Adam optimizer with learning rate 0.001, batchsize 512, epochs 50.
On German dataset, we use
CFAIR:
Encoder: linear, representation = E 18
Discriminators: one hidden layer with width 50, using ReLU activation.
Classifier: linear.
Training: AdaDelta optimizer with learning rate 0.05, batchsize 64, epochs 300.
CFAIR-EO:
Encoder: linear, representation E 6U
Discriminators: hidden layer with width 50, using ReLU activation.
Classifier: linear.
Training: AdaDelta optimizer with learning rate 0.05, batchsize 64, epochs 300.
ICVAE-US:
Encoder: hidden layer with width 64, output representation R030 using Tanh activation.
Decoder:
one hidden layer with width 64, using Tanh activation.
Classifier: one hidden layer with width 64. using Tanh activation.
Training:
Adam optimizer with learning 0.001, batchsize 64, epochs 300.
ICVAE-S:
Encoder: one hidden layer with width 64, output representation R:30 using Tanh activation.
Decoder: one hidden layer with width 64, using Tanh activation.
Classifier: one hidden layer with width 64, using Tanh activation.
Training: Adam optimizer with learning rate 0.001, batchsize 64, epochs 300.
During training, followed GradMatch (Geiping et al., 2020) and did not shuffle training data after each epoch. For better comparison, victims were always initialized with random seed 1 to remove randomness during the pre-training procedure. In different replications, selected different poisoning samples with different random seeds. Experiments that consist of replications used seed to 5 respectively.
17

---

Published as a conference paper at ICLR 2024
E MORE EXPERIMENT RESULTS
E. PRACTICABILITY OF PERTURBING PRE-PROCESSED DATA
Many poisoning attacks on images classifiers perturb raw data (namely pixels, (Huang et al., 2020; Geiping al., 2020)), in this work perturb pre-processed data. However, this does not necessarily undermine the practical significance of our work. To see why attacking the pre-processed data is practical, we take a view from the scope of data security. Many FRL methods are applied in high- stakes domains, such as loan application or job market screening. Due to the privacy concern, Data anonymization has been used by more and more data providers to protect their data privacy and is often done as a part of the data pre- processocsigg as discussed in Iyengar (2002); Ram Mohan Rao et al. (2018). In such cases, a malicious data provider can release a poisoned pre-processed (anonymized) dataset and launch the attack on victim models trained with it.
E.2 FLD SCORE APPROXIMATES BCE LOSS
We evaluated how well the optimal BCE loss of logistic regression can be approximated by three FLD scores used our experiments: FLD, sFLD, and EUC. To this end, we train each victim for 50 epochs and compute the empirical optimal BCE loss of logistic regression to predict from representation after each epoch. Then we compare the trend of BCE loss versus FLD, sFLD, and EUC scores.
Figure visualizes the negative value of FLD, sFLD, and EUC and associated BCE losses of four victims trained Adult and German dataset with and without poisoning samples crafted by cor- responding ENG attack after each epoch. In cases, the three scores approximate how BCE loss changes very well.
E.3 EFFECTS ON DP VIOLATIONS
Here report increase of DP violations of victims attacked by three ENG-based attacks and AA- based baselines in Figure 5. As analyzed in Section 2, our attack successfully increased the DP violation significantly on most setting, clearly establishing their effectiveness.
E.4 EFFECTS ON ACCURACY
Here report change of accuracy of predicting y from representation learned by victims attacked by three ENG-based attacks and AA-based baselines in Figure 6. In general, all attacks had rela- tively subtle influence prediction accuracy, and our proposed ENG-based attacks often changed accuracy less. These results imply that poisoned representations are still of high utility, and it will be difficult for the victim trainer to identify the attack by checking the prediction performance alone.
E.5 PERFORMANCE OF ELASTIC-NET PENALTY
Here we report more experimental results on evaluating the performance of elastic-net penalty under varying hyper-paree parameters. Figure 7 and 9 reports how elastic-net penalty affects the attack perfor- mance, corresponding I, and L2 norms of learned perturbations are reported in Figure and 11, respectively.
We further compare elastic-net penalty versus L1 norm penalty for regulating the attack. Figure shows corresponding decrease of BCE loss and Figure 12 shows corresponding Lı and L2 norms. Compared with clastic-net penalty, penalizing L, norm only usually resulted in larger L1 and L2 norms while the attack performance was hardly improved, especially when small- to intermediate- level coefficient(s) (A₃ and X₂) of the regularizer is used.
Given the advantages of utilizing elastic-net penalty, one may ask if it be used in other attacks such as AA (Mehrabi et al., 2021) as well. Here we delve into the difficulty such an application, highlighting the suitability of elastic-net penalty for our attack by nature.
According to Mehrabi et al. (2021), both RAA and NRAA constructed poisoning samples from chosen anchor point by perturbing its randomly within 7-ball and flipping its y a. The
18

---

Published as a conference paper at ICLR 2024
Vis CIAR
Vitim CTMB-CO
Wedm IEWE-5
Vittm -n JOME-US
alsccc FuD attuch
Victm TOME
Wataac
Vi stadd: BCE Lo
Tuch
Do Vitim CIAI-EO
Ded Vitti ICWE-S
Qec
Wit. CIAR
Vittmm IOMEE-U
W. dllle LoE
We stted LUC
20
15
(a) Adult dataset
VC aaatioo REP
20
spech
Spach
Victim CRAIT
Victim CHAIR 8D
Victin ICWAS
Victim = I2VEE US
vis arta
Verien
Vettm
AUTEEC Lok, LoB
(b) German dataset
Figure 4: Changes of FLD, sFLD, and EUC loss (the negative score) and corresponding BCE loss. Results are averaged over 5 independent replications and bands show standard errors.
Wertin
CFAR
Vetm CFAR-DO
Vetim ICWAC-S
Vietim ICUNEASS
RAh-
2.22
MMyy
-3.3
8.1
0713
-312
0.0
31
035
0.5
0.5
2.05
535
00
0.5
3.
011
(a) Adult dataset
0003
0003
-0.00
Mayy
-2.08
0000
-0,00
2350
0107
2mm
CVC v112
+0.10
7.04 Fartic of Pobzaing Surplo
n/%
Tatoe of Pounting Sirplo
Pet Pooor Cample
Per of of Campiti
(b) German dataset
Figure 5: Increase of DP violations from different attackers using 5% 15% training samples for poisoning, Results are averaged over 5 independent replications and bands show standard errors.
19

---

Published as a conference paper at ICLR 2024
Velta - CHAIN
C741-80
Vidtan
ICWA US
0.010
0.150
03125
1150
LMG LSC
0042
L120
L357
E354
Parton hover Sanplo
Tutten Plooorag Samplo
Portoe Foozing Samplo
Phoceng Semolo
(a) Adult dataset
Victim =n CIMES
VxDmm CHSHt
VAtift J.VARSS
V.Dim R.00E-1
0.79
0.111
3739
0.32
1700
0.78-
0734
(5B PLD
3730
05
0.5
6.5
0.0
0.56
63
635
0.0
0.55
0.1
0.15
0732
00
065
03
0.55
(b) German dataset
Figure 6: Accuracy of predicting y from when different attackers using 5% 15% training samples for poisoning, Results are averaged over independent replications and bands show standard errors.
main difference lies in how they choose these anchor samples: RAA draws them uniformly from the training data. NRAA computes the pairwise distance between all training data and counts how many neighbors and chooses the training samples that have the most neighbors with flipped y as anchor data. Here two samples are neighboring if their and a are same and the difference in their æ is smaller than a pre-specified threshold.
is viable to define 7-ball based on the elastic-net norm. Nonetheless, the use of 7-ball will make NRAA harder to analyze in general. NRAA is thought stronger (which is empirically verified in both Mchrabi et al. (2021) and our experiment) because the chosen poisoning samples are likely to have higher influence as they have more neighbors. However, sampling from 7-ball will make some poisoning samples have more neighbors while others have less. In fact, the authors of AA used = 0 throughout their experiments and we adopted this setting. This difficulty remains unsolved when using elastic-net norm to induce the 7-ball. In conclusion, ENG, as perturbation-learning based attack, has a better affinity to the elastic-net penalty than AA baselines.
E.6 ATTACKING MULTI-CLASS SENSTTIVE FEATURE
Here we show empirical results of attacking victims trained on Adult dataset using race as the sensitive feature. Due to data imbalance issue we keep white and black group as they are. All other sensitive groups are re-categorized as others. This results in 3-way sensitive feature a and we measure I(2,a) a) by the decrease cross-entropy (CE) loss to predict a from z with a linear Softmax classifier. Note that both the CE loss and the averaged one-vs-all BCE loss (see Appendix for details) lower bounds I [(z, a), but we report CE loss for the sake of better interpretability. Due to the lack of official implementation of CFAIR and CFAIR-EG EO on multi-class sensitive feature, only attack ICVAE-US and ICVAE-S. As shown in Figure 13, ENG-based attack outperforms four AA baselines.
E.7 DEFENSE AGAINST ENG ATTACKS
Here we present reducing batch size as a defense strategy against ENG attacks. We only defend against successful attacks and focus on Adult dataset where attacks used ê₁ 0.0025, A2 0.005. We vary the batch size between 256 to 1024 and keep all other settings same as in Section 4. Resul- tant decrease of upper-level loss and BCE loss are reported in Figure 14. Reducing batch size successfully helped weaken the performance of attacking 3 out of victims in terms of -s.
Note that according to Theorem 3.4, it is conceptually viable to increasing learning rate to defend. However, using a large learning rate in a well-trained victim model may have side effect of making it diverge. Because of this, consider reducing batch size better and more practical choice.
20

---

Published as a conference paper at ICLR 2024
5320
37
0330
AEEEAAE 4,00
002
A.-00 w082
539
AL
0.00
0330
Portect Posaring Simplo
Patice of Nouring NVorinnSrrrtt
Petfonn
(a) ENG-FLD attack
Vitm CIRI
Vidims CH6R-50
Vidim JEVAR-S
Vetinn ICWBE-S
0604
0.35
0.30
0.30
0.02
9.0
0.32
1.44
0.31
00
0.5
011
635
00
3,5
0,1
20
0.05
0.1
035
00
0.5
31
035
(b) ENG-sFLD attack
0.35
#EAC22 4.0000
sE-Es 4:000
9.32
Ag mima0,, moor
Ag m24005. 1o000.4maa0 4ymas
0.34
0.11
0.30
039
Pc of Folooing Lampiis
Perdon oistiaiing Samrall
FSr100 of Fibiiiingg sangles
Pcrdon PfFSE Samp
(c) ENG-EUC attack
Figure Decrease of BCE loss achieved by different ENG-based attacks with varying hyper- parameters, victims are trained on Adult dataset. Results averaged over 5 independent repli- cations and bands show standard errors.
0357
1L-03325
3.34
0303
3.55
ALY521
0351
3.1
1.0
16/00 paluaning SPVEN
Perfor molaning Sindee
(a) ENG-FLD attack with only La penalty
Vittin CFAR
Vettm CINB-DO
Vitm IDWES
Veltm ICWAE-U
0.004
0.00
AL
2.0
0.3
0
0.0
3.55
0.1
1:5
05
C.8
035
00
0.5
0.1
315
00
3.55
82
615
(b) ENG-sFLD attack with only penalty
2.04
0677
0.0
0.22
040
240
Podker Pstermn Fandlo
of of Campit
Porder of Petsornn Sandila
Po0 of Sarcle
(c) ENG-EUC attack with only L1 penalty
Figure& Decrease of BCE loss achieved by different ENG-based attacks with varying hyper- parameters LI penalty and no La penalty, victims are trained on Adult dataset. Results are averaged over 5 independent replications and bands show standard errors.
21

---

Published as a conference paper at ICLR 2024
Vitim
Vidim
0005
0.110
4,
A
0000
0.000
200
300
Pot Samplo
Phooning Sentio
Notton Pcooiin Semplo
Neten pf Cemple
(a) ENG-FLD attack
MICI CHAH
Vider CHAR-80
Viduri KNAA-
ViDm K3A0--S
3.3125
A) 11-EAR2 Ay 000
A1-EAC5,
2204
0.100
332
4:0000
230
20
2.55
31
635
055
0.1
0.
0.55
0.1
635
0.55
3.1
0.5
(h) ENG-sFLD attack
A 4000
2100
6.4
A
2330
000
Pf Sinutte
Nut.8n Tcscring Sempls
Farter of
(c) ENG-EUC attack
Figure 9: Decrease of BCE loss achieved by different based attacks with varying hyper- parameters, victims trained on German dataset. Results are averaged over 5 independent repli- cations and bands show standard errors.
E.8 MORE EXPERIMENT RESULTS ON COMPAS AND DRUG CONSUMPTION DATASETS
Here we present more experimental results of attacking four FRL methods trained on COMPAS and Drug Consumption datasets. We adopted the same setting for data preprocessing and training- testing splitting pipelines as presented in Section 4 on Adult and German datasets. In specific, we report decrease of BCE losses in Figure 15, increase of DP violations in Figure 16, and accuracy of predicting from in Figure 17. Again, our attacks succeeded on all cases, outperforming AA baselines a large extent.
In terms of model architecture, we adopted the recommended architectures from Zhao et al. (2019) for CFAIR and CFAIR-FO on COMPAS dataset; and the default setting presented in Appendix on Drug Consumption datasets due to the lack of official implementations. For ICVAE-US and ICVAE-S, we used the default setting from Appendix D on both datasets because the lack of official implementations.
22

---

Published as a conference paper at ICLR 2024
Vet CAR
Wite D'ALLOO
Witti IEWA
Vatle IDMC-U
655
Campln
(a) ENG-FLD attack, norm
Vit CIAIE
C'AIEOO
Vetii ICWGGG
IDWGAS
A0-300
AL -1--31---632
655
215
(b) ENG-sFLD attack, L1 norm
Vis CIAE
Vidim-
Vitii ICWGGG
IDWGAS
A--
rotann chrllciing
ofrilaiing Jample
(c) ENG-EUC attack, L₁ norm
Vitter CANI
CAUR
1986
Mithin CWS us
Kanglin
(d) EFG-sFLD attack, L2 norm
Sanglen
ong
(c) EUG-EUC attack, L2 norm
Fradin wn
2005. 2-055
Petter Witoerle Sampin
Ceeee Pelerin Sundin
detenelliiein Surdies
Netto of Sambler
(I) ENG-sFLD attack, L2 norm
Figure 10: Averaged I, and La of perturbations learned by different ENG-based attacks with varying victims are trained on Adult dataset.
23

---

Published as a conference paper at ICLR 2024
Vit CAAR
Wite D'ALLOO
Wittin JEWM
Vatle IDME-U
6i5
Cample
(a) ENG-FLD attack, norm
Ves CIAIE
Vidim C'AREOO
Vetti ICWGG-
ICWAAS
A--300
655
(b) ENG-sFLD attack, L1 norm
Ves CIAEE
Vidim-
Witin ICWGG
IDWGUS
A--
Km 11.0-m
Fortin chreliiingggg
veriiiaing Jampli
(c) ENG-EUC attack, L₁ norm
Vitter CANI
1986
Within CWS
Karglin
Net
(d) ENG-sFLD attack, L2 norm
Sanggin
(c) EUG-EUC attack, L2 norm
Fradin
20050 10005
Pet1e Witaerie Samplnn
Foeiee Pelering Sundin
Leteenlloieeee Samries
NeTTO of Sambiee
(I) ENG-sFLD attack, L2 norm
Figure 11: Averaged L, and La of perturbations learned by different ENG-based attacks with varying victims are trained on German dataset.
24

---

Published as a conference paper at ICLR 2024
Vettm CIAB
CMA-5
Vitee FOREUS
0.5
1.13
X65
6.55
(a) ENG-FLD attack, norm
Victi
CMNC
Vilten ICKEES
1.-3
1,+300
815
025
21
11
265
6.5
(b) ENG-sFLD attack, L1 norm
Victin CINSS
CMCC
Vidten ICKEUS
15y roliring
615
rotie drrooooppppppi
(c) ENG-EUC attack, L₁ norm
CAUR
CANR
Vinti
Wittin LOWS Us
259
10%
Potare Sergln
Nel Sarglin
(d) ENG-sFLD attack, L2 norm
41-30
Potard
Ne1 farqlie
(c) ENG-EUC attack, L2 norm
23
74 7,+0005
Pot Wioeereg Samplnn
of Sendee
1911 denheeeessseeee
(I) ENG-sFLD attack, L2 norm
Figure 12: Averaged La and L2 norm of perturbations learned by different ENG-based attacks with varying byper-parameters on L1 penalty and no L2 penalty, victims are trained on Adult dataset.
25

---

Published as a conference paper at ICLR 2024
Victi ICVAE-S
Victim ICVAE-US
0.020
NESAA-
0.010
RAVra
0.010
0.003
Decresse
0.005
ENC-FLD
SFLD
0.000
-0.005
0.0
0.35
0.1
0.15
0.0
0.05
0.1
0.15
Portion Poisening samples
Portion of Poisening Samples
Figure 13: Decrease of CE loss from attacking victims trained Adult dataset with sensitive feature race.
1412-
1.5
1.25
V12
255
110
0.7
600
3.2
0.5
Hentutatin eadgrt
(a) Decrease of upper-level loss
Vieties CPAIR
Vittm CTAIR DO
Vietie ICWAE-
Victim ii ICVAD US
0325
111370
0:30
1324
3.0
0010
304
3000
302
0000
5000
2.0
6390
00
600
0,25
005
0.1
0.55
00
053
0003
8.55
31
0.5
00
0.11
0,25
C35
0.5
80
053
0002
625
0.5
(b) Decrease BCE loss
Figure 14: Effectiveness of reducing batch size defense against the proposed ENG-based attacks, different attackers using 1% 15% training samples for poisoning. Results are averaged over 5 independent replications and bands show standard errors.
Victin = CRAI
Vidtmm = EAAR-FO
Vclim ESAE-S
Vidim EVBE-US
0.50
5.35
0.02
-132
-0.10
EMB FLL
200
-0.15
0.
0.05
015
00
0,5
63
015
00
000
015
00
005
03
015
(a) COMPAS dataset
0.004
3,050
0.00
MAAy
04-1000
-2004
00005
010
DMG
0007
-2206
010
Partic of Polciing sungla
12-ken of Filten Sansles
PSrSer Follenng Sanudles
Necan of we Surpls
(b) Drug Consumption dataset
Figure 15: Decrease of BCE loss from different attackers using 5% 15% training samples for poisoning, Results are averaged over 5 independent replications and bands show standard errors.
26

---

Published as a conference paper at ICLR 2024
Victim C'A.R
Vittm CTAIS-BO
Victim ICWAE-S
Vicai ICWK
maze
DAA
1330
MAy
070
+0.30
DAC
0000
2.95
6.1
6.5
00
00
1.0
015
Poloning Semple
Pebooing Sampla
Parice of Foloring Semple
Porker Folaatoy
(a) COMPAS dataset
Wilti 1CM6
Vidmm
0.354
0.00
0399
0003
LAG LUC
0062
-0.00
0.000
Tatee of Sumplo
Pution Pooor Sample
Por Compity
(b) Drug Consumption dataset
Figure 16: Increase of DP violations from different attackers using 5% 15% training samples for poisoning, Results are averaged over 5 independent replications and bands show standard errors.
ICWW-S
5477%
0.100
EVD V1.
0,09
E358
Pation of Pouccing Sorgla
Futics Pooonng Sempha
Fotice of Foozinn Serplo
M12 of Ploooeg Serplr
(a) COMPAS dataset
Vitm en LIBES
Valum CHait
Y-le - 1.99 US
6711
3730
Wihk
0755
0.72
EM0
3756
6726
0.5
0.6
(b) Drug Consumption dataset
Figure 17: Accuracy of predicting from z when different attackers using 5% 15% training sam- ples for poisoning, Results are averaged over independent replications and bands show standard errors.
27


Reviews:
[{"review_id": "RuI9oMkD6z", "reviewer": "Reviewer_Ddgu", "summary": "The authors propose a poisoning attack strategy to compromise fair representation learning, aiming to increase the fairness gap for underprivileged groups. The attack relies on approximations to solve a bilevel optimization problem where the outer problem, which describes the attacker’s objective, aims to maximize the mutual information between the representation for the privileged and the underprivileged groups. Since the optimization of this objective is not tractable, the authors use Fisher’s Linear Discriminant (FLD) score as a proxy. Then, the whole bilevel optimization problem is approximated using a gradient matching strategy. The authors also provide some theoretical analysis on the poisoning ration required to compromise the target models.", "questions": "+ Equation (2) relies on strong assumptions about the distribution (Gaussian distribution and continuous variables) of the different subpopulations. How is this a good proxy for approximating the original problem? How does this compare with existing attacks (as the ones mentioned before)?\n\n+ Could the authors provide more details on the threat model (see comments above)?\n\n+ How assumption 2 works in the current threat model for the attack? The authors say: “Before the attack, the victim is well trained.” What does this mean?\n\n+ Also, for the theoretical analysis: Why do the authors think that assumption 3 is reasonable?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ Poisoning fair representations have received less attention in the research literature on data poisoning and preliminary works show that these attacks can have a significant impact on the fairness of the target algorithms. Exploring more scalable poisoning attack strategies capable of increasing the fairness gap for deep neural networks is timely and a topic of interest. \n\n+ The authors strived to provide a theoretical analysis on the ratio of poisoning points required to compromise the target algorithms.", "weaknesses": "+ The paper lacks a clear threat model. For example, it is unclear whether the attacker’s objective is useful for compromising algorithms with and without mechanisms for mitigating the fairness gap. On the other side, it is unclear what is the attacker’s objective and the relation of the attack strategy with respect to the model’s performance. Other works in the research literature, like Chang et al. or Van et al. (“Poisoning Attacks in Fair Machine Learning”) have already considered the trade-off between targeting performance and the fairness gap. \n\n+ In the end the attack proposed by the authors rely on the maximization of the FLD score in the outer optimization objective. This relies on strong assumptions on the distribution of the data and its representation and may not hold for many practical scenarios. In this sense, it is unclear why this strategy is better compared to other attacks already proposed in the research literature, like Solans et al., Mehrabi et al., Chang et al., or Van et al. (“Poisoning Attacks in Fair Machine Learning”), which have strong connections to this work. On the other side, there is not mention to “Subpopulation Poisoning Attacks” by Jagielski et al., which are also very relevant to this work and proposes more scalable alternatives for crafting poisoning attacks targeting fairness. \n\n+ Given the existing works in the research literature, I believe it is to bold that the authors claim that “We propose the first data poisoning attack on FRL as outlined in Figure 1.” I think that the authors should clarify this and position better their paper and contributions with respect to other existing works. \n\n+ The experimental evaluation is not convincing: In the experiments the authors just reported results evaluating the BCE loss but did not consider any of the existing metrics for measuring the fairness gap. On the other hand, there is not mention to how the attack affects the accuracy at all. Apart from that, it is necessary a more comprehensive comparison with other methods in the research literature, e.g., Solans et al., Chang et al. Van et al. (“Poisoning Attacks in Fair Machine Learning”), Jagileski et al. (“Subpopulation Poisoning Attacks”). For some of these attacks, the authors can use the same outer objective and use the same approximation for solving the bilevel optimization problem. \n\n+ The authors used a gradient matching strategy for approximating the solution of the bilevel optimization problem. However, Jagielski et al. (“Subpopulation Poisoning Attacks”) shows that other strategies can be more efficient for this. I think this aspect requires further analysis. \n\n+ The authors say: “Heuristics such as label flipping (Mehrabi et al., 2021) lack a direct connection to the attack goal, thereby having no success guarantee and often performing unsatisfactorily.” I think this is not true. Although more limited on the attacker’s capabilities, smart manipulation of the labels can lead to successful attacks. See for example “Subpopulation Poisoning Attacks” by Jagielski et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mGHrKFRt73", "reviewer": "Reviewer_yTSY", "summary": "This paper studies data poisoning attacks against fair representation learning (FRL) on deep neural networks. To achieve the attack goal, the authors propose a new MI (mutual information) maximization paradigm. Besides, the experiments show that the proposed attack outperforms baselines by a large margin and raises an alert of the vulnerability of existing FRL methods.", "questions": "Similar to weakness 1, could the authors clarify their threat model more clearly? Specifically, does the attacker need to know the victim’s trained model to generate poisoning data? If not, does the attacker only need to know the structure of the victim’s model and the model will be trained on the poisoning data from the attacker?\n\nSimilar to weakness 2, could the authors give more insight to explain why their attack cannot be mitigated by fair representation learning (FRL)?\n\nCould the proposed attack be applied to deeper neural networks? The experiments of this paper are just on two-hidden layer CNNs.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This is a pioneering data poisoning attack on deep learning-based fair representation learning to degrade fairness while the existing fairness attacks are focusing on shallow-model classifiers. \n\nThe authors propose a new attack goal based on MI to amplify the difference between representations from different subgroups. \nThe authors derive the first theoretical minimal number of poisoning samples required by their attack, which is crucial for practical attacks.", "weaknesses": "The assumption of the threat model is strong. The proposed attack is under the assumption of a white-box threat model, where the attacker has full access to and control over the victim's trained model. This implies that the attack is primarily effective in scenarios where the victim has already trained a model and relies on the attacker's data for subsequent fine-tuning. Such a specific condition might limit the general applicability of the attack in diverse real-world scenarios.\n\nLack the reason why the method can attack FRL. The foundational principle of Fair Representation Learning (FRL) is to ensure fairness by removing sensitive features from the intermediate representation. The proposed attack, on the other hand, seeks to amplify the presence of sensitive information within these representations. The paper does not adequately elucidate why FRL techniques, designed to minimize sensitivity, are unable to counteract or mitigate the effects of the proposed attack. This leaves a gap in understanding the inherent vulnerabilities of FRL against the described attack strategy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b6YSpQBdyq", "reviewer": "Reviewer_93wi", "summary": "This study proposes a novel data poisoning attack against fair representation learning algorithms. Compared to previous attack methods against fair classification, this method proposes to craft the training dataset, in order to maximize the mutual information between the learned representation and sensitive raw features. This mutual information powered attack algorithm shows superior attack performances compared to anchor attacks against 4 different fair representation learning methods.", "questions": "Discussion over the choice of the two penalty parameters $\\lambda_{1}$ and $\\lambda_2$ in the elastic penalty.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1/ This is the first research effort in organising data poisoning attacks against fair representation learning attacks. Different from fair classification problems, manipulating fair representation needs to control the statistical relation between high-dimensional embeddings  and raw feature inputs. This is challenging for directly extending previous fair learning poisoning methods. I'd appreciate the efforts poured towards this difficult problem. \n\n2/ It is intuitive to increase the mutual information between the embeddings and sensitive raw features, in order to violate the fairness constraint of the victim embedding learning algorithm. However directly maximising mutual information of high-dimensional embeddings is very difficult. I know there are some differentiable approximation tool to MI, like MINE. But it is computationally costly and prone to the potential estimation gap. It is interesting to read the theoretical analysis and practices of using Fisher Linear Discriminator scores to bound MI. Apparently, optimising FLD scores is much easier and economic in computation. \n\n3/ Inspired from Geiping et al's gradient matching work, this study propoes to matching the upper and lower bound of gradients instead of solving the bi-level poisoning problem. This smart optimization strategy enables an analytical solution to the proposed attack.", "weaknesses": "One of the problem of introducing elastic penalty is how to choose properly the two penalty parameters $\\lambda_{1}$ and $\\lambda_2$. Though it can be chosen empirically, it can be dataset-dependent. Would it make significantly difference if we simply choose the L1 norm penalty instead?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vJOjxtt2P0", "reviewer": "Reviewer_LCr6", "summary": "This work studies an interesting topic i.e. how to conduct data poisoning against fair representation learning tasks. Experiments are conducted on Adult and German datasets to demonstrate its effectiveness.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Motivation is well-stated and interesting.\n- Authors develop related  theoretical analysis on the needed number of poisoning samples is derived and shed light on defending against the attack.\n- Personally I like the organization of Introduction section : ) It's clear and easy for reviewers to know the meaning of this work.", "weaknesses": "- Authors use their own defined vanilla metric, and lack related fairness-aware metrics like Equality odds (EO)\n- Authors are encouraged to conduct more experiments on more datasets like COMPAS and Drug Comsumptionm, please kindly follow this AAAI paper which authors have cited: Exacerbating Algorithmic Bias through Fairness Attacks.\n- Personally, I reckon authors are encouraged to conduct experiments on deeper NN (I think simple MLP is not that DEEP to be called \"DNN\"), though the datasets are relatively simple. I'm curious about these experiments to investigate ENG. Authors are encouraged to conduct more analysis on the further version of this work, which is good for community: )", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WOqFVf89gL", "reviewer": "Reviewer_LzqR", "summary": "This paper designs a new data poisoning attack tailored for fair representation learning. The key idea is to generate poisoning samples that maximizes the mutual information between the representation (of the poisoned sample) and the sensitive attribute. This can be solved by bi-level optimization, yet the outer level problem (estimating high-dimensional mutual information) is intractable. To solve this issue, the authors propose to use Fisher’s linear discriminant (FLD) score as a cheap proxy to MI, which has a closed-form solution. For inner level problem, it is solved by matching the gradients of a victim model and a clean model. Through the approximations made, the original objective is now fully tractable and can be learned by SGD. The effectiveness of the method is evaluated on two tabular datasets.", "questions": "How does the proposed FLD-based method scale with the dimensionality of representation d? I am concerned about this since Gaussianity assumption typically violates in high-dimensional cases. Will the method still work well for e.g. d=128?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- **Significance**: The problem studied in this paper (data poisoning to attack the learned representation in fairly trained models) is interesting and important;\n- **New information-theoretic viewpoint for poisoning attack**: The authors propose an information-theoretic objective for poisoning attack in fair machine learning. This framework, to the best of my knowledge, is new within the specific context considered and is very-well motivated. The reminder on the advantage of MI-based fairness as compared to conventional metric (e.g. DP) is also useful;\n- **Cheap proxy to mutual information**: I also like the authors’ idea to use Fisher’s linear discriminant analysis (FLD) as a cheap-but-still-effective proxy to MI. I would also highly praise the authors’ efforts to mention the possibilities of other analytic proxies, as well as a discussing between FLD and these alternative methods. \n- **Feature selection for fairness**: in addition to the main contribution, the authors also show how their developed attack can be further applied to identify robust features for fair classification. This is quite interesting, especially in that it offers a new perspective for understanding and interpreting the behavior of a fairly trained model. I personally think this part deserves a separate section and can be highlighted as a 2nd main contribution of the paper.", "weaknesses": "- **On the necessity of working at representation level**. After reading the paper, it is still unclear to me why do we need to consider I(Z; a) rather than I(Y(X); a). Here Y(X) is the prediction of the model. In fact, considering I(Y(X); a) has several benefits: first, its maximization still yields an unfair model; second, its estimation is much easier due to the low-dimensionality of Y(X) and a (for example, we can estimate it easily by the well-known KSG estimator [1] which typically works quite well in low-dimensional cases). Importantly, maximizing I(Y(X); a) also do not require an access to Y. Can the authors justify the reason behind considering I(Z; a) instead?\n\n- **Issues in the discussion related to other MI proxies**: some comments in remark 2.1 do not seem completely sensible to me. For example, the author mention that other analytical proxies like (K)CCA and sliced mutual information suffer from differentiation difficulties. This may not be true in my view (for example, in CCA, you can first solve the optimal weights analytically, then substitute it back to the formula of CCA. The resultant formula has a structure very similar to eq2 in your paper). In addition, I think the authors miss the possibility of using non-parametric dependence measure e.g. distance correlation (dCorr) [2]. This is also fully analytical and may potentially be applicable in your scenario. Ultimately, I think the real advantage of your FLD-based method is it provably optimises a variational lower bound of MI, whereas other proxies (KCCA, slice MI, dCorr) may not. This seems to be a better justification of the use of your method. \n\n- **Slightly limited evaluation**: most of the evaluations in this work are conducted on tabular data where the network size is small. Whether the method developed in this work will scale to larger networks e.g. those in computer vision and NLP remain unclear to me. However, given the nature of many existing literature in fairness research, which also only focus on tabular data, this should not be a main criticism for the paper.\n\n[1] Estimating mutual information, Physical Review E, 2004\n\n[2] Measuring and testing dependence by correlation of distances, Annals of Statistics, 2007\n\n----------------------------------------------\nUpdate after rebuttal:\n\nThe authors have addressed most of my concern above and I have updated my score to 8 accordingly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)