Dr-Fairness: Dynamic Data Ratio Adjustment for Fair Training on Real and Generated Data
Authors: Yuji Roh
Weili Nie
De-An Huang
Steven Euijong Whang
Arash Vahdat
Anima Anandkumar
Venue: Submitted to ICLR 2023 (2023)
Keywords: trustworthy AI
fairness
visual recognition
generated data

Abstract: Fair visual recognition has become critical for preventing demographic disparity. A major cause of model unfairness is the imbalanced representation of different groups in training data. Recently, several works aim to alleviate this issue using generated data. However, these approaches often use generated data to obtain similar amounts of data across groups, which is not optimal for achieving high fairness due to different learning difficulties and generated data qualities across groups. To address this issue, we propose a novel adaptive sampling approach that leverages both real and generated data for fairness. We design a bilevel optimization that finds the optimal data sampling ratios among groups and between real and generated data while training a model. The ratios are dynamically adjusted considering both the model's accuracy as well as its fairness. To efficiently solve our non-convex bilevel optimization, we propose a simple approximation to the solution given by the implicit function theorem. Extensive experiments show that our framework achieves state-of-the-art fairness and accuracy on the CelebA and ImageNet People Subtree datasets. We also observe that our method adaptively relies less on the generated data when it has poor quality.

Under review as a conference paper at ICLR 2023
DR-FAIRNESS: DYNAMIC DATA RATIO ADJUSTMENT FOR FAIR TRAINING ON REAL AND GENERATED DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Fair visual recognition has become critical for preventing demographic disparity. A major cause of model unfairness is the imbalanced representation of different groups in training data. Recently, several works aim to alleviate this issue using generated data. However, these approaches often use generated data to obtain similar amounts of data across groups, which is not optimal for achieving high fairness due to different learning difficulties and generated data qualities across groups. To address this issue, we propose novel adaptive sampling approach that leverages both real and generated data for faireess. We design a bilevel optimization that finds the optimal data sampling ratios among groups and between real and generated data while training a model. The ratios are dynamically adjusted considering both the model's s accuracy as well as its fairness. To efficiently solve our non-convex bilevel optimization, we propose a simple approximation to the solution given by the implicit function theorem. Extensive experiments show that our framework achieves state-of-ther thetarrt fairness and accuracy on the CelebA and ImageNet People Subtree datasets. We also observe that our method adaptively relies less on the generated data when it has poor quality.
INTRODUCTION
Model fairess in visual recognition is becoming essential to prevent discriminatory predictions over demographics. Recently, numerous unfairness issues have been reported (Wang et al. 2020 Najibi 2020), and several fair image classification approaches have been proposed that do not discriminate against specific groups such as gender, age, or skin color (Ramaswamy al. 2021 Roh et 2021)
With the rapid progress in deep generative learning (Karras et al. 2020 Dhariwal & Nichol 2021).
there is a new research direction improve fairness by augmenting training data with generated data. Recent breakthroughs in generative learning make generated data practical enough to use in real-world applications [ipenAT[202)) and many high-quality pre-trained generative models are now open the public (Rombach al. [2022), which obviates the need to train such models ourselves. Thus, generated data increasingly used to improve model performances, including fairness. From a fairness perspective, generated data complements real data by making it more diverse. For example, a specific group's 's data collected from a limited data source that does not have the full data distribution, that group may be discriminated in model training due to the bias (Mehrabi et al. 2021. In this case, generated data can be used to supplement that underrepresented group.
However, most fair training approaches that use generated data simply generate similar amounts of data across groups (Ramaswamy et al.] 2021 Choi et al. 2022)), which may not be optimal to improve group fairness such equalized odds (Hardt al. (016) and demographic parity (Feldman et al. 2015). Such suboptimality could originate from I) the learning difficulty differences across groups and 2) the potential bias (i.e., typically in the form of missing modes) and quality issues the generated data that can hurt the accuracy and fairness of the model under training. Therefore, it is essential to find the right mix of generated and real data for the best accuracy and fairness.
In this paper, we harness the potential of both real and generated data via adaptive sampling to improve group fairness while minimizing accuracy degradation. To this end, we design a new sampling approach called Dr- Dr-Fairnsss (Dynamic Data Ratio Adjustment for Fairness) that adaptively adjusts data ratios among groups and between real and generated data over iterations, as in Figure Ta

---

Under review as a conference paper at ICLR 2023
Real Gen. data data
Group
Update the data ratios
Accuracy
88.0
Fairness metric
85.0
Classification
EO Disp.
Training data
Dr-Fairness
Model
0.10
Adaptive sampling
0.00
In each iteration
1:1 ratio FairBatch Ours (b) Accuracy and fairness performances.
Original
(a) Framework overview.
Figure I: (a) Our framework iteratively updates the data ratios among groups and between real and generated data based the fairess and accuracy of the intermediate model. (b) Performances CelebA, using gender as the group attribute and age as the label attribute. Compared
original model, the 1:1 ratio baseline (Ramaswamy et al. (2021) does not significantly improve group fairness, measured through equalized odds (EO) disparity. FairBatch (Roh et al. (021) shows high fairness by adaptively selecting real data only, but loses accuracy. In comparison, Dr-Fairness (ours) achieves high fairness, while not sacrificing accuracy.
Table Functionality comparison of algorithms.
In Table 1 we compare the unique properties of
Dr-Fairness against two representative methods:
Uses generated
Finds
Utilizes
1) an equal ratio baseline (1:1 ratio) (Ramaswamy
Method
the optimal
the optimal accumey
real & gen. for ratio
al., 2021) that uses generated data and 2) a
data
group ratio
data updates
farress-aware adaptive sampling baseline (Fair-
ratio
Batch) (Roh et al., 2021) that finds the optimal group ratio for fairness only using real data. We can see that Dr-Faimess subsumes the two baselines and improves them by also optimizing the ratio between real and generated data and utilizing accuracy for ratio updates.
FairBatch Dr-Fairness
x
/
To perform adaptive sampling systematically, we design a novel bilevel optimization problem along with an efficient algorithm for solving it. Our bilevel optimization consists of 1) an outer optimization that adjusts data sampling ratios considering both fairness and accuracy and 2) an inner optimization that minimizes the standard empirical risk both real and generated data, given the current sampling ratios. Although various exact algorithms have been proposed to solve bilevel optimizations (Maclaurin al.) 2015), they often scale poorly in our scenario with large models and data. We thus propose an approximate algorithm that uses the implicit function theorem (Krantz & approximation (Luketina et al., (016) to efficiently compute the gradient of our bilevel optimization. Specifically, instead of computing the expensive inverse Hessian matrix, we approximate it with a simple diagonal identity matrix.
Parks, 2002) and identity-
Experiments on CelebA (Liu et al. 2015) and ImageNet People Subtree (Yang et al., 2020) show that our approach achieves the -fate-of-the-ee art fairness and accuracy performances. For instance, Figure highlights our results on CelebA, where our framework largely outperforms FairBatch, which only uses real data and the 1:1 ratio baseline see Sec. 3 for comparisons using more baselines and other fairness metrics, which show consistent results. On the ImageNet People Subtree classification problem, which represents large-scale real- world scenario, we achieve better accuracies than the best baseline, with an absolute improvement of 5-9%, while obtaining similar fairness scores. We also observe that our framework adaptively relies less on the generated data when it has poor quality.
Summary of Contributions: (1) We propose Dr-Fairness, novel adaptive sampling framework for fair training that enjoys the potential of both real and generated data. (2) To perform adaptive sampling systematically,
formulate bilevel optimization to train fair and accurate models real and generated data. (3) We also design an approximate algorithm based on the implicit function theorem and identity-matrix approximation efficiently solve our optimization. extensive experiments on CelebA and ImageNet People Subtree to show that Dr-Fairness achieves the state-of- ofte-offthe-ar accuracy and faimesss. (5) Finally, we believe that our work reveals the importance of using generated data together with real data to improve model fairness.
(4) We perform
2 FRAMEWORK
In this section, we first formulate bilevel optimization problem for optimizing sampling ratios for real and generated data. We then design a new algorithm that efficiently solves the optimization problem. Throughout this paper, we use the following notations and fairness definitions.
Notations Let x E X be the input feature, and let y E Y and y E be the true label and the predicted label, respectively. Let z Z be sensitive group attribute, e.g., gender, age, skin
2

---

Under review as a conference paper at ICLR 2023
color. Let m be the total number of training samples, and my. be the number of samples in the set {i y; = y, 4 = =2} with label y and group label z. Similarly, My, := I2 : y, = yHl and m,. == [i : Z, = 2||. Let w be the model weights, and the overall empirical risk is given by L(w)= = :n E. E(y. 8z), where E(- represents the loss function. Let Lyy: be the empirical risk over samples in the set (i y, = y, 2 =}. i.e. (w)) my.a -ix,= ((y,, 82). Finally, let (.) and (.) be the empirical risks on real data and generated data, respectively.
Fairness Definitions For the method design, we focus on two prominent group fairness defi- nitions: equalized odds (EO) (Hardt et al. 2016) and demographic parity (DP) (Feldman et al.) 2015). EO is satisfied when the accuracies conditioned on the true label are the same for differ- ent groups (i.e., Pr(ý = yly = y, 7. 21) = Pr(ý = yly = y, z = 22), Vy E Y, 2 22 E Z). DP is satisfied when the positive prediction rates are the same for the groups (i.e., Pr(S 1/z 21) = Pr(ý- = 1|2 = z2). ). Yz1, E 2), where DP is designed for binary classifications (i.e., E (0, with favorable label class (e.g.. "approval" in loan decision).
Generated Data In general, any synthetic data, including data from deep generative models, can be considered generated data for fair training. Here, the key role of the generated data in algorithmic fairness is supporting the limited subset of the real data. Also, we implicitly assume that the domains of the generated data and real data are the same, but the distributions of the two data can be different. For example, if the real data represents human faces, then we assume the generated data also contains human faces. However, generated data may have fairer distribution than real data. In this paper, we assume we can get group-specific generated data by using conditional image generation techniques (Nie et al. 202 Dhariwal & Nichol 1021 see details in Secs. and B.3
2.1 BILEVEL OPTIMIZATION FOR FAIRNESS WITH REAL AND GENERATED DATA
To design an adaptive sampling strategy on real and generated data, we first formulate a bilevel optimization for training fair and accurate models. The bilevel optimization consists of inner and outer objectives: 1) we maintain the standard empirical risk minimization (ERM) in the inner problem, and 2) we capture the desired fairness properties in the outer problem. The bilevel formulation allows us to support prominent group fairness metrics and utilize generated data for fairness. Moreover, through this formulation, we can achieve the desired fairness properties while keeping the standard model training process without re-configuring the model architecture or loss functions. We discuss more advantages of using bilevel optimization compared to other problem formulation methods like distributionally robust optimization (Sinha et al. 2017 in Sec. A.6
We now explain how our optimization improves group fairness and accuracy together by using both real and generated data. The outer objective aims find the optimal data ratios among sensitive groups and between real and generated data to minimize the fairness and accuracy losses on the real data distribution. Given the current data ratios, the inner objective runs a weighted ERM with both real and generated data. We can support various prominent group fairness metrics by modifying the outer objective and the constraints. As an illustration, we state our bilevel optimization w.r.t. BO as follows (see the DP version in Sec. I.
min 7.p
max =2
{
(w(A, 1)) real (w(A, x)) + Σyer, my.e L (w(X, 14), m
My. w (A, 11) = arg min Σ ZyEH,EEZ My, Xy.z L.al (w) + (1 -M1-My. L
10
s.t. A E 10
= E V
where Ay.s is the ratio for group = in class y, My,z is the ratio for real data in the (y, 2)-class, and A and μ are the sets of all Ay,z and My,., respectively. In the outer objective, the first term indicates the fairness loss, and second term indicates accuracy loss. The hyperparameter & tunes the importance of the two losses. We note that the Xy.s and Myys values are the data ratios within one mini-bater Thus, among all samples in the real and generated data, our framework serves bini-batches according to Xy. and My.. Here, we can capture the EO disparity as the maximum of the loss differences in see the details in Sec. A.2
different groups within the same label (i.e., max [ (w) - - (w) )
Through the above formulation, the amount of generated data is automatically adjusted to augment the real data. One problem with only using real data is that there can be an accuracy degradation of the model due to over-sampling minority groups for better fairness. Generating data for these groups can prevent the model's overfitting and lessen the accuracy degradation.
3

---

Under review as a conference paper at ICLR 2023
2.2 ALGORITHM
We now design our algorithm to solve the above bilevel optimization. In this section, we first describe how to efficiently approximate our optimization by utilizing the implicit function theorem (Krantz & Parks 2002) and adapting identity-matie approximation (Luketina et al. 2016) in fairesss setting. We then introduce the overall training procedure, and show the validity of our approximate algorithm on synthetic data.
Algorithm Design Solving bilevel optimization is known to be challenging (Liu et al. 2021] especially when the objectives are non-convex as in our problem. Thus, we resort to stochastic gradient descent to find the optimal parameters of the bilevel optimization gradually. To obtain the gradients, we first convert our optimization into the unconstrained version:
(ILg (w(), L. (w ())I + E my L (w (X, p1),
min max A, 2E
m
w (A, u)) arg mrg Σ EyEY, m
my.+ ogy (y,,)) ( (w) + (1 S(p ) ( (w)),
w
where (y(2) == exp(Ay,))/ cxp(Ay,2.) Cx.pAAA (i.e., the softmax function), and S(n,,)) 1/(1 + exp(-1ly.2) (i.e., the sigmoid function). Denoting the outer and inner objectives as fouter(A, (A, fi, w(), 14) and finne((, n, w) respectively, the inner optimization w(d, u)) arg min finner (A, ft, w) can be solved efficiently using SGD-like algorithms. The main ques- tion is how to solve the outer optimization. We can state the gradient of fauter W.I.t. À as follows:
dfmater
Souter
Somtee
Oer 11)
dA
OA
(w..A p))
0A
Term
Term
Term C
where Term A and Term are the direct gradients w.r.t. and w( 14), respectively, and Term is the best-response Jacobian. Note that w(A, X, 41) is the best-response of model weights.
In Eq. [ the best-response Jacobian is hard to directly compute. Although various algorithms have been proposed to explicitly find the best-response Jacobian, most of them require propagating the entire history of the gradients (Maclaurin et al.) 2013), which is very time-consuming. Instead, we implicitly measure the best-responss Jacobian using the implicit function theorem (Krantz & Parks 2002). This approach does not need to investigate the entire gradient history (Rajeswaran et al. 20119 Lorraine et al., 2020) and builds on the assumption that the inner optimization has converged to local minimum, i.e., fime O. Using this assumption, we can convert the best-response Jacobian dw
into the multiplication of two matrices (see more details in Corollary of Sec. AA.)
dfouter i fouler
① ffuter
8 funer 1-1 8² finer
X -1 dwor
Owl(,, 11)
OwoX
However, obtaining the inverse Hessian (i.e., [8? funer/ funer/dwrrrt @wwww) -1) in Eq. 2 is also computationally expensive. Thus, we consider the identity matrix approximation LLkeetina et al. 2016 Finn et al.) 2017 Geng al. 2021) that replaces the inverse Hessian with the identity matrix. Despite its simplicity, such approximation may be valid for neural networks with normalization layers that make the Hessian matrix diagonally dominant (c.g. BatchNorm), and in practice, it often performs on par with other approximation methods in various applications (Raiko et al.) 2012) Pedregosa 2016 Liu
al. 2018 Wilder et al., 2019; Fung et al. (2022). Given this, we can rewrite Eq. 2 simply l as:
dfouter dX Ox aw 11) Owox
Joouter
Of 82 fineer
where the second term on the right-hand side is efficiently computed via vector-Jacobian prod- uct Passke et al. 2017). Similarly, we can also approximate the gradient of fouter w.r.t. μ..
Overall Training Process We now describe the overall training process in Algo. [ We first initialize the model parameters and the data ratios, and for each iteration, we then get minibatch from Dr-Fairness (Algo. D In Algo. 2
we first update the data ratios among groups (A) and between
real and generated data (ạ1) by calculating dfasz
dx and daaz
dj as in Eq. We then draw a minibatch
according to o,y(A) (A) and S(u). Note that the batch sampling with a (A) and S(u) provides an unbiased estimator of the weighted ERM in our inner optimization (Roh al., (02I). Finally, we
update the model parameters w with the given minibatch. Here we can optionally use an exponential moving average (EMA) that averages the model parameters u for improving training stability.
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Model Training with Dr-Fairness
Algorithm 2: Dr-Fairness
Input: real data (Freal, Ureal, Preal), generated data (Zgenn Ugen: 2reat)
Input:
model parameters w, data dreal and dgen: group ratio X, real data ratio μ
dreal, dggen + (Zarain Mrminn Zreal), ), (Iigen: Mlgen; Freal)
w f- initial model parameters
Calculate fouter and finner according to w. Greal, dgen, X, and μ
X. f < initialize sampling ratio logits
Get current sampling ratios oy (A), S(
Get Muaa and dean as in Eq. du
for each iteration do
minibatch=Drr- Fetx-Faarneee rness w. dreal: dgens X, μi) Update according to the minibatch (optionally with exponential moving average (EMA))
Update X by ofun and fe by dfauter
dA using optimizers (e.g., Adam) Draw minibatch w.r.t. (y(A), S(p) Output : minibbtch
des
Output :ooel parameters w
Validity of Our Algorithm We empirically verify how close the solutions from our approximation strategy are to the optimal ones. To this end, we follow the synthetic binary setting in Roh et al. (202T), where FairBatch has a theoretical guarantee to find the optimal group ratios, and compare the optimized group ratios of Dr-Fairness and FairBatch- see details on the setup in Sec. A.4 Note that in this synthetic setting, we set the fairness metric to equal opportunity (i.e., a relaxed version of EO that focuses on the positive label) and only use the real data to optimize the group ratios X, as FairBatch cannot handle I for the generated data. Idcally, if the approximation error in our method is small, Dr-Fairness should obtain the same group ratios and performance FairBatch.
Figure 2 shows that our algorithm converges to similar group ratios to those in FairBatch, although the key ıdeas of the two algorithms updating the group ratio very different. Also, the two algorithms have the similar faimess scores (0.012 equal opportunity disparity for both). These results imply that our approximations are good enough to find reasonable solutions, which consistent with the observations in other applications (Lorraine et al. 2020 I Luketina et al. 2016). In the next section, we will show that Dr-Fairness achieves much higher accuracy with similar or better fairness than FairBatch on real-world datasets, as our algorithm scales to multiple groups and labels and is capable of harnessing the potential of both real and generated data. We note that although FairBatch is known to have theoretical guarantees, they only apply to limited settings (e.g., binary groups and labels), so there is room improve fairness in other settings.
We also verify that, in the above setting, the identity matrix approximation indeed gets almost the same group ratios with the exact inverse Hessian computation. See more details in Sec. A.5
Group Ratio
1.0
Group Ratio
0.761
0.5
2
0.5
1,0
0.239
0,227
0.0
0.0
100
200
300
400
100
200
300
Iterations
Iterations
(b) Group ratios in FairBatch. Figure 2: Comparison of group ratios A from Dr-Fairness and FairBatch. Both converge similar ratios.
(a) Group ratios Dr-Fairness.
3
EXPERIMENTS
We perform various experiments to evaluate our algorithm. We repeat all experiments with three random seeds and measure all performances on separate test set - see more information, including hyperparameter details of the algorithms, in Sec. [..l We ResNer50 (He al. 2016) and the
Adam optimizer (Kingma & Ba 2015) see results on ViT (Dosovitskiy et al.) 2021 in Sec.
Datasets We utilize two real-world datasets: 1) CelebA (Liu et al., 2015) to compare our algorithms with baselines and perform various analyses, and 2) ImageNet People Subtree (Yang et al. 2020) to further observe the algorithm performances on a large-scale real-world scenario. Note that we are using large datasets instead of the traditional smaller tabular benchmarks for fairness because our goal is to make Dr-Fairness work in large-scale real-world applications.
[CelebA] Contains celebrity images, where each image has 40 attributes (e.g., gender, age, and smiling). We choose group and label attributes that are less subjective and traditionally considered for faimess.. The group attributes are gender (male and female) and age (young and old). The label attributes are age, ha rcolor or (black, blond, and others), and smi 1 ng (smiling and not-smiling).
5

---

Under review as a conference paper at ICLR 2023
Note that age can be used as either the group or label attribute. The sizes of the training, validation, and test sets are 160k, 20k, and 20k, respectively.
[IaggNNrr People Subtree] Contains 284 label classes and 3 group attributes: gender (male, female, and unsure), ski i color (light, medium, and dark), and age (child, adult, middle, and retired). We first filter out classes that are vague, duplicates, or too small with few samples, which leaves us with 112 classes see details in Sec. [8.2] These classes contain about 111k samples, but only 10% of them have group attribute annotations. We split the group-labeled data into 40%, 20%, and 40% for training, validation, and testing, respectively.
Data generation We create the generated datasets using state-of-the- artt generative models that conditionally synthesize images for each (y, 2l-clss. For CelebA, we use a StyleGAN-based controllable generation method called LACE (Nie et al. (021). For ImageNet People Subtree, we fine-tune a diffusion model (Dhariwal & Nichol 2021) pre-trained on ImageNet (Deng et al. 2200)) and use classifier guidance (Song et al. [0200] Dhariwal & Nichol 2021) to sample images cach (y, 2))-class Note that the controllable generation for ImageNet People Subtree is more challenging due to its large number of (y, 2)-classes and labeling noises. Thus, the resulting generated data has lower quality than the generated data in CelebA. More details on data generation are in Sec. B.3
Baselines We compare our algorithm with three types of baselines: 1) vanilla (non-fair) baseline, 2) fair pre-procss baselines, and 3) fair in- in-processing baselines.
simple sampling, pair-angmenting (PairAug) (Ramaswamy et al. 2021), and pair-angmenting with our generated data (PairAug*) *) For simple sampling, we over- and under-sample the real data ensure an equal ratio among groups. PairAug is fair augmentation technique that uses the generation methods to synthesize balanced images for groups to reduce the correlation between the group and label attributes. For a fair comparison, we also implement an extension of PairAug (denoted by PairAug*), which uses the same balaneing ratio in [Ramaswamy et al. (02T), but uses our generated data.
For fair pre-processing training, we consider three baselines:
For fair in- in-processing training, we consider three baselines: fairness constraint (Zafar et al. 2017a b). domain independence (Wang et al. 2020), and FairBatch (Roh et al. (02I). Fairness constraint adds faimesss penalty term to the loss function to reduce the unfairness. Domain independence trains separate classifiers per each group to reduce the correlation between the group and label attributes. At inference, one can ensemble the outputs of the trained classifiers to get the final predictions. FairBatch adaptively adjusts batch ratios among groups to improve fairness only using real data.
Metries We focus on two accuracy metrics and three fairness metrics. [Accuracy] We measure the standard accuracy over all samples and the balanced accuracy that averages -ellssswise accuracies. [Fairness] We focus on equalized odds (EO) (Hardt et al. 2016). demographic parity (DP) (Feldman let al. 2015), and bias amplification (Zhao et 2017) For EO and DP, we measure the disparities (i.e., unfairness) among groups: EO disp. = max= |Pr(ý=y|z=2, y=y) Pr(ý-yly=y),, and DP disp.= m
Together with either EO or DP, we mea- sure bias amplification to see how much the data bias is amplified in the model: Bias amp. max Pr(z=2lý=y) - Pr(z=2ly=y), where & : arg max Pr(z=z'lý=g). Here, a good performance is indicated by high accuracy values and low EO disp., DP disp., and bias amp. values.
3.1
CELEBA EXPERIMENTS
We evaluate Dr-Fairness on CelebA by comparing it with baselines (Sec. B.I.II and analyzing the impact of its hyperparameters (Sec. 3.1.2). components (Sec. (.1.3), and generated data (Sec. 3.1.4..
3.
ACCURACY AND FAIRNESS
Table 2
shows the accuracy and fairness performances of different algorithms CelebA when training w.r.t. EO (see results of training w.r.t. DP in Sec. B.5). Here we consider two scenarios: 1) binary setting of y (age) & z (gender) and 2) non-binary setting of y (haircolor) & z (gender, age). In Sec. [.6] we show similar results for the experiments on other group and label combinations. Also,
in Sec. B.7 we visually demonstrate the couurac--fiimess tradeoffs of the algorithms.
The fair pre- pre-processing baselines (in rows 2-4) improve the fairness performances compared to the original foirfair baseline, but still perform worse (i.e., lower EO disp. and lower bias amp.) than the fair in processing baselines and Dr-Fainness. Thus, simply equalizing the data ratio among groups may not be enough achieve high group fairness. Note that it is not straightforward to get the

---

Under review as a conference paper at ICLR 2023
Table 2: Performances the CelebA test set when training w.r.t. EO scenarios: binary (age) & (gender) and non-binary y (haircolor) z (gender, age). We compare Dr-Fairness with three types baselines: 1) non-fair baseline. 2) fair pre-processing baselines: Simple Sampling, PairAug, and PairAug*, and 3) fair in-processing baselines: Fair. Const., Dom. Indep., and FairBatch. Note that PairAug and Fair. Const. cannot be trivially extended to the non-binary label setting, so only show their results in the first column.
y: age z: gender
y: haircolor and x: (gender, age)
Method
Acc.
Bal. Acc. EO Disp.
Bias Amp.
Acc.
Bal. Acc. BO Disp. Bias Amp.
Non-fair
86,4+03
76,4+02
0.173+axe
0.101+
83.8+aa
82.0+ea
0.535 +anm 0.014+300
Simple Sampling
86.8+02
78.3+05
0.132+300p
0.052+3017
83.2+aa
80,9
+ 0.026=300
PairAug (Ramaswamy et al. 2021
85.6106
79.8101
0.124
0.030gam
PairAug Ramaswany ot 2021
86.7 79.3
0.134
go1s
83.0kea
80.
2.406 0.022
Fair. Const. Zafar et 20176
86,8+u2 79.6+0s
0.106
0,028+ami
Dom, Indep. [ang
85.0 72.4121
0.070
0.034
81.7
74.8
z.340 0.026:ams
FairBatch (Roh et al. 2021
84.7
72.5+i
0.023
-0,043=
79.4
68.2+47
a.096
0.033+300
Dr-Fairness
87.7
81.0saa
0.020
-0.026=am03
85.0
84,4
0.079
0.012
Matthatth
Accuracy
8000
EO Disp.
EO Disp
Dr-Eatrmss
2000
0.1
0,2
0.2
0.1
6000
0.0
500
1000
1500
2.00
300
10:0
1500
2000
780
80,0
$2.0
81.
96.0
Herations
Iterations
Accuracy
(a) Accuracy when varying k.
(b) Fairness when varying
(c) Tradeoff curves.
Figure 3: The performances of Dr- Fa-immssss by varying the hyperparameter &s when (y, z) (haircolor, (gender, age)). The first two graphs show the performance changes during the training, and the last graph shows the accuracy-fairnes. tradcoff curves. Compared to FairBatch, Dr- Dr-Fairness shows a better tradcoff.
generated data from the original PairAug work in the non-binary label setting, so are not able to report the numbers (e.g., the right columns of 2). But we expect that the results would be similar to PairAug*, as observed in the binary setting. Additionally, FairnessGAN (Sattigeri et al.] 2019) is another previous method that aims to generate fair images, but this method has been reported to show worse fairness and accuracy performances than PairAug- see Sec. Scc.B.4 for a detailed comparison.
The fair in-processing baselines (in rows 5-7) improve fairness (esp. EO), but tend to sacrifice accuracy because they only utilize real data. Here, in the baselines with higher fairness, the decrease in accuracy becomes more significant. For example, FairBatch adaptively adjusts the group ratio on real data to improve fairness, but we observe that some small-sized groups end up being oversampled, which is detrimental to the accuracy performance on the test set.
In comparison, Dr-Fairness achieves high fairness performances while even improving accuracies by adaptively finding optimal data ratios among groups and between real and generated data. There are two takeaways: 1) we can find better group ratio than the 1:1 ratio for fairness, and 2) an optimal combination of real and generated data can mitigate the accuracy degradation of fair training.
3.1.2 HYPERPARAMETER ANALYSIS
We now evaluate Dr- Fairness by varying its main hyperparameter & used in the bilevel optimization. A larger l puts more weight on the accuracy loss than the fairness loss. Figures Ba and [3] show the accuracy and faimess of Dr-Fairnes, during the training with the different & values. As expected, increasing k (say A = 50) results in higher accuracy and lower fairness. By varying k. we can also compare the accuracy-fairnes tradeoff' curves of Dr-Fairness and FairBatch in Figure 3c Dr-Fairness shows a better tradeoff, which is consistent with the results in Sec. 3.1.1
3.1.3 ABLATION STUDY
We perform an ablation study on our framework to evaluate the impact of each component in the optimization on fairness and accuracy. For fairness, we conduct two ablations: FI) remove both the faimess loss in the outer objective and Xy, in the inner objective, and F2) only remove the fairness loss in the outer objective. For accuracy, we consider three ablations: A1) remove the accuracy loss in the outer objective; and My.s and the generated data loss in the inner objective, A2) remove My. and the generated data loss in the inner objective, and A3) only remove the accuracy loss in the outer objective. We note that A2 also represents how Dr-Fairness works only with real data when
7

---

Under review as a conference paper at ICLR 2023
cannot utilize generated data. Through this sequence of ablations, we observe that each part of our algorithm gradually improves the fairness and accuracy performances.
In Table the faimess ablations (in rows 1- 2) show worse fairness as the fairness loss and yy are discarded, and the accuracy ablations (in rows 3-5) demonstrate lower accuracy and balanced accuracy as some of the accuracy loss, generated data, and My.s are removed. We thus conclude that all components in our bilevel optimization contribute to the overall performances.
Table 3: Ablation study on CelebA, where we consider the setting of non-binary (haircolor) and binary z (gender). We mark noticeable performance degradttion with underlines.
Method
Acc.
Bal. Acc.
EO Disp.
Fl: w/o fair. loss and Ay,
85.7gm
83.9_1n
0.432_mom inos
F2: w/o fair. loss
85.1
84.6-m
0.261
Al: w/o acc. loss, Myls and gen. data
79.3
66.7
0.090
moss
A2; w/o My.z. and gen. data
80.0-12
70.8-M
0.031 H0012
A3: w/o acc. loss
21.5
82.1
0.0462009
Dr- Dr-Fairness
84.5go>
82.3
0.029200m
3.1.4 GENERATED DATA OF DIFFERENT QUALITIES
We analyze the robustness of our frame- work against the generated data qual- ity as shown in Table 4 We vary the quality of generated data by adding ran- dom Gaussian noise to the original im- ages. Interestingly, when the gener- ated data quality decreases (i.e., adding more noise to the images), Dr- Fairness automatically reduces the usage of the generated data, as shown in the second column of Table With this automatic adjustment, Dr-Fairness shows robust performances the last three columns. When the generated data is fully replaced with Gaussian noise (i.e., severe noise), the accuracy and fairness performances become worse than the clean setting as expected, but the fairness score is still much better than the non-fair baseline by reasonably sacrificing the accuracy. These results show that Dr-Fairness is effective even with low-quality generated data.
Table 4: Analysis of the behavior of Dr- when the quality of generated data changes. We add different amounts of Gaussian noise the original (clean) generated data. In the severe case, we fully replace the generated data with the noise. We set to haircolor (non- monn-bnarry) and to gender (binary).
Noise level in gen. data
Final gen. data ratio
found in Di-Faimess
Acc.
Bal. Acc.
EO Disp.
Clean
0.224 0.210
84.5e0:
82.3-u2
0.029
Light noise
84,4
81.6-10
0.033
Mid noise
19
84.51s1
81.9_sx
0.0391aam
Severe noise
0.1 14:
83.1
78.3-us
0,108
Non-fair baseline
83.8
82.0_ms
0.399
3.2 IMAGENET PEOPLE SUBTREE EXPERIMENTS
We finally perform experiments on ImageNet People Subtree, which represents large-scale real- world scenario. As only 10% of the data has group annotations, following Zhao et al. (2021), we first pre-train non-fair model on the entire training set with y labels and then fine-tune the pre-trained model to improve fairness on the small set with group labels.
Tables (beloww) and (in Sec. B.8) show the performances of the algorithms on four group scenarios: gender, skin color, age, and all combinations of them. The overall results are consistent with the CelebA experiments, where we can Dr-Fairness outperforms the baselines in accuracy, fairness, or both. Specifically, our algorithm shows the best or second-best performance on EO disparity and bias amplification in almost all group settings while obtaining better classification accuracies compared to the baselines with similar fairness scores. For example, we obtain classification accuracies better than FairBatch, with an absolute improvement of 5-9%, while achieving similar fairness scores.
As ImageNet People Subtree shows more complicated real-world scenario than CelebA, we have two additional observations. First, when we train the basclines w.r.t. EO, the bias amplification metric occasionally gets worse compared to the original model (e.g., Dom. Indep. on gender and FairBatch on skin color). This result shows that improving EO, which aims to minimize the label-specific accuracy gap between groups, does not necessarily lead to reducing the bias in the model compared to the data. In addition, as domain independence trains separate classifiers per each group, we suspect that the final model may have undesirable results (e.g., worse bias amp.) if some of the classifiers fail. Second, this dataset contains a large number of (y. 2l-csasses where many of them are extremely small- sized. Here the controllable data generation becomes challenging where the generated labels may be noisy, which negatively affects the fair training as well. Nonetheless, Dr-Fairness still shows a clear improvement in fairness compared to the baselines, and we believe more data with clean labels could further improve its performance.
The above experiments use ResNet50 (He al. (016)) as the model backbone. In Sec. B.9 we also conduct experiments using ViT (Dosovitskiy et al. 2021 1 and observe similar results. The results imply our method is applicable to different network architectures.
8

---

Under review as a conference paper at ICLR 2023
Table 5: Performances the ImageNet People Subtree test set when training w.r.t. equalized odds (EO) for either age or all combinations of groups. Other settings are identical to Table
Zi uge
z: (gender, skin color, age)
Method
Acc.
Bal, Acc,
BO Disp.
Bias Amp.
Apc.
Bal. Acc.
EO Disp.
Bias Amp.
Non-fai
61.4
61,81cx
0.8491omm
1.19
61.41ox
61.81sx
0.918
0.270
Simple Sampling
53,4
53.6
0.822 come
0.201 soum
54.6109
54.8
0.885
08181
PairAug" (Ramaswamy al. 2021
55.6+12 55.9413
0.820+am
0.168
57.6+94
57.9+95
0.871+300s
0.153
Dom, Indep. (Wang al. 2020
59,9
60.1
0,838
0.220
50.0
50.1
a.897
0,254
FairBatch (Roh et at. 2021
S1.3uia
51.511a
0.798
0.198
Lowen 1orm
10.1
50.3
0.853 jamms
0.185
Di-Fairness
59.7
59.9
0.784:
0.149 jaunz
S8.7
59.1
0.866
0.1 sure
4 RELATED WORK
As model faimess becomes indispensable for Trustworthy AI, numerous works have been recently
proposed to better measure fairness and design fairness-aware algorithms (Narayanan) 2018). Among
various fairness definitions, we focus on group fairness measures (Hardt al. 2016 Feldman
ctt al. 2015), which are widely studied in the fairnsss literature see representative works in Sec. C Untortunately most of the existing algorithms are not designed handle large number of groups or
labels, and our contribution is to support such large-scale scenarios for real-world applications.
Among the previous techniques, FairBatch (Roh al. 2021) is the most relevant to our work as finds the optimal group ratio for fairness on real data and shows the sfate-of-the-art faimess performances on various tabular datasets, including COMPAS (Angwin et al., 2016) and AdultIncome (Kohavi 1996). However, FairBatch may suffer from accuracy degradation due to oversampling on very small-size groups, especially in vision datasets. In particular, FairBatch cannot utilize accuracy-base objectives and generated data, which may result in worse accuracy and fairness tradeoff. Also, the theoretical guarantees of FairBatch do not apply in our setting because the outer objective of our bilevel optimization problem is non-convex. In contrast, Dr-Fairness minimize the accuracy degradation of fair training by optimally utilizing both real and generated data based on the fairness and accuracy objectives.
There is an emerging line of research for fairness in visual recognition (Najibi 2020 Wang et al. 2020) where using generated data is critical. Many visual recognition tasks involve multiple classes of varying sizes, and only using real data is often insufficient to improve fairness. In response, several works have proposed new algorithms to create a balanced dataset by augmenting the biased real dataset with well well-controlled generated data (Sattigeri et al. 2019 Choi et al.) 2020 Ramaswamy al. 202T). However, simply balancing the data sizes is not enough to achieve high- group fairness, as the learning difficulty and generated data quality can differ across groups. Although recent work (Zietlow et al., 2022) suggests an adaptive data augmentation that generating more data for worse-perioe ponse-performing groups, it uses heuristics to adjust data ratios without proper optimization and thus has limited fairness performance. In comparison, Dr-Fairness solves a novel bilevel optimization problem to find optimal data ratios and thus obtains both high fairness and accuracy.
In addition, there are other related studies on fair data reweighing (Li & Liu| 2022] Jiang & Nachum 2020: Krasanakis al. 2018), fair augmentation (Chuang & Mrouch| [021], and fair representa- tions Shui et al. 2022). Compared to our work, these studics only use real data or do not scale to
large datasets see detailed discussion in Sec. C
CONCLUSION
We proposed a novel adaptive sampling approach called Dr-Fairness that utilizes both real and generated data for fairess.. To perform adaptive sampling systematically, we first formulated bilevel optimization, where the goal is to find the optimal data ratios among sensitive groups and between real and generated data to achieve high group fairness while minimizing accuracy degradation. To solve the bilevel optimization problem, we then designed an efficient approximate algorithm based on the implicit function theorem and identity-mante matrix approximation. Extensive experiments on the C and ImageNet People Subtree datasets showed that Dr- Faimees achieves state-of-ther -oe-te--rtttttttt art faimess and accuracy performances. We believe Dr-Fairness opens up new opportunities for effectively using generated data in large-scale real-world scenarios.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
We believe our work can positively impact society by reducing discrimination in Al applications. In particular, our framework shows that generated data can compensate for unfairness issues real data (e.g., size bias and lack of diversity) to help obtain better accuracy and fairness results that would not have been possible otherwise. As a result, real-world applications have a better chance of ensuring fairess without sacrificing accuracy unnecessarily.
We do note that choosing an appropriate fairness metric for each application is essential, as a poor choice may lead to unintended discrimination. Thus, needs carefully choose the target fairness metrics based on the social context in each application. Also, in terms of privacy, we did not involve human subjects or use any direct personal identifiers in the experiments, except for the human images in the publicly available benchmark datasets.
REPRODUCIBILITY STATEMENT
We provide implementation and experimental details (e.g., libraries, hyperparameters, data prepro- cessing, and data generation) in Sec. and Sec. Dec.BIu enable the reproduction of our results. To help with the reproducibility of results in this paper, we will make our souree code publicly available in the future.
REFERENCES
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna M. Wallach. reductions approach to fair classification. In ICML, pp. 60 20-69, 2018.
Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias: There's software used across the country to predict future criminals. And its biased against blacks. ProPublica, 2016.
L. Elisa Celis, Lingxiao Huang, Vijay Keswani, and Nisheeth K. Vishnoi. Fair classification with noisy protected attributes: A framework with provable guarantees. ICML, 2021.
Kristy Choi, Aditya Grover, Trisha Singh, Rui Shu, and Stefano Ermon. Fair generative modeling via weak supervision. In JCML, 2020.
Ching-Yao Chuang and Youssef Mroueh, Fair mixup: Fairness via interpolation. ICLR, 2021.
Oswaldo de Oliveira. The implicit and the inverse function theorems: easy proofs. Real Analysis Exchange, 39(1 2007-288 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurlPS, 34:8780-877 8794, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard S. Zemel. Fairness through awareness. In ITCS, pp. 214-226, 2012.
Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubra- manian. Certifying and removing disparate impact. In SIGKDD, pp. 259-268, 2015.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leariin for fast adaptation of deep networks. In ICML,, pp. 1126-1135. PMLR, 2017.
10

---

Under review as a conference paper at ICLR 2023
Samy Wu Fung, Howard Heaton, Qiuwei Li, Daniel McKenzie, Stanley Osher, and Wotao Yin. Jfb: Jacobian free backpropagation for implicit networks. AAAI, 2022.
Zhengyang Geng, Meng-Hao Guo, Hongxu Chen, Xia Li, Ke Wei, and Zhouchen Lin. Is attention better than matrix decomposition? In ICLR, 2021.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. In NeurIPS, pp. 3315-3323, 2016.
Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographics in repeated loss minimization. In ICML, pp. 1929-1938, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, pp- 770-778, 2016.
Heinrich Jiang and Ofir Nachum. Identifying and correcting label bias in machine learning. In AISTATS, pp. 702-712, 2020.
Faisal Kamiran and Toon Calders. Data preprocessing techniques for classification without discrimi- nation. Knowl. Inf. Syst. 33(1):1-33, 2011.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In IEEE CVPR, pp. 81 8110-8119, 2020.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), ICLR, 2015.
Ron Kohavi. Scaling up the accuracy of naive-bayes classifiers: decision-tree hybrid. In SIGKDD, pp. 202-207, 1996.
Steven George Krantz and Harold R Parks. The implicit function theorem: history, theory, and applications. Springer Science & Business Media, 2002.
Emmanouil Krasanakis, Eleftherios Spyromitros-Xioon Xioufis, Symeon Papadopoulos, and Yiannis Kom- patsiaris. Adaptive sensitive reweighting to mitigate bias in fairness-aware classification. In WWW, pp. 853-862, 2018.
Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual faimesss. In NeurIPS, pp. 4066-4076. 2017.
Peizhao Li and Hongfu Liu. Achieving faimess at no utility cost via data reweighing. ICML., 2022.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In ICLR, 2018.
Risheng Liu, Jiaxin Gao, Jin Zhang, Deyu Meng, and Zhouchen Lin. Investigating bi-level opti- mization for learning and vision from unified perspective: A survey and beyond. IEEE TPAMI, 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In IEEE ICCV, pp. 3730-3738, 2015.
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In ATSTATS, pp. 1540-1552. PMLR, 2020.
Jelena I Luketina, Mathias Berglund, Klaus Greff, and Tapani Raiko. Scalable gradient- based tuning of continuous regularization hyperparameters. In ICML, pp. 2952-2960. PMLR, 2016.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-bace hyperparameter optimization through reversible learning. In ICMI., pp. 2113-2122. PMLR, 2015.
Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. ACM ICM, pp. 1485-1488, 2010.
11

---

Under review as a conference paper at ICLR 2023
Ninarch Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Computing Surveys, 54(6):1-35, 2021.
Aditya Krishna Menon and Robert C Williamson. The cost of faimess in binary classification. In FAT*, pp. 18,-118, 2018.
Alex Najibi. Racial discrimination in face recognition technology. Harvard Online: Science Policy and Social Justice, 24, 2020.
Arvind Narayanan. Translation tutorial: 21 fairness definitions and their politics. In ACM FAccT, volume 1170, 2018.
Weili Nie, Arash Vahdat, and Anima Anandkumar. Controllable and compositional generation with latent-space energy-based models. NeurIPS, 34:139773119 2021.
OpenAl. Dall-e /ittpss//opeeaa..comm/ta
2022.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Fabian Pedregosa. Hyperparameter optimization with approximate gradient. In ICML, pp. 737-746.
PMLR, 2016.
Tapani Raiko, Harri Valpola, and Yann LeCun. Deep learning made easier by linear transformations in perceptrons. In Artificial intelligence and statistics, pp. 924-932. PMLR, 2012.
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. Advances in neural information processing systems, 32, 2019.
Vikram V Ramaswamy, Sunnie SY Kim, and Olga Russakovsky. Fair attribute classification through latent space diaaiisi In IEEE CVPR, pp. 9301-9311 2021.
Yuji Roh, Kangwook Lee, Steven Fuijong Whang, and Changho Suh. FR-Train: A mutual information approach to fair and robust training. In ICML, pp. 8147-8157, 2020.
Yuji Roh, Kangwook Lee, Steven Euijong Whang, and Changho Suh. FairBatch: Batch selection for model fairness. In ICLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. Stable thubb//gttht...m. wtt.:tttwbbcccccoceoooeoeeetttte http://ghhhhbbbocoo////mmmoa//// fttp:://ssstaaaalaeff.uuuuu 2022.
diffusion. https: httpp:://githb:
Prasanna Sattigeri, Samuel C Hoffman, Vijil Chenthamarakshan, and Kush R Varshney. Fairness gan: Generating datasets with fairness properties using generative adversarial network. IBM Journal of Research and Development, 63(4/5):3 2019.
Aili Shen, Xudong Han, Trevor Cohn, Timothy Baldwin, and Lea Frermann. Optimising equal opportunity faimess in model training. ACL, 2022.
Changjian Shui, Qi Chen, Jiaqi Li, Boyu Wang, and Christian Gagné. Fair representation learning through implicit path alignment. ICML, 2022.
Aman Sinha, Hongseok Namkoong, and John C. Duchi. Certifying some distributional robustness with principled adversarial training. In ICLR, 2017.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training cata-efficient image transformers & distillation through attention. International Conference Machine Learning, 2021.
12

---

Under review as a conference paper at ICLR 2023
Zeyu Wang, Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards fairness visual recognition: Effective strategies for bias mitigation. In IEEE CVPR, pp. 8919-8922 2020.
Bryan Wilder, Eric Ewing, Bistra Dilkina, and Milind Tambe. End to end learning and optimization on graphs. NeurIPS, 32, 2019.
Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky. Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy. In FAT*, pp. 547-558, 2020.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez-Rodriguez, and Krishna P. Gummadi. Fairness constraints: Mechanisms for fair classification. AISTATS, pp. 962-970, 2017a.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez-Rodriguez, and Krishna P. Gummadi. Fair- ness beyond disparate treatment disparate impact: Learning classification without disparate mistreatment. In WWW, 1171-1180, 2017b.
Richard S. Zemel, Yu Wu, Kevin Swersky, Toniann Pitassi, and Cynthia Dwork. Learning fair representations. ICMI., pp. 325-333, 2013.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In AIES, pp. 330-340, 2018a.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ICLR, 2018b.
Eric Zhao, De-An Huang, Hao Liu, Zhiding Yu, Anqi Liu, Olga Russakovsky, and Anima Anandku- mar. Scaling fair learning to hundreds of intersectional groups. 2021.
Jieyu Zhao, Tianlu Wang. Mark Yatskar, Vicente Ordonez, and Kai- Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In EMNLP, pp. 9979-2989. 2017.
Dominik Zietlow, Michael Lohaus, Guha Balakrishnan, Matthäus Kleindessner, Francesco Locatello, Bernhard Schölkopf, and Chris Russell. Leveling down in computer vision: Pareto inefficiencies in fair deep classifiers. In JEEE CVPR, pp. 10410-10421, 2022.
13

---

Under review as a conference paper at ICLR 2023
A
APPENDIX - OPTIMIZATION AND ALGORITHM
A.1 BILEVEL OPTIMIZATION FOR DEMOGRAPHIC PARITY
Continuing from Sec. 2.1 we formulate our bilevel optimization w.r.t. demographic parity (DP) as follows:
min A. 2E2
max
( Myie Leeal (w (A, ni)
My.22
L. seal (w (W 14)B +
Σ
My.
real ( (wo(), p1)),
ma..
m2.22
VER
m
w \ 14) = arg min
Σ
Treal (. (w) + - My 1 (w)), Myit Xy,z Myy
s.t. AE [0,1], [0, EZEZdU. = 1, Y.
where V = (0, Note that DP is designed for binary classification. For designing the fairness loss, we are inspired by Roh et al. (202I), which gives a hint on formulating DP loss in bilevel optimization. Intuitively, the fractions in the fairness loss make the model reduces the disparity of each prediction ratio across groups, without considering the sizes of true label classes. This strategy can be sufficient condition for DP, as the goal of DP to achieve the same positive prediction ratio among groups see more details in Roh et al. (202T).
A.2 CAPTURING EQUALIZED ODDS WITH THE LOSS CONSTRAINT
Continuing from Sec. 2.1 explain how the loss-based constraints capture equalized odds.
We recall our notation: the empirical risk over samples in the set (y y, 2 z) is = ==
My.z -iy,=y.4== ((y,, 8.). where f(y,, 8c) is the loss function. Here, when the loss function &y..) 8,)
1.
1/0-loss (i.e., e(y,, 8,) -( # 52), where 1(-) is indicator function), the loss-based constraint can perfectly express the equalized odds disparity. Specifically, Ly. with 1/0-loss is equivalent to the probability of the correct predictions in each (y, 2)-class (i.e., Pr(ý yly y, 2 2 )). Therefore, our faimess loss constraint (i.c.. maxgeY, ,,, e | (w(A 14) L
y. (w (A, p))l becomes the equalized odds metric. which describes the class-conditioned accuracy disparity among groups (i.e., mna.yE....:2ez.. Pr(ý yly = y. z 21) Pr(ý yly = y. 22))). In practice, we can also use other loss functions like cross-entropy loss instead of the 1/0-loss, as other loss functions have been empirically verified as reasonable proxies for capturing group fairness metrics (Roh et al.) 2021 Shen et al. 2022).
A.3 USING THE IMPLICIT FUNCTION THEOREM
Continuing from Sec. 2.2 we describe how we convert the best-response Jacobian in Eq. using the implicit function theorem. We note that among various methods of solving bilevel optimization prob- lems, the implicit function theorem significantly improves the algorithm efficiency with theoretical evidence. Here, we first state the original implicit function theorem:
Theorem 1. (Implicit Function Theorem, stated in Krantz & Parks (2002); de Oliveira (2014) Let F I x R. + R. be a continuously differentiable function, where the input of is (æ, y) E R x R Assume there is an inpul point (a, b) that satisfies F(a, b) = 0, and aF(a.b) (i.e., the
dy
Jacobian matrix) is invertible. Then, there exist open sets U C IR" and V C 1 that contain a and respectively, and satisfy the following:
There is unique continuously differentiable function G, where G(a) = b and F(x, G(x)) 0 for all x e U.
We have the Jacobian matrix of partial derivatives of G U as follows:
aG(x)) Ix
Qy 0F(x, G(x)) OF(x, G(z)) dx
We now apply the above theorem in our setting. To get the best-response Jacobian w.r.t. X, we consider 0/mmz as F F(ar, (ar,)) y) and w(A) as G(m). Note that when accessing the gradient w.r.t. X, Ow
14

---

Under review as a conference paper at ICLR 2023
can ignore without loss of generality, and vice versa. Then, rewrite Theorem I for scenario as follows:
: I R - I be continuously differentiable function, where the input of fisee is (X, w) R R Assume there iun
Corollary (Implicil Function Theorem our setting) Let Husee
the
is an input point (a. b) that satisfies am 0, and Imner(a.b) Ben
BeoBa (i.e., the Jacobian matrix) is
invertible. Then, we have the Jacobian matrix of partial derivatives of (//w() ) as follows:
@wv(X)
_8 Jimner w) )
8 Immeer( (2,w)
0X = -[
iwdw
OwdX
Thus, with the assumption that the inner optimization has converged to local minimum, i.c., ffue 0, we can convert the best-response Jacobian in Eq. | to the multiplication of two matrices in Ou
Eq. Similarly, we can convert the best-response Jacobian w.r.t. μa by setting F(x, y) 0f (p, 8m0
and (xx) to w()).
A.4 SETTING FOR THE VALIDITY CHECK
Continuing from Sec. 2.2 we describe the synthetic binary setting in Roh al. (2021), which is used to empirically verify how close the solutions from our approximation strategy are to the optimal ones.
For generating the synthetic dataset, we use method in Zafar et al. (2017a), which produces two input attributes (X1, X2), one binary label attribute y, and one binary group attribute z. We draw each sample (X1, X2, y) from Gaussian distributions and make z follow a biased distribution.
In detail, we generate each sample (X1, X2, y) from two Gaussian distributions: X2)yy 0 ~ ([[[2; - 2), [10, 1; 1, 3)) and X2)yy = ~ [[ 2]. [5, 1; 1, 5)). Then, we make z follow biased distribution: Pr(z 1) Pr( (r X 1)/[Pr((x x²)
0) - Pr((x), x2)ly 1)]
where (x', x²] = (x1 (T/4) - X2 sin / 4), X1 (in( 4) X2 cos π (44)). This synthetic dataset contains training, validation, and test sets with 2k, 1k. and Ik samples, respectively.
In this experiment, we use logistic regression models for all algorithms, as in Roh al. 2021))
A.5 COMPARISON WITH EXACT INVERSE HESSIAN COMPUTATION
Continuing from Sec. 2.2 we compare our identity matrix-based approximation results with the exact inverse Hessian computation results. We use the same setting described Sec. A.4 where tractable to compute the exact inverse Hessian.
Figure 4 shows the group ratios of Dr-Fairness (with the identity matrix approximation) and Dr- Fairness with the exact inverse Hessian computation. We can that Dr-Fairness, which uses the identity matrix approximation to estimate the inverse Hessian in Eq. 2 converges to similar group ratios to those in computing the exact inverse Hessian. It implies that solution is close to the exact solution despite the method's 's simplicity. Another observation is that the data ratios in Figure 4b converge within fewer iterations than those in Figure We note that although the number of required iterations is fewer when computing the exact inverse Hessian, the training time is much slower if the number of parameters increases. Thus, the approximation used in Dr-Fairness can be a reasonable solution to estimate the inverse Hessian, which is usually intractable for large models and data.
Group Ratio
1.0
0.761
Group Ratio
1.0
0.763
0.5
mm
0.5
2,3 A,
0.239
0.237
0.0
0.0
100
200
300
400
0
100
200
300
400
Iterations
Iterations
(a) Group ratios in Dr- Dr-Faiiess.
(b) Group ratios with the exact inverse Hessian.
Figure 4: Comparison of group ratios from Dr-Fairness (with the identity matrix approximation) and Dr-
Fairness with the exact inverse Hessian computation. Both converge to similar ratios.
15

---

Under review as a conference paper at ICLR 2023
A.6 COMPARISON WITH OTHER PROBLEM FORMULATION METHODS
Continuing from Sec. 2.1 we discuss the advantages of using bilevel optimization compared to other
problem formulation methods, especially using distributionally robust optimization (DRO) (Sinha
et al.. 2017
DRO is one of the prominent problem formulation methods in machine learning, which can solve target objective in min-max formulation, but we believe that our bilevel formulation is more suitable to handle both real and generated data while improving group fairness. In our scenario, real data and generated data play very different roles in the bilevel objectives, and such roles are difficult to capture via a DRO formulation.
In detail, at the test-time evaluation, we only care about the faimess and accuracy losses (in our outer objective) on the real data distribution. Therefore, the empirical risk for generated data (T,ten) only appears in the inner objective for model parameters update.
Directly applying DRO to empirical risks of real and generated data does not lead to the same effect as our bilevel objective because this ignores the EO-based loss in our outer objective, and more importantly, it is unclear what is the benefit of optimizing the real/generated data sampling ratio to maximize the empirical risks. If we consider an example where we have a high loss on generated data and low loss on real data, then DRO should increase the sampling ratio for the generated data to increase the overall loss. However, the high loss on generated data could be the result of the low quality of generated data, and increasing its sampling ratio might instead hurt the performance and fairness. On the other hand, as shown in Sec. 3.1.4] Dr- Dr-Fairness would decrease the ratio of generated data when its quality is low.
B APPENDIX EXPERIMENTS
B.1 EXPERIMENTAL SETTINGS
Continuing from Sec. we provide detailed information on the experimental settings. We use PyTorch for all experiments and utilize the pre-trained ResNet50 (He et al., 2016) provided by PyTorch library (i.e., torchvision (Marcel & Rodriguez 2010). We change the last fully-connected layer of each model with the number corresponding label classes. When training, we update all model parameters in the pre-trat pre-trained model. The batch size of all experiments is 128. We set the learning rate for updating model parameters to 0.0001. For the data ratio (A and μ)) updates in Dr-Fairness, we use the Adam optimizer and set the learning rate for the ratio update to 0.005 in all experiments. When calculating the gradients w.r.t. model parameters or data ratios in Eq. 3 we use the autograd functionality in PyTorch. We apply the exponential moving average when updating the model parameters in Dr-Fairness. To prevent the overfitting, we use the validation set when measuring the fairness and accuracy losses in the outer objective of our bilevel optimization. Similarly, use the validation set in other baselines if they require the computation of additional (fairness) losses in the algorithm.
Hyperparameters For Dr-Fairness, we choose k from candidate set {0.1, 1, 10, 20} to have the best fairness score while minimizing the accuracy degradation in the validation set. We set the learning rates for X and to 0.005. We initialize À to the original (y, z)-ratios in the real data. We initialize μ to 0.5 for CelebA (i.e., we start with 50% real and 50% generated data) and 0.99 for ImageNet People Subtree (i.c., 99% real and 1% generated data). We use higher (conservative) μu initially for ImageNet People Subtree because its generated data has lower quality than that for CelebA. For all baselines, choose the hyperparameters that show the best fairness while minimizing the accuracy degradation in the validation set.
B.2 FILTERING LABEL CLASSES IN IMAGENET PEOPLE SUBTREE
Continuing from Sec. 3 explain how we filter the label classes in the ImageNet People Subtree dataset. Initially, the dataset contains 284 label classes. Among them, we filter out classes that are vague, duplicates, or small with few samples. First, we filter vague classes like "ex-president"
16

---

Under review as a conference paper at ICLR 2023
and "junior", which are hard to classify even for human annotators. To decide whether each class is vague or not, we perform internal crowdsourcing. For each class, we gather 3 expert decisions and do majority vote. Also, remove classes that conceptually duplicates of others. Finally, set the allowed minimum sample size to 50 and ignore the classes with fewer than 50 samples. As result, 1 122 classes used in experiments.
B.3 DATA GENERATION
Continuing from Sec. we explain the details on data generation.
For experiments on CelebA, we use LACE (Nie et al. 2021 to generate data. LACE is a controllable generation method that uses an energy-based model (EBM) in the latent space of a pre-trained generative model such as StyleGAN2 (Karras et al.) 2020j. We consider StyleGAN2 pre-trained on the CelebA-Ho BeQ dataset as our base generative model. In LACE, we first need train the latent classifiers in the w-Space of StyleGAN2, each of which corresponds to energy function for an individual attribute in the EBM formulation (see Eq. (4) in (Nie et al. 202T)). Since we mainly focus on five attributes (i.e., age, gender, smi gl asses,, and ha rcolorr in the CelebA experiments, we end up with five latent classifiers. Next, for each combination of attribute values (e.g.. age="young', gender= use the ordinal differential equation (ODE) sampler in the latent space to sample the corresponding images. We repeat the above sampling process until cover all the combinations of attribute values.
smi te=iieee'''',, glwsses=l 'true", and caircolor==bee
we
For experiments on ImageNet People Subtree, we use a guided diffusion model (Dhariwal & Nichol 2021) with classifier guidance (Song et al. 2020: Dhariwal & Nichol 2021) to generate data. Since there exist no ADM checkpoints pre-trained on ImageNet People Subtree, we first fine-tune the SeaggeNet-rretraineed ADM on ImageNet People Subtree, where the ADM model that we use conditions on the 1 12 labels. For classifier guidance, we also need to first train three time-dependent demographic attribute (i.e., gender, skin color, and age), on noisy images produced by the diffusion process. In particular, we fine-tune the noisy image classifier (also pre-trained on ImageNet) with three new prediction heads on 10% of the annotated data. Next, for each combination of label and attribute values, we pass the label value as the input of the conditional ADM and use the classifier guidance (i.e., the guidance from the fine-tune noisy image classifier in Eq. (10) of Dhariwal & Nichol (221)) with the scale repeat the above sampling process until we cover all the combinations of label and attribute values.
attribute classifiers, each corresponding
15. Similarly,
Here we describe the number of generated samples in each dataset. In CelebA, consider 5 attributes for the controllable generation: gender r (male and female), age (young and old), smile (true and false), ql glasses (true and false), and haircolor (black, blond, and others). Thus, these 5 attributes yield 48 class combinations (i.e., 2 x 3). We generate a total of 96k samples, where there are 2k samples for each attribute combination (e.g., 2k samples for (age='young' r='emalle""" ami 1 e= "t=ttree,, g asses='true' and ha iaairloooo realocolrrbblkk People Subtree, there are 112 label classes and group attributes: gender (male, female, and unsure), skin color (light, medium, and dark), and age (child, adult, middle, and retired). Thus, we have 4,032 combinations (i.e., 112 x 3 x 4) for the controllable generation. We generate 32 samples for each combination, which results in about 129k samples in total.
In ImageNet
Figures 6 examples of the generated images. We note that the controllable generation for ImageNet People Subtree is more challenging due to its large number of (y, =)-classes and labeling noises. Thus, the resulting generated data is noisier than the generated data in CelebA.
B.4 COMPARISON BETWERN FAIRNESSGAN AND PATRAUG
Continuing from Sec. maswany et al., 20 Table shows the accuracy and fairness performances of the two algorithms on CelebA, where they consider the setting of binary label attribute (attractive) and binary group attribute (gender). We show the numbers that are reported in Ramaswamy al. (022). As result, PairAug shows better accuracy and equalized odds performances compared to FairnessAAN
we compare FairnessGAN (Sattigeri et al. 2019) and PairAug (Ra-
17

---

Under review as a conference paper at ICLR 2023
B
age - gerder falsa, glasses
age - young. gender true, glasses
old, gender male, smie true, glasses false, haircolor others
young gender - male, smil true, glasses true, haircolor blond
Figure 5: Examples of the generated images CelchA
Anchor, gender femele, skin color medium, age - middle
Astronaut, gerder = fernale, skin color light age = adult
Captain, gender male, skin color dark age middle
Tennis male, color light child
Figure 6: Examples the generated images on ImageNet People Subtree.
B.5
OTHER RESULTS ON CELEBA W.R.T. DEMOGRAPHIC PARITY
Continuing from Sec. 3.I.I perform experiments on the CelebA dataset w.r.t. demographic parity (DP). Table shows the accuracy and faimess performances of the algorithms. Similar to the results Table 2 Dr-Fairness shows higher fairness than the pre-po peeprocessing (1:1 data ratio) baselines (i.e.. simple sampling. PairAug, and PairAug*) and higher accuracy than the in processing baselines (i.e., fairness constraint, domain independence, and FairBatch). Here, as the pre-processing baselines are not explicitly designed for DP, their fairness performances in terms of DP are sometimes worse than the original ResNet50 results. Since DP aims to ensure the same positive prediction rates without considering the true labels, the training sometimes needs to overfit on the positive labels for specific groups to improve DP. Thus, simply equalizing the data ratio among groups may not be enough to improve DP compared the EO case.
B.6 OTHER RESULTS ON CELEBA WITH DIFFERENT SETTINGS
Continuing from Sec. 3.1.) we perform experiments on CelebA dataset with different group and label combinations. Tables 8 and show the accuracy and fairness performances the following two settings: (y. 2) = (smiling, gender) and (y. z) = (haircolor, gender). In both cases, we observe consistent results those in Sec. 3.1.1 where Dr-Fairness achieves high faimess (esp. EO) while not sacrificing accuracy. We note that in these two settings, the bias amplification values of the non-fair baseline already very small (i.e., good enough), the fair algorithms may not further improve the bias amplification.
B.7 ACCURACY-FAIRNESS TRADEOFFS
Continuing from Sec. visually demonstrate the accuracy-fainee fairness tradeoff's of Dr-Fainness and the baselines on the CelebA and ImageNet People Subtree datasets. Figure shows the accuracy and unfairness performances of the haselines and Dr-Faimess. Here, being the lower-right indicates
18

---

Under review as a conference paper at ICLR 2023
Table 6: Additional comparisons between baselines using generated data. We compare PairAug and Fairness- GAN, where results are from Ramaswamy et al. (021). The baselines use attractive attribute as label and gender attribute as the group.
Method
Acc.
EO Disp.
PairAug (Ramaswamy et al. 2021
80.0
0.21
FairnessGANN (Sattger al. 2019
71.0
0.25
Table Performances on the CelebA test set w.r.t demographic parity (DP) on the binary y (age) and (gender) scenario, Other settings are identical to Table We mark the best and second best performances among the faimess algorithms with bold and underline, respectively.
y: age and z: gender
Method
Acc.
Bal. Ace.
DP Disp.
Bias Amp.
Non-fair
86.42m
76.4
0. 0.141 Loom
0.101 10002
Simple Sampling
86.8
78.3
0.132
0.052
PairAug (Ramaswamy al.
2021
85.6=06
79.8=0a
0.151 2000
0.030
PairAug* (Ramaswamy ct al. 2021
86.7=02
79.3
0.144200m
0.053
Fair. Const. (Zafar al. 201 7a
85.9
74.321a
0.103 0.124
Dom. Indep. [Wang 2020
85.020a
72.4
0.091 2:000
0.034
FairBatch (Roh el al. 2021
84.9=03
74.0
0.074
-0.034
Dr-Fairness
86.5
78.2
0.088
-0.028
higher accuracy and fairness and is thus desirable. In CelebA, Dr-Faireess achieves both better accuracy and fairness performances compared to all the baselines. In ImageNet People Subtree, I) the baselines Simple Sampling, PairAug, and Dom. Indep. show strictly worse performances than Dr- Dr-Fairness, 2) FairBatch achieves higher fairness than ours, but the accuracy degradation is severe, and 3) the non-fair baseline shows high accuracy, but much worse fairness. Thus, conclude that Dr-Fairness achieves the best accuracy-FFe -airrac tradeoffs in both datasets.
0.6
Nondfir
Nonsfuir
Indep.
Stuple Sumplieg
Stuple Sumpline
PairBuich
PsirAng
De-Phiranes
EO Disp.
10.1
tndep-
EO Disp.
0.90
Dr- Foir
0.88
0.2
0.85
78.0
80.0
82.0
$4.0
86.0
30.0
55.0
60.0
Accuracy
Accuracy
(a) Tradeoffs CelchA
(b) Tradeoffs in ImageNet People Subtree when 7 (gender, skin color, age).
when (y, (haircolor, (gender, age)).
Figure 7: Accuracy-unfaimee graphs to visualize the algorithm performances on CelebA and ImageNet People Subtree datasets. Being on the lower right is desirable (high accuracy and fairness).
We explain when Dr- Fa-Faiiness can improve both fairness and accuracy compared other fairness baselines. We believe this phenomenon is related to the optimal accuracy accuracy-fairness tradeoff, which is known to be determined by the data distribution (Meoon Williamson| 2018). When the performance of fair algorithm lies on the optimal accuracy-fairnest tradeoff, any other algorithm can only achieve either better fairness or better accuracy, but cannot improve both. However, when the fairness algorithms do not achieve the optimal accuracy-fairee tradeoff in the given data, there is an opportunity to improve the model's performances toward the optimal tradeoff line. For example, we observe that Dr-Fairness improves both accuracy and fairness compared to other baselines in CelebA, implying that the baselines do not achieve the optimal accuracy-fait fairness tradeoff in the first place.
19

---

Under review as a conference paper at ICLR 2023
Table 8: Performances the CelebA test set w.r.t. equalized odds (EO) on the binary y (smiling) and binary (gender) scenario. We mark the best and second best performances among the fairness algorithms with bold and underline, respectively. Other settings are identical to Table
age
Method
Acc.
Bal. Acc.
EO Disp.
Bias Amp.
Non-fair
91.7
91.7
0.027
0.006
Simple Sampling
91.8ga1
91.8=a1
0.019
0.012
PairAug (Ramaswamy al. 2021
91.4+aa
91.4+aa
0.022
0.003
PairAug* (Ramaswamy et al.
2021
92.0aa1
92.0aa1
0.031
0.003
Fair. Const. (Zafar et al. 2017b
91.3+as
91.3+as
0,014=00007
0.009 +0.002
Dom. Indep. [Wang 2020
91.2
91.2200
0.014
0.009
FairBatch (Roh et 2021
91.61a
91.6410
0.012
0.018 1000000
Dr-Fairness
92.7
92.7 :01
0.013
0.009 1:00044
Table 9: Performances on CelebA test set w.r.t. equalized odds (EO) on the non-binary y (haireolor) and
binary z (gender) scenario. We mark the best and second best performances among the fairness algorithms with bold and underline, respectively. Other settings are identical to Table 2
y: haircolor and z: gender
Method
Acc.
Bal. Acc.
EO Disp.
Bias Amp.
Non-fair
83,8
82.0
0.399 20,019
00010
Simple Sampling PairAug* (Ramaswamy et al. 2021
83.1+ar
78.9
0.322
0.021
82.7
80.0asa
0.374
0.018 a
Dom. Indep. (Wang et al. 2020
83.2#aa
0.308 79.4
0.011 0.06
FairBatch (Roh et al. 2021
78.4114
68.7
0.085
0.072
Dr-Fairness
84.5
82.3
0.029
0.023
B.8 OTHER RESULTS ON IMAGENET PEOPLE SUBTREE WITH DIFFERENT GROUPS
Continuing from Sec. 3.2
we perform experiments on the ImageNet People Subtree with different groups. Table 10 shows the accuracy and fairness performances when the group attribute is gender or skin color. We observe consistent results to those in Sec. 3.2
Table 10: Performances the ImageNet People Subtree test when training w.r.t. equalized odds (EO) for either gender skin color. We mark the best and sccond best performances among the fairness algorithms with bold and underline, respectively, Other settings are identical to Table
z: gender
2: skin color
Method
Acc.
Bal, Acc.
EO Disp.
Bias Amp.
Acc.
Bal, Acc.
EO Disp.
Bias Amp.
Non-fair
61.4sus
61,8+us
0,87
0.256
61.4uu
61.8+us
0.874-ux04
0,239
Simple Sampling
54.8
54.940s
0.834
0.282=0a5
55.5
55.7+10
0.849
0.219=aa35
PairAug" (Ramawwayy et
2021
54.6
54.9+0s
0.821+am5
0.241
18.1
58.3+as
0.830+a.00
0.225 +0000
Dom. Indep, (Wang et al. 2020
59.9
60.1
0.857
insi
0.381
60.2
60.5
0.874_nnos 0.204 Loom
FairBatch (Roh et 201/1021
52.9
53.1
0.8161am
0.191
gnass
21.9
52.1
0.811
gaase
0.346
Dr-Fairness
58.2.aa
58,3
0.817
0,220=ma27
60.8
61.2=1a
0.845
0.137
B.9 OTHER RESULTS ON IMAGENET PEOPLE SUBTREE WITH VIT
Continuing from Sec. 3.2 we perform experiments using DeiT-S (Touvron et al. 2021 variant of
ViT (Dosovitskiy et al. (021), on the ImageNet People Subtree dataset w.r.t the age group attribute. Table show the accuracy and fairesss performances of Dr-Fairness and the representative baselines. Here we observe similar results to those in Sec. 3.2 This experiment shows that Dr-Fairness is applicable to various network architectures, including VIT.
20

---

Under review as a conference paper at ICLR 2023
Table 11: Performances the ImageNet People Subtree test set w.r.t equalized odds (EO). We use DeiT- S (Touvron all 202T), variant of ViT (Dosovitskiy (02T), for all algorithms. We mark the best and second best performances among the fairness algorithms with bold and underline, respectively. Other settings are identical to Table 2
z: age
Method
Acc.
Bal. Acc.
EO Disp.
Bias Amp.
Non-fair
64.8 8200
65.2
0.890
0.198
Simple Sampling
55.3107
55.7
0.807 i0om
0.261 1ooo
PairAug* (Ramaswamy al. 2021]
59.5
59.9
0.824
0.237 1000S
Dom. Indep. (Wang et al.
2020
51.8
52,0&t.
0,817
0.634 00.245
FairBatch (Ron et al. 202
58.2
58. 28.6
02017
0.162
Dr-Fairness
64.4
64.8 e.s
0.774:t0.01
0.176+0..0
C APPENDIX RELATED WORK
Continuing from Sec. we discuss more related work.
The main approaches for satisfying group fairness are: 1) fix the training data to mitigate bias (Kami ran & Calders 2011 Zemel et al. 2013), 2) modify the training process to prevent the model from
learning bias (Zafar et al.| 7a b Zhang al.] 2018a] Agarwal al. 2018 Roh et al. 2020
2021), or 3) alter the outputs of the trained model to achieve fairness metrics (Hardt et al. (006)). As
discussed in Sec. 4 most of these algorithms are not designed to handle large number of groups or labels, and our contribution is to support such large-scale scenarios for real-world applications.
In addition, there are other related studies on 1) fair data reweighing (Li Liu 2022 Jiang &
Nachum 2020; Krasanakis et al. 2018), 2) fair augmentation (Chuang & Mrouch 2021). and 3) fair
representations (Shui et al. (002))
Krasanakis et al.) 2018) are indeed relevant to our data sampling framework, as they keep finding data weights to improve group fairness. Compared to our work, Jiang & Nachum (2020) and Krasanakis et al. (2018) require multiple re-training of the model, and & Liu (2022) uses additional assumptions, including the loss function being twice differentiable and strictly convex in the model parameters. Therefore, applying these reweighing techniques when training models on large-scale data may lead to significant training times due to multiple re-trainings performance degradation due to violations of the assumptions. In comparison, Dr-Fairness works well in large-scale scenarios as in our experiments.
The data reweighing techniques (Li & Liu 2022; Jiang & Nachum 2020
There is another interesting work called FairMixup (Chuang & Mrouch 2021). which augments training data for faimess using mixup methods (Zhang et al. 2018b). However, the key difference from ours is that FairMixup only augments the data within the original training data distribution. FairMixup also cannot dynamically adjust sampling ratios from different groups explicitly. In contrast, Dr-Fairness can utilize any additional data, which is not limited to the original training (real) data distribution, and also find the optimal sampling ratios among groups and between real and generated data.
Recently, a fair representation paper (Shui et al. 2022) uses the bilevel optimization formulation with the implicit function theorem and shows promising results. Although this work also uses a bilevel formulation, it targets a different problem from ours, where the goal is to map the input feature X into the latent variable X' for fairness. In comparison, we use the bilevel formulation to adjust the ratio among groups and between real and generated data to improve fairness. Specifically, we design the inner objective by explicitly separating the group-wise terms and real/generated data terms to adequately apply the data weights. We note that this inner structure is different from (Sui et al. (2022), where they apply the common outputs of the outer objective to all terms in the inner objective that only considers real data. In addition, when approximating the inverse Hessian matrix resulted by the implicit function theorem (e.g., Eq. 2 Shui al. (022) use the conjugate gradient (CG) method (Rajeswaran et al. 2019). whereas we utilize the identity- matrix approximation (Luketina et al. 2016 Gen et al.
2021 We note that the identity-matrix
21

---

Under review as a conference paper at ICLR 2023
approximation is known to be much more efficient and may achieve on-par sometimes better performances compared to the CG method (Lorraine et al. 2020).
Although not immediate focus, there other important research lines for fairness: 1) fulfilling other fairness definitions (c.g., individual fairness (Dwork et al. (012)) and causal fairness (Kusner et al. 2017)) and 2) handling noisy missing group labels (Hashimoto et al. [2018] Celis et al. 2021
We believe that supporting these aspects can be promising future directions.
22


Reviews:
[{"review_id": "s3sorT12L3", "reviewer": "Reviewer_SsCr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper formulates the fairness setting with a bi-level optimization on finding good group ratios. Some approximation techniques are proposed to overcome the difficulty of bi-level optimization.\n", "summary_of_the_review": "This paper formulates a new fairness setting with a bi-level optimization on finding good group ratios. However, to overcome the difficulty of bi-level optimization approximation techniques are proposed, which may hurt the quality of found group ratios.", "strength_and_weaknesses": "Strength\n\n+ This paper formulate the fairness setting with a bi-level optimization on finding good group ratios.\n\nWeaknesses\n\n- \"λy,z is the ratio for group z in class y, µy,z is the ratio for real data in the (y, z)-class, and λ and µ are the sets of all λy,z and µy,z\". It is reasonable to adjust λ due to different generated data qualities, but why is adjustment of µ beneficial (not using all available real data)? simply due to class imbalance? For different generated data qualities, the ideal case is to weed out generated data with low qualities? Is this tried in the proposed algorithm? Group ratio is a quite coarse measure.\n\n- One limitation of this paper is that it applies only when synthetic data samples are generated.\n\n- If FairBatch has a theoretical guarantee to find the optimal group ratios, why does \"Dr-Fairness achieves much higher accuracy with similar or better fairness than FairBatch\" if they converge to similar group ratios?\n\n- Experiment results are better in CelebA than ImageNet People Subtree. So will more complex data hurt Dr Fairness even more? Any insight?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "b5nVCGa0Vg", "reviewer": "Reviewer_geXT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies machine learning fairness in visual recognition. The unifying idea of this paper is to use both real and generated data to improve fairness via adaptive sampling. The iterative procedure adaptively adjusts data ratios between real and generated data based on feedback from a classifier. The authors cast this problem into a bi-level optimization and accelerate the solving process with some tricks. Experiments on image data demonstrate the effectiveness.", "summary_of_the_review": "In sum, I think the paper brings some interesting ideas to this field. However, I hold some concerns about the generated data part and the later usage. I am looking forward to the authors' response.", "strength_and_weaknesses": "Pros:\n\n1. The high-level idea makes sense to me. The authors want to precisely sample data from different groups and generated data to achieve better fairness and lower accuracy cost. Sample data more carefully should work for this problem because all biases inherently come from data.\n\n2. The paper is technically sound.\n\n3. The experiments are comprehensive and solid. Compared to FairBatch, the proposed method shows a better fairness as well as good tradeoffs between fairness and accuracy.\n\nCons:\n\n1. The authors give limited explanations on 'generated data.' I would query what kinds of data are 'generated data' for algorithmic fairness, and what is the distributional assumption of such kinds of data.\n\n2. Some clarifications are needed in the current version. For example, the statement 'In particular, proper usage of generated data can reduce the accuracy degradation caused by over-sampling minority groups from real data' lacks corresponding explanations.\n\n3. The motivation for using generated data can be further elaborated. In my opinion, the proposed method can also be adapted to reweighing training data only without using any external data. Why there are different treatments between the original data and generated data in this case?\n\n4. Since this paper works on data resampling and reweighing, probably the authors want to focus on similar techniques in the related work part. I list three relevant works on data reweighing for authors' reference [1,2,3].\n\n[1] Adaptive Sensitive Reweighting to Mitigate Bias in Fairness-aware Classification, WWW'18\n[2] Identifying and Correcting Label Bias in Machine Learning, AISTATS‘20\n[3] Achieving Fairness at No Utility Cost via Data Reweighing, ICML'22", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HxNz-FptVyb", "reviewer": "Reviewer_niP4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission develops a novel adaptive sampling approach that leverages both real and generated data for fairness. The authors design a bilevel optimization that finds the optimal data sampling ratios among groups and between real and generated data while training a model. The proposed method exhibits desired trade-off between accuracy and fairness on the CelebA and ImageNet People Subtree datasets.", "summary_of_the_review": "Although the proposed method is reasonable and achieves desired trade-off between acurracy and fairness, the proposed method only combines existing components. \n\n", "strength_and_weaknesses": "Strength: Different from Fairbatch, which finds the optimal group ratio for fairness only using real data, Dr-Fairness optimizes  not only group ratio but also sampling ratios between real and generated data. Therefore, Dr-Fairness can enjoy the potential of both real and generated data. It design a bilevel optimization to perform adaptive sampling.\n\n\nWeakness: The main novelty of the method lies in extending Fairbatch to Dr-Fairness which seems marginal. Specifically, Dr-Fairness adopts a conditional generator to geerate samples and adopt the to-be-learned sampling ration for real and generated data, which is similar to the group ration. The authors stated that they propose a simple approximation. I am not sure which part is new given the implicit function theorem and its corresponding work. Besides, this work needs more auxiliary information from pre-trained generator. And, from the experiments on ImageNet People Subtree test set, we find that the performance of Dr-Fairness is highly dependent on the quality of the generated samples. Also, it is unclear why the authors claimed that Dr-Fairness achieves the state-of-the-art accuracy and fairness, which is not.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8lp9oA2iRa", "reviewer": "Reviewer_ZWfo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied the unbalancing problem in data generation based fairness, i.e, the generated data should be balanced or adjusted to control the prediction/fair performance with raw data. Based on this idea, an bi-level optimization (through implicit function) is proposed to learn the balancing coefficient $\\lambda$, $\\mu$. Empirical results on CelebA/ImageNet demonstrated its effectiveness. \n", "summary_of_the_review": "[ I would recommend that *the paper is generally fine, but not good enough* (4/10). While it seems that no such a score is provided. Thus I would currently recommend 3.]\n\nThis paper studied the unbalancing problem in data generation based fairness, i.e, the generated data should be balanced or adjusted to control the prediction/fair performance with raw data. Based on this idea, an bi-level optimization (through implicit function) is proposed. \n\nThis paper has merits, while they are rather modest in technical parts (see 1-3, 6 in detailed comments) and understanding fairness (see 4-5, 7 in detailed comments).\n", "strength_and_weaknesses": "### Pros\n- This paper considered a meaningful problem in data-augmentation based fairness, i.e, rebalancing the training loss of generated and real data. Furthermore a bi-level objective is proposed to automatically balance the sensitive attribute and label balancing weights. The idea sounds reasonable and empirically evaluations are conducted. \n\n### Cons:\n- This paper focused on a very narrow improvement in technical fair learning without proper motivation in social aspects. Thus I would say the paper scope and significance are quite modest. \n- The bi-level objective is not well-motivated. The framework could also be realized by DRO (distribution robust optimization).  \n- Lack of clear discussion on previous work.\n\nOverall I would say, this paper has merits, while they are rather modest in terms of technical part and understanding fairness.\n\n### Comments on cons\n\n1. [Paper scope] This paper focused on the specific aspect in data augmentation based fairness. I.e, data generation to improve unfair predictions. However, data generation through generative models is generally difficult. Thus I would be a bit unsure about the importance of fair generative models in data-augmented fair learning or general fair learning. \n2. [Technical motivation] The key contribution is to introduce the label and sensitive attribute reweighted loss to balance different factors. While I think this framework could also be achieved by the DRO framework by properly adjusting the weights of different groups/labels. Why not DRO? Is it necessarily to formulate the problem as a bi-level objective?\n3. [Related work/Technical novelty] The bi-level objective in fair learning or data-augmentation are not entirely new. Discussions on related work seem quite lacking. For example, fair-mixup [1] as a typical data augmentation approach. As for the bi-level in learning fairness is not new, for example fair-path [2] indeed introduced bi-level objective and implicit function to learn fair representation. \n4. [Impact on real-world fairness] This paper focused on a nich technical point in learning the balancing coefficient. However, this paper does not consider the real-world fairness impact. Please take note that algorithmic fairness is not merely a method to improve state-of-the-art. An important aspect is to understand the social impact of proposed fair analysis. Unfortunately, this part is completely missing in the paper, making the contribution in fairness (non technical) quite modest. \n5. [Experiments, Impact on real-world fairness] The experiments are fine to validate the improved performance on benchmark but are not well-motivated to support the real-world fairness. We could always have better results on some benchmarks while the real-world discrimination still is not addressed. \n6. [Technical correctness] In Sec 2.1 and Sec 2.2 the loss constraint is not equivalent to enforce EO. They could be equivalent only in certains specific assumptions such as the loss is calibrated. This part is not correct for me. \n7. [Claims on achieving fair and accuracy] In many parts, this paper claimed the fair and accuracy could be both improved, this is quite a big claim. In fact, this paper did not provide rigorous justifications on why/when/how these could satisfy. I would think there is an over-claim of the contribution.  \n\nReference \n[1] Fair mixup: Fairness via interpolation. ICLR 2021\n[2] Fair Representation Learning through Implicit Path Alignment. ICML 2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject