A Stochastic Optimization Framework for Fair Risk Minimization
Authors: Andrew Lowy
Sina Baharlouei
Rakesh Pavan
Meisam Razaviyayn
Ahmad Beirami
Venue: Accepted by TMLR ()

Abstract: Despite the success of large-scale empirical risk minimization (ERM) at achieving high accuracy across a variety of machine learning tasks, fair ERM is hindered by the incompatibility of fairness constraints with stochastic optimization. We consider the problem of fair classification with discrete sensitive attributes and potentially large models and data sets, requiring stochastic solvers. Existing in-processing fairness algorithms are either impractical in the large-scale setting because they require large batches of data at each iteration or they are not guaranteed to converge. In this paper, we develop the first stochastic in-processing fairness algorithm with guaranteed convergence. For demographic parity, equalized odds, and equal opportunity notions of fairness, we provide slight variations of our algorithm–called FERMI–and prove that each of these variations converges in stochastic optimization with any batch size. Empirically, we show that FERMI is amenable to stochastic solvers with multiple (non-binary) sensitive attributes and non-binary targets, performing well even with minibatch size as small as one. Extensive experiments show that FERMI achieves the most favorable tradeoffs between fairness violation and test accuracy across all tested setups compared with state-of-the-art baselines for demographic parity, equalized odds, equal opportunity. These benefits are especially significant with small batch sizes and for non-binary classification with large number of sensitive attributes, making FERMI a practical, scalable fairness algorithm. The code for all of the experiments in this paper is available at:
https://github.com/optimization-for-data-driven-science/FERMI

Published in Transactions on Machine Learning Research (11/2022)
A Stochastic Optimization Framework for Fair Risk Minimization
Andrew Lowy*t
lowya edu
Deportment of Mathematics University of Southern California
Sina Baharlouci
baharlou edu
Department of Industrial & Systems Engineering University of Southern California
Rakesh Pavan
rpavan edu
Department of Computer Science University of Washington
Meisam Razaviyayn
rozaviga Gusc. edu
Department of Industrial Systems Engineering, Computer Science, and Electrical Engineering University of Southern California
Ahmad Beirami*
beirami@ggoole..co
Google Research
Reviewed on OpenReview: https: // pptp//veeeere.eeeeemmee/e. net/ forum? id= P9C
Abstract
Despite the success of large-scale empirical risk minimization (ERM) at achieving high accuracy across variety machine learning tasks, fair ERM is hindered by the incompati- bility of fairness constraints with stochastic optimization. We consider the problem of fair classification with discrete sensitive attributes and potentially large models and data scts, requiring stochastic solvers. Existing in-processing fairness algorithms are either impractice in the large-scale setting because they require large batches of data at each iteration or they are not guaranteed to converge. In this paper, we develop the first stochastic in-processing fairness algorithm with guoranteed convergence. For demographic parity, equalized odds, and equal opportunity notions of fairness, we provide slight. variations of our algorithm called algorithm-calledd FERMI and prove that each of these variations converges in stochastic optimization with any batch size. Empirically, (non-binary) sensitive altributes and non-binary largets, performing well even with minibalch size as small as one. Extensive experiments show that FERMI achieves the most favorable tradeoffis between fairness violation and test accuracy across all tested setups compared with state-of-the-art baselines for demographic parity, equalized odds, equal opportunity. These benefits are especially significant with small batch sizes and for non-binary classification with large number of sensitive attributes, making FERMI practical, scalable fairness algorithm. The code for all of the experiments in this paper is available at::
show that FERMI is amenable to stochastic solvers with multiple
https://gitte wttp://ww.....o.piiiiaaaaaaaooaa tte wtw.errrrreeiiieneeeeee/ee
"AL and SB contributed equally to this paper. 'The work of AI., SB and MR supported in with funding from the NSF CAREER Award 2144985, from gift from 3M, and from the USC-Meta Center for Research and Education Al Learning. TTe work of AB done Meta AT.

---

Published in Transactions on Machine Learning Research (11/2022)
Reference
NB target
NB attrib.
NB
Beyond logistic
Stoch. alg, (unbiased**)
Converg. (stoch.)
code
FERMI (this work)
(5)
Cho al.
2020b
4
1
/
/
(X)
Cho et al.
2020a
8
/
/
(
Baharlouei et al.
2020
/
1
1
/
x
/ (X)
Rezaei et al.
2020
x
x
x
x
x
Jiang al. 2020
x
x
x
Mary et al.
2019
8
/ (2)
Prust et al.
2019
x
x
x
/ (2)
Donini et al. 20118
x
/
x
/
x
x
Zhang al. 2018
x
(X) V (X)
Agarwal et al.
[2018]
x
X
X
Table 1: Comparison of state-of-the-ar in-processing methods (NB = non-binary) on whether they (a) handle non-binary targets (beyond binary classification), (b) handle non-binary sensitive attributes, (c) release code that applies to non-binary (and whether the gradients are unbiased), (f) provide convergence guarantees (for stochastic optimization). FERMI is the only method compatible with stochastic optimization and guaranteed convergence. The only existing baselines for non-binary classification with non- binar sensitive attributes are [Mary et al. 2019 Baharlouei al. 2020 Cho et al. 2020b) (NB code). "We refer to the in propocesssng method of (Jiang et al. 2020). not their post-processing method. "We use the term "unbiased" statistical estimation sense; not to be confused with bias the fairness sense.
(d) extend to arbitrary models, (e) provide code for stochastic optimization
1 Introduction
Ensuring that decisions made using machine learning (MI.) algorithms are fair to different subgroups is of utmost importance. Without any mitigation strategy, learning algorithms may result in discrimination against certain subgroups based on sensitive attributes, such gender or race, even if such discrimination is
absent in the training data (Mehrabi et al.]
2021 and algorithmic fairness literature aims to remedy such
discrimination issues [Sweeney] 2013
Datta et al.
2015
Feldman et al.
2015
Bolukbasi et al. 2016 Angwin
et al. 2016 Calmon et al.
2017b; Hardt et al.
2016
Fish et al. 2016
Woodworth et al.] 2017
Zafar et al.
2017
Bechavod & Ligett, 2017 Agarwal et al.
2018
Kearns al.
2018
Prost et al. 2019
Lahoti al,
2020
Modern ML problems often involve large-scale models with hundreds of millions or even billions of
paramcters, c.g., BART (Lewis et al. 2019 ViT (Dosovitskiy et al. 2020). GPT-2 (Radford et al. 2019 In
such cases, during fine-tuning, the available memory on a node constrains us to use stochastic optimization with (small) minibatches in each training iteration. In this paper, we address the dual challenges of fair and stochastic ML, providing the first stochastic fairness algorithm that provably converges with any batch size.
A machine learning algorithm satisfies the demographic parily fairness notion if the predicted target is independent of the sensitive attributes (Dwork et al. 2012). Promoting demographic parity can lead to poor performance, espocially if the truc outcome is not independent of the sensitive attributes. To remedy this, Hardt et al. (2016) proposed equalized odds to ensure that the predicted target is conditionally independent of the sensitive attributes given the true label. A further relaxed version of this notion is equal opportunity which is satisfied if predicted target is conditionally independent of sensitive attributes given that the true label is in an advantaged class (Hardt et al.) 2016). Equal opportunity ensures that false positive rates are equal across different demographics, where negative outcome is considered an advantaged class, e.g., extending a loan. Soe Appendix for formal definitions of these fairnoss notions.
Measuring fairness violation. In practice, the learner only has access to finite samples and cannot verify demographic parity, equalized odds, equal opportunity. This has led the machine learning community to define several fairness violation metrics that quantify the degree of (conditional) independence between random variables, e.g., Lox distance (Dwork el al. 2012 Hardt et al. 2016), mutual information (Kamishina et al. 2011 Rezaei et al. 2020 Steinberg et al. 2020 Zhang et al. 2018 Cho et al. 2020a Roh et al.) 2020),
Pearson correlation (Zafar et al.) 2017 Beutel et al.
(aaeenneeeBBcaaaaaa (Bochavod 2019), false positive positive/negative negative rate difference
& Ligett 2017), Hilbert Schmidt independence criterion (HSIC) (Pérez-Suay et al. (2017). kernel-based minimum mean discrepancy (MMD) (Prost et al.
2019 Rényi correlation (Mary et al. 2019: Baharlouei
2

---

Published in Transactions on Machine Learning Research (11/2022)
and exponential Rényi mutual information (ERMI) (Mary et al. 2019 In this paper, we focus on three variants of ERMI specialized to demographic parity, equalized odds, and equal opportunity. The motivation for the use of ERMI is two-fold. First, we will see in Sec. 2 that ERMI is amenable to stochastic optimization. Morcover, we observe (Appendix C that ERMI provides an upper bound on several of the above notions of lairness violation. Consequently, a model trained to reduce ERMI will also provide guarantees on these other fairness violations
et al. 2020 Grari et al. 2019 2020
Related work & contributions. Fairness-promoting machine learning algorithms can be categorized in three main classes: processoes. posteprocsssing and in-processing methods. Pre-process Pre-processing algorithms (Feldman et a 2015 Zemel et al. 2013: Calmon et al. 2017b) transform the biased data features to new space in which the labels and sensitive attributes are statistically independent. This transform is oblivious to the training procedure. Post- Post-processing approaches (Hardt et al. 2016 Pleiss et al.) 2017] mitigate the discrimination of the classifier by altering the final decision. In-processing approaches focus on the training procodure and impose the notions of fairness as constraints or regularization terms in the training procodure. Several regularization-basee methods are proposed in the literalure to promole fairness (Ristanoski al.
2013
Quadrianto & Sharmanska 2017) in decision-trees trees Kamiraa et al. 2010 Raff et al. 2018 Aghaei
ct al.
2019), support vector machines DDnini ct al. 2018). boosting (Fish ct al.
2015). neural networks
Grrar et al.
2020
Cho et al.) 2020b; Prost et al.) 2019), or (logistic) regression models
Zafar et al. 2017
Berk et al.
2017
laskesen et al. 2020 Chzhen & Schreuder,
2020 Baharlouei et al.
2020 Jiang et al.
2020
Grari et al.
2019
See the recent paper by |Hort et al. (2022) for a more comprehensive literature survey.
While in-procesiing approaches generally give rise to better tradeoffs between fairness violation and perfor- mance, existing approaches are mostly incompatible with stochastic optimization. This paper addresses this problem in the context of fair (non-binary) classification with discrete (non-binary) sensitive attributes. See Table for a summary of the main differences between FERMI and existing in- in-processing methods.
Our main contributions are as follows:
1. For each given fairness notion (demographic parity, equalized odds, or equal opportunity), we formulate an objective that uses ERMI as a regularizer to balance fairness and accuracy (Eq. (FRMI obj.)), and derive an empirical version of this objective (Eq. (FERMI obj-)). We propose an algorithu (Algorithun for solving each of these objectives, which is the first stochastic in- ir-prrcessing fairness algorithm with guaranteed convergence. The main property needed to obtain convergent stochastic algorithm is to derive a (stochastically) unbiased estimator of the gradient of the objective function. The existing stochastic
fairness al algorithms by ZZhang et al. (2018) Mary et al. (2019) ; Prost et al. (2019 ; Cho et al. (2020a bb are not guaranteed to converge since there is no straightforward way to obtain such unbiased estimator of the gradients for their fairness regularizers For any minibatch size (even as small as 1), we prove (Theorem
that our algorithm converges to an approximate solution of the empirical objective (Eq. FERMI obj.)
2. We show that if the number of training examples is large enough, then our algorithm (Algorithm converges to an approximate solution of the population-level objective (Theorem 2 The proofs of these convergence theorems require the development of novel techniques (see e.g. Proposition and Proposition 2), and the resourceful application of many classical results from optimization, probability theory, and statistics.
3. We demonstrate through extensive numerical experiments that our stochastic algorithm achieves superior fairness-accuracy tradcoff curves against all comparable basclines for demographic parity, equalized odds, and equal opportunity. In particular, the performance gap is very large when ménibalch size is small (as is practically necessary for large-scale problems) and the number of sensitive attributes is large.
Nevertheless, Lxo distance for measuring fairness violation in our numerical experiments, since Loo is broadly used. 2e suspect might be possible to derive a provably convergent stochastic algorithm from the framework in Prost al. 2019: using our tochniques, but their approach limited binary classification with binary sensitive attributes. By contrast, provide (empirical and population-level) convergence guarantees for our algorithm with any number of sensitive attributes and any number of classes.
3

---

Published in Transactions on Machine Learning Research (11/2022)
2 Fair Risk Minimization through ERMI Regularization
In this section, we propose a fair learning objective (Eq. (RMM obj.)) and derive an empirical variation (Eq. (FER obj.)) of this objective. We then develop a stochastic optimization algorithm (Algorithm D that we use solve these objectives, and prove that our algorithm converges to an approximate solution of the two objectives.
Consider a Iearner who trains a model to make a prediction, Y, c.g., whether or not to extend loan, supported on [m] := (l...., m]. The prediction is made using a set of features, X, e.g,, financial history features. Assume that there is a set of discrete sensitive attributes, S, e.g., race and sex, supported on [R]-
We now define the fairness violation notion that we will use enforce fairness in our model.
Definition 1 (ERMI- exponential Rényi mutual information). We define the exponential Rérayi matual information between random variables & and S with joint distrituution Pp.s and marginals pq: ps by:
Ps. (P.) pp (Y)sss()))
1-1= E E s SELM) PE(N Pp(i)s((r)
DR(P; = EE
-.
(ERMT)
Definition [ is what we would use if demographic parity were the fairness notion of interest. If instead one wanted to promote fairness with respect to equalized odds or equal opportunity, then it is straightforward to modify the definition by substituting appropriate conditional probabilities for py.s Pp: and ps in Eq. (ERM)) see Appendix B In Appendix B we also discuss that ERMI the diverreee (which is an f-divergence
between the joint distribution, ps.s. and the Kronecker product of marginals, Pp @ ps (Calmon et al. 2017a
In particular, ERMI is non-negative, and zero if and only if demographic parity (or equalized odds or equal
opportunity, for the conditional version of ERMI) is satisfied. Additionally, we show in Appendix that ERMT provides an upper bound on other commonly used measures of fairness violation: Shannon mutual information (Cho et al. 2020a), Rényi correlation (Baharlouei et al., 2020), Lg fairness violation (Kearns
ct al. 2018 Hardt et al. 2016). Therefore, any algorithm that makes ERMI small will also have small fairness violalion with respect lo these other notions.
We can now define our fair risk minimization through exponential Rényi mutual information framework] 3
min { FRMM(O) == L()) + {)
(FRMI obj.)
0)] for given loss function é (e.g. L2 loss or cross entropy loss); À >>> 0 is a scalar balancing the accuracy versus fairness objectives; and Yo is the output of the learned model (i.e. the predicted label in classification task). While (x f(x: 0) inherently depends on X and 0, in the rest of this paper, we sometimes leave the dependence of γ on X and/or 0 implicit for brevity of notation. Notice that we have also left the dependence of the loss on the predicted outcome Y = (x( implicit.
where (())
As is standard, we assume that the prediction function satisfies (0. = j|x) = F, where F(O X) = (0, x),...,, (m. (0,00.)) x))² E [0, 1mm is differentiable in O and Σn= X) could represent the probability label given by a logistic regression model or the output of a neural network after softmax layer. Indeod, this assumption is natural for most classificrs. Further, even classifiers, such as SVM, that are not typically expressed using probabilities can often be well approximated by a classifier of the form F(Y X)
=1 For example,
x = F, X), e.g. by using Platt Scaling (Platt et al. 1999 Niculescu-Mizil &
Caruana 2005).
The work of Mary et al.
2019) considered the same objective Eq. FRMI obj.). and tried to empirically solve it through a kernel approximation. We propose a different approach to solving this problem, which we shall describe below. Easentially, we express ERMI as a "max" function (Proposition IT. which enables us to re-formulate Eq. (FRMI obj.) (and its empirical counterpart Eq. FEEMI obj.)) as stochastic min-max
IIn this scction, present all results in the context of demographic parity, leaving off all conditional expectations for clarity of presentation. The bygoortthmresulhhh are readily extended equalized adds and equal opportunity by using the conditional version of Eq. [ERM]) (which described Appendix B): we use these resulting algorithms for numerical experiments.
4

---

Published in Transactions on Machine Learning Research (11/2022)
optimization problem. This allows to use stochastic gradient descent ascent (SGDA) to solve Eq. (FRMI
obj.). Unlike the algorithm of Mary et al.] (2019), our algorithm provably converyes. Our algorithm also
empirically outperforms the algorithm of Mary et al. (2019), as we show in Sec. 3 and Appendix E.2
2.1 A Convergent Stochastic Algorithm for Fair Empirical Risk Minimization
In practice, the true joint distribution of (X, Y, Y) is unknown and we only have N samples at our disposal. Let == {x, 8,, yr. [xx; [e[N]] denote the features, sensitive attributes, targets, and the predictions of the model parameterized by 0 for these given samples. For now, we consider the empirical risk minimization (ERM) problem and do not require any assumptions on the data set (e.g. allow for different samples in D to be drawn from different, heterogeneous distributions). Consider the empirical objective
min FERMI(0) = Σ(0) +X)((6(X))5))
(FERMI obj.)
where E(0) :- $ Ein ((x,, yes 0) is the empirical loss and
4
Ar P p (s)()
Pv. (t. Po (i)
DR(P, 5) =E
Σ
j'(mm] rE(k]
is empirical ERMI with P denoting empirical probabilities with respect to D: ps(r) = #े4 Em= (m;=r)3 pb() N x:); and f (1.r = + 1 x:)si,r for j E [m],r E [k). We shall see (Proposition that empirical ERMI is a good approximation of ERMI when N is large. Now, it is straightforward to derive an unbiased estimate for ((0) via 1
[B] ΣiEB (xx, ys: 0) where B S [N] is a random minibatch of data points drawn from D. However, unbiasedly estimating Dn(P,S) in the objective function Eq. (FERMI obj.) with [ < N samples is more difficult. In what follows, we present our approach to deriving statistically unbiased stochastic estimators of the gradients of DR(Ý, S) given a random batch of data points B. This stochastic estimator is key to developing a stochastic convergent algorithm for solving Eq. (FEM obj.). The key novel observation that allows us to derive this estimator is that Equation FERMI obj. can be writton as a min-mar optimization problem (see Corollary D. This observation, in turn, follows from the following result:
Proposition 1. For random variables Y and S with joint distribution pv.s: where Y [m], [k], we have
Dn(P:S)=
= max {- P 1-1/ -1}, WER
P. diag(pz 0g P. (sag()s())....$$((()) and (Pg.eles = Bt,s(i.)) with (p(1), for i E [m].j E [k]]
The proof is direct calculation, given in Appendix D Let [x,, 0) E {0. 1) and 8; (8 S e {0, 13 be the one-hot encodings of f(x;, 0) and 8;, respectively for i E [N]. Then, Proposition provides a useful variational form of Eq. FERM )bj.), which forms the backbone of our novel algorithmic approach:
Corollary 1. Let (xi, vi) be a random draw from D. Then, Eg. FERMI obj.) is equivalent to
min (F(O,W)== := [(0) (w
(1)
W WERkXm
where @ W) = Tr(W Tr(PP,WT) + T B.N P. 1/2 - 11 = N ΣN Wr (0, W) and
(e W) == Tr(WE x, 0) 0/ x ) 2TTH 0)s x 8; P 1-1122-- -
(Tiag (8, x;) Fm x;) WT) 2 TT ) [x,, si P.1///
Corollary Cimplies that for any given data set D, the quantity f(x,, yis 0) x5> W) is unbiased estimator of F(O. W) (with respect to the uniformly random draw of i [N]). Thus, we can use stochastic optimization (e.g. SGDA) to solve Eq. (FERM obj.) with any batch size 1 S [ S N, and the resulting algorithm will
5

---

Published in Transactions on Machine Learning Research (11/2022)
Algorithm 1 FERMI Algorithm
1: Input: 0 E Rdd, 8 = - E Rkxm step-sizes (no, Mw), fairness parameter À 2 0, iteration number T, minibatch sizes B.|.t {0,1,-. . W
Frobenius norm ball of radius D around 0 E R for D
given in Appendix D
2: Compute P, 1/2 (1) 1/2 ..... Ps (s()))
T 3: for L = 0, 1,..,...7 do
4: Draw a mini-batch Br of data points {(x,, Ss yi KeB
5: Set 0++1 + 0-
1 - IVEB,
Vol(x, yu; 0²) + Ao W*).
6: Set W++1 Nww W + 22m
1il EiEB,
(xx 0) x.] 1/1
(ilsss"((;;))) (x; 0x sil)
end for
8: Pick E uuiffrrmly at random from {I,...,TT T).
Return: 0
be guaranteed to converge since the stochastic gradients are unbiased. We present our proposed algorithm, which we call FERMI, for solving Eq. (FERM obj.) in Algorithm [
Note that the matrix P, 1/2 depends only the full data set of sensitive attributes {s sx and has no dependence on 0, and can therefore be computed just once, in line 2 of Algorithm [ On the other hand, the quantities (x (x,, 0)*|x and [x;; 0)s; x,, st] depend on the sample (x,, 84, De) that is drawn in a given iteration and on the model paramcters 0, and are therefore computed at each iteration of the algorithm.
Although the min-mas max problem Eq. FERMI obj.p that we aim to solve is unconstrained, we project the iterates Wt (in line 5 of Algorithm onto a bounded set W in order to satisfy a technical assumption that is nceded to prove convergence of Algorithm M We choose W to be sufficiently large ball that contains W (0) == arg maxw F(0, W) for every O in some neighborhood of 0* arg ming maxw F(O, W),
that Eq.
F FERMI ) is equivalent to
min max () =(() Z(0)+ AA 0 WEW
See Appendix D for details. When applying Algorithm ① in practice, it is not necessary to project the iterates; e.g. in Sec. 3 we obtain strong empirical results without projection in Algorithm I
Since Eq. FERMI obj.) is potentially nonconvex in 0, global minimum might not exist and even computing local minimum is NP-hard in general (Murty & Kabadi 1985). Thus, is standard in the nonconvex optimization literature, we aim for the milder goal of finding an approximate stationary point of Eq. (FERM obj.). That is, given any e > O, we aim to find a point 0* such that B/]/FERM((0"] S e, where the expectation is solely with respect to the randomness the algorithm (minibatch sampling). The following theorem guarantees that Algorithm [ will find such a point efficiently:
Theorem 1. (Informal statement) Let e <>0. O. Assume that f(x, y: and F(.,x) are Lipschitz continuous and differentiable with Lipschitz continuous gradient (see Appendia D for definitions), pos(i) > for all sensitive attributes j E [k] and by (1) 2 p for all labels I [m] and at every iterate 0¹. Then for any batch sizes I [Bll N,, Algorithm | converges to an e-first order stationary point of the Eg. FERMI obj.) objective in O (2.) stochastic gradient cvaluations.
The formal statement of Theorem I can be found in Theorem 3 in Appendix D Theorem implies that Algorithm I can efficiently achieve any tradcoff between fairness (ERMI) violation and (empirical) accuracy, depending on the choice of X o However, if smaller fairness violation desired (i.e. if larger X is chosen), then the algorithm needs to run for more iterations (see Appendix D.. The proof of Theorem
follows from Corollary 1 and the observation that vi is strongly concave in W (see Lemma 11 Appendix
We overload notation slightly here and use E to denote expectation with respect to the empárical (Joint) distribution. Naaelly, bounded H eusures that the gradient of F is Lipschitz contimuuu every iterate and the variance of the stochastic gradients bounded.
6hhis sentence accurate the degree that an approximate stationary point of the non-convex objective Eq. corresponds to approximate risk minimizer.
FERMI obj.
6

---

Published in Transactions on Machine Learning Research (11/2022)
This implics that Eq. is nonconvex-strongly concave min-max problem, so the convergence guarantee
of SGDA {Lin et al. 2020) yields Theorem 1V The detailed proof of Theorem [ is given in Appendix D Increasing the batch size to ()) improves the stochastic gradient complexity to ()) On the other
hand, increasing the batch size further to |B|] = N results in a deterministic algorithm which is guaranteed
to find a point 0* such |FERMT(0))) S e (no expectation) in O(e iterations (Lin et al. 2020 Theorem
4.4). Ostrovskii et al.] 2020 Remark 4.2); this iteration complexity has the optimal dependence on c (Carmon eet al., 2020; Zhang et al. 2021). However, like existing fairness algorithms in the literature, this full-batch variant is impractical for large-scale problems.
The condition by 2 μ in Theorem 1 is assumed in order to ensure strong concavity of F(O',) at every iterate 8', which leads to the ((c-)) convergence rate. This assumption is typically satisfied in practice: for erample, if the iterates 6f remain in a compact region during the algorithm and the classifier uses softmar, then by (1) 2 μ O. Having said that, it is worth noting that this condition is not absolutely necessary to ensure convergence of Algorithm 1. Even if this condition doesn't hold, then Eq. (1) is still conconuee-cconcvre min-maz problem. Hence SGDA still converges to an e-stationary point, albeit at the slower rale of ((==)) [Tim el al. 2020). Alternatively, one can add a small lg regularization term to the objective to enforce strong concavity and get the fast convergence rate of O)c
Remark 1.
2.2 Asymptotic Convergence of Algorithm I for Population-level FRMI Objective
So far, we have let N 2 be arbitrary and have not made any assumptions the underlying distribution(s) from which the data was drawn. Even so, we showed that Algorithm | always converges to a stationary point of Eq. (ERMI obj.). Now, we will show that if D contains i. samples from an unknown joint distribution 'D and N >> 1, then Algorithm [ converges to an approximate solution of the poyulation risk minimization problem Eq. FRMI obj.p. Precisely, we will use a one pass sample-wire without--replacmment ("online") variant of Algorithum to obtain this population loss guarantee. The one-pass variant is identical to Algorithm []] except that: a) once we draw a batch of samples B,, we remove these samples from the data set so that they are never re-used; and b) the for-loop terminates when we have used all samples.
Thcorem 2. Let c > O. Assume that E(x, and F(,x)) x) are Lipschitz continuuus and differentiable with Lipschilz continuous gradient, and that minre[k] (r( > O. Then, there erists N E such that if 2 N and D ~ D", then a one-pass nample-without-rppll
variant of Algorithm 1 converges to on e-first order stationary point of the Eq. (FRMI obj.) objective in O (+) stochastic gradient evoluations, for any batch sizes Bel.
Theorem 2 provides a guarantee on the bairness/accuraas loss that can be achieved on unseen "test data." " This is important because the main goal of (fair) machine learning is to (fairly) give accurate predictions on test data, rather than merely fitting the training data well. Specifically, Theorem 2 shows that with enough (i..d.) training examples at our disposal, (one-pass) Algorithm | finds an approximate stationary point of the population-level fairness objective Eq. (FRMI obj. Furthermore, the gradient complexity is the same it was in the empirical case. The proof of Theorem 2 will be aided by the following result, which shows that th is an asymptotically unbiased estimator of V, where maxw (0, W) equals ERMI:
Proposition 2. Let ( = {xi, 85, M: be drawn i.i.d. from an unknown joint distribution D. De- -1/2"
((0) (0. = (ly (xx, 0) (x.WT) +2 (WE|)(x; 0)S ( si] (P())
note vi
1,, where
p(m) = = -リ diag (,,=1),-' (()))))) Denote v(0, W) - TTTT) + 2 TT Ps Pg.oP,1/3 1, where Pg n = diag(E.F E,Fi (0.)).. (f)),, (Pg.s)irr = Ex.,s, (0, for j E [m], E [k], and
P. = diag ddag(Ps(1)) ..., ,P)(k)) Assume Ps(r) > 0 for all r E [k)]. Then,
max V(O, W/) = DR(P(();;)))) W
and
lim B m (0, W)] = v(O,
A faster convergence rate of 3)) could be obtained by using the (more complicated) SREDA method of Luo et instead of SGDA to solve FERMI objective. We omit the details here.
2020
7

---

Published in Transactions on Machine Learning Research (11/2022)
The proof of Proposition 2 is given in Appendix D.1 its proof, while the second claim is proved using the strong law of large numbers, the continuous mapping theorem, and Lebesgue's dominated convergence theorem.
The first claim is immediate from Proposition 1 and
Proposition 2 implies that the empirical stochastic gradients computed in Algorithm | are good approximations of the true gradients of Eq. (FRMI obj.p. Intuitively, this suggests that when we use Algorithm | to solve
the fair ERM problem Eq. (FERMI obj.j, ; the output of Algorithm [ will also be an approximate solution
of Eq. ( FRMI ) While Theorem 2 shows this intuition does indeed hold, the proof of Theorem 2 requires
additional work. A reasonable first attempt at proving Theorem 2 might be to try to bound the expected
distance between the gradient of FRMI and the gradient of FERMI (evaluated at the point i that is output by Algorithm D via Danskin's theorem (Danskin 1966) and strong concavity, and then leverage Theorem to conclude that the gradient of FRMI must also be small. However, the dependence of on the training data prevents us from obtaining a tight enough bound on the distance between the empirical and population gradients at 0 Thus, we take a different approach to proving Theorem 2 in which we consider the output of two different algorithms: one is the conceptual algorithm that runs one-pass Algorithm [ as if we had access to the true sensitive attributes P. (* ("Algorithm A"); the other the realistic one-pass Algorithm that only uses the training data (* ("Algorithm B"). We argue: 1) the output of the conceptual algorithm is stationary point of the population-Ievel objective; and 2) the distance between the gradients of the population-level objective at A and 0g is small. While 1) follows easily from the proof of Theorem D and the online-to-bated conversion, establishing 2) requires careful argument. The main tools we use in the proof of Theorem 2 are Theorem 1 Proposition 2 Danskin's theorem, Lipschitz continuity of the arg max function for strongly concave objective, the continuous mapping theorem, and Lebesgue's dominated convergence theorem: see Appendix D.I for the detailed proof.
Note that the bnlthe-to-batch conversion used to prove Theorem 2 requires convergent stochastic oplimization algorithm; this implies that our arguments could not be used to prove an analogue of Theorem 2 for existing fair learning algorithmss, since existing convergent fairness algorithms are not stochastic. An alternate approach to bounding the "generalization error" of our algorithm would be to use standard covering/ evverngg/uiffrrm uniform convergence
argument. However, this approach would not yield as tight a guarantee as Theorem 2 Specifically, the accuracy and/or gradient complexity guarantee would depend on the dimension the space (i.e. the number of model parameters), since the covering number depends (exponentially) on the dimension. For large-scale problems with a huge number of model parameters, such dimension dependence is prohibitive.
As previously mentioned, we can interpret Theorem 2 as providing a guarantee that Algorithm generalizes well, achieving small fairness violation and test error, even on unseen "test" examples as long as the data is i..d. and N is sufficiently large. In the next section, we empirically corroborate Theorem 2 by evaluating the fairness-accuracy tradcoffs of the FERMI algorithm (Algorithm I in several numerical experiments.
3 Numerical Experiments
In this section, we evaluate the performance of FERMI in terms of the fairness violation vs. test error for different notions of fairness (e.g. demographic parity, equalized odds, and equality of opportunity). To this end, we perform diverse experiments comparing FERMI to other state-of-the-art approaches on several benchmarks. In Section 3.1 we showcase the performance of FERMI applied to a logistic regression model on binary classification tasks with binary sensitive attributes on Adult, German Credit, and COMPAS datasets. In Section 33. we utilize FERMI with a convolutional neural network base model for fair (to different religious groups) toxic comment dotection. In Scction 3.3 we explore fairnoss in non- non- binary classification with non-binary sensitive attributes. Finally, Section 3.4 shows how FERMT may be used beyond fair empirical risk minimization in domain generalization problems to learn model independent of spurious features.
3.1
Fair Binary Classification with Binary Sensitive Attributes using Logistic Regression
3.1.1
Benchmarking full-batch performance
In the first set of experiments, we use FERMI to learn a fair logistic regression model on the Adult dataset. With the Adult data set, the task is to predict whether or not a person carns over $50k annually without discriminating based the sensitive attribute, gender. We compare FERMI against state-of-the-art in-
8

---

Published in Transactions on Machine Learning Research (11/2022)
processing full-batch ((B)
N) baselines, including (Zafar al. 2017
Feldman et al.
2015 Kamishima
et al. 2011: Jiang et al. 2020 Hardi et al. 2016; Prost et al.] 2019 Ba Baharlouei el al. 2020
Rezaei et al.
Cho et al. 2020b). Since the majority of existing fair learning algorithiss cannot be < N, these experiments allow us to benchmark the performance of FERMI against 8 wider range of baselines. To contextualize the performance of these methods, we also include a Naïve Baseline that randomly replaces the model output with the majority label (0 in Adult dataset), with probability p (independent of the data), and sweep p in [0. 1]. At one end (p - 1), the output will be provably fair with performance reaching that of naive classifier that outputs the majority class. At the other end (p 0). the algorithm has no fairness mitigation and obtains the best performance (accuracy). By sweeping
2020: Donini et al. 2018
implemented with B
P. we obtain a tradeoff curve between performance and fairness violation.
Equality of Opportunity Vielation
Demographic Parity Violation
0.20
Equaliize Odds violation
Detertoul 2009
0.10
Zehecital 2017
Don'
0.10
Zolar et 2017
0.15
0.08
Ta 2020
0.08
Ca 2006
Hottal 10:00
15:09
0.10
0.06
0.06
0.04
0.04
0.05
0.02
0.02
0.00
0.17 0.18 0.19 0.20 0.21 0.22 0.23 0.24
0.17
0.18 0.19 0.20 0.21 0.22
0.17
0.18 0.19 0.20 0.21 0.22
Test Error
Test Error
Test Error
Figure 1: Accuracy/Faii /Fairness trade-off of FERMI and several state-of-the-art in-processing approaches on Adult dataset. FERMI offers the best fairness vs. accuracy tradeoff curve in experiments against baselines. Rezaei et al. (2020) only allow for a single output and do not yield tradeoff curve. Further, the algorithms by Mary et al. (0019) and Baharlouei et al. (2200) are equivalent in this binary setting and shown by the red curve. In the binary/binary /binary setting, FERMI, Mary et al. (2019) and Bahharlouii et al. (2020) all try to solve the same objective Eq. (FRMT obj. However, the empirical formulation Eq. (ERRI obj. and FERMI algorithm that we use results in better performance, even though are using full-batch for all baselines in this experiment.
In Fig. D we report the fairness violation (demographic parity, equalized odds, and equality of opportunity violations) vs. test error of the aforementioned in-processing approaches on the Adult dataset. The upper left corner of the tradeoff curves coincides with the unmitigated baseline, which only optimizes for performance (smallest test error). As can be seen, FERMI offers a fairness- accuracy tradeoff curve that dominates all state-of-the-art baselines in each experiment and with respect to each notion of fairness, even in the full batch setting. Aside from in in-processing approaches, we compare FERMI with several procpsscs and post-processing algorithms on Adult, German Credit, and COMPAS datasets Appendix [E.5 where we show that the tradeoff curves obtained from FERMI dominate that of all other baselines considered. See Appendix E for details the data sets and experiments.
Tt is noteworthy that the empirical objectives of Mary et al. 2019) and Baharlouei et al. 20200) are exactly the same in the binary/binary setting, and their algorithms also coincide to the red curve in Fig, This is because Exponential Rényi mut ual information is equal to Rényi correlation for binary largels and/or /or binary sensitive attributes (see Lemma 2), which is the setting of all experiments in Sec. 3.1 Additionally, like us, in the binary/binard setting these works are trying to empirically solve Eq. (FRMI obj.) alboit using different estimation techniques; i.e., their empirical objective is different from Eq. FERMI obj. This demonstrates the effectiveness of our empirical formulation (FFRMT obj-) and our solver (Algorithm 1 even though we are using all baselines in full batch mode in this experiment. See Appendix E.5 for the complete version of Fig. D which also includes pre-procossing and post-processing basclines.
Fig. in Appendix E illustrates that FERMI outperforms baselines in the presence of noisy outliers and class imbolance. Our theory did not consider the role of noisy outliers and class imbalance, so the theoretical investigation of this phenomenon could be an interesting direction for future work.
3.1.2 The effect of batch size on bairess//cccuras tradeoffs
Next, we evaluate the performance of FERMI on smaller batch sizes ranging from 1 to 64. To this end, we compare FERMI against several state-of state-of-the-art the-art in-processing algorithms that permit stochastic implementation for demographic parity: (Mary al. 2019), (Baharlouei et al. 2020), and (Cho et al.

---

Published in Transactions on Machine Learning Research (11/2022)
2020b). Similarly to the full batch setting, for all methods, train logistic regression model with respective regularizer for each method. We use demographic parity Loo violation (Definition T0 to measure demographic parity violation. More details about the dataset and experiments, and additional experimental results, can be found in Appendix E
0.25
2.25
0.20
0..0
0.20
lave
0.30
0.50
0.10
0.10
0.5
0:25
5.00
3.05
0.00
0.30
2.00
3.00
319
035
0.16
022 Text
5:36 037
1:3 022 022
378 036
025 634
Dror
Text Ener
Teat Erer
Figure 2: Performance of FERMI, Cho et al. (200a)) Mary et al. (2019) Baharlouei et al. (020) with different
batch-sizes on Adult dataset. FERMI demonstrates the best accuracy /fainess tradeoff across different batch sizes.
Fig. 2 shows that FERMI offers superior fairness-accuracy tradeoff curve against all baselines, for each tested batch size, empirically confirming Theorem [ as FERMT is the only algorithm that is guaranteed to converge for small minibatches. It is also notcworthy that all other basclines cannot beat Naive Baseline when the batch size is very small, e.g., [B] = 1. Furthermore, FERMI with B = fairness-accuracy tradeoff as the full batch variant.
4 almost achieves the same
3.1.3 The effect of missing sensitive attributes on fairness/accuragy tradeoffs
Sensitive attributes might be partially unavailable in many real-world applications duc to legal issues, privacy concerns, and data gathering limitations (Zhao et al. 2022 Coston et al. 2019). Missing sensitive altributes
make fair learning tasks more challenging in practice.
The unbiased nature of the estimator used in FERMI algorithm mo- livates that it may be able to handle cases where sensitive altributes are partially available and are dropped uniformly at random. As a case study on the Adult dataset, we randomly masked 90% of the
Demographic Parity Parityy Violation
0.25
May d.l. 2013
sensitive attribute (i.e., gender entries). To estimate the fairness regularization term, we rely on the remaining 10% of the training samples (~ 3k) with sensitive attribute information. Figure 3 depicts the tradcoff between accuracy and fairness (demographic parity) vio- lation for FERMI and other baselines. We suspect that the superior recuracy-fairness tradcoff of FERMI compared to other approaches is due to the fact that the estimator of the gradient remains unbiased since the missing entries are missing completely at random (MCAR). Note that the Naïve Baseline is similar to the one implemented in the previous section, and Full-sensitive FERMI is an oracle method that applies FERMI to the data with no missing altributes (Tor com- parison purposes only). We observe that FERMI achieves a slightly worse fairness-accurast tradcoff compared to Full-sensitive FERMI oracle, whereas the other baselines are hurt significantly and are only narrowly outperforming the Naïve Baseline.
FEMM
0.15
0.10
0.05
0.00
0.10
0.17
0.18 0.19 0.20 0.21 0.22 0.23
Test Error
Figure 3: Performance of FERMI and other state-of-the-art approaches on the Adult dataset where 90% of gender en- tries missing. Full-sensitive FERMI is obtained by applying FERMI on the data without any missing entries.
3.2 Fair Binary Classification using Neural Models
In this experiment, our goal is to showcase the efficacy of FERMI in stochastic optimization with neural network function approximation. To this end, apply FERMI, and (Mary et al. 2019) (which coincides with (Baharlouci et al. dataset where the underlying task is to predict whether a given published comment in social media is toxic. The sensitive attribute is religion that is binarized into two groups: Christians in one group; Muslims and Jews in the other group. Training neural network without considering fairness Icads to higher false positive rate for the Jew-Muslim group. Figure 4 demonstrates the performance of FERMI, MinDiff (Prost et al.,
PProst et al. 2019), (Baharlouei et al. 2020), 2020)) to the Toxic Comment Classification
2019) Baharlouei et al. (2020), and naïve baseline on two different. batch-sizes: 128 and 16. Performance is
10

---

Published in Transactions on Machine Learning Research (11/2022)
Batch Size = 128
Batch Size 16
rotM: Froe mn 2010
0.34
Pront 4t 2015
rals Positvv Rate GAD
0.32
False Postive Rate Gap
Helec
larvee Beeirie
0.10
0.00
0.00
0.06
0.06
0.04
0.02
0.04
0.05
0.06
0.07
0.08
0.09
0.10
0.04
0.05
0.00
0.07
0.08
0.08
0.10
0.11
Test Error
Test Error
Figure Fair toxic comment detection with different batch sizes. For [B] = 128, the performance of (Prott et al, 2019) and FERMI are close to each other, however, when the batch size is reduced to 16, FERMI demonstrates a better fairness/ performance trade-off. The performance and fairness are measured by the test error and the false positive gap between different religious sub- groupsropp (Christians vs Muslim-Jews), respectively.
0.385
Cho DuZ26
aeharicus 2020
violation 0.02
0.365
0.360
Choetal, 20200
DP violation - 0.03
0.556
0.355
violation Bansriouni 2000
Cro et 2330b
0.360
0.365
Gaharous 2600
Chu 20020
vio 0.05
0.370
0.375
PEIM
Marv et 2010
0.350
0.355
el. 2019
0.355
0.110
Mary et 20.0
PEROI
0.350
0.355
PESMI
Mary ai. 2015
0.305
0.345
0.360
0.345
0.345
0.340
0.355
0.340
0.340
0.315
0.335
0.335
0.330
11
15
15
17
0.330
15
0.330
9
15
35
37
9
11
15
15
17
ED violatian
EC vislacion 0.03
vialation
ED violation 0.05
Baharicus at. 2020
Banerrouei at 2020
Banariousi ctal. 2000
Bahar ous at 2000
Cho st al. 20226
cto 20200
Cna at al, 2230:
0.300
Chu at al. 20206
Mery atal 2019 TEAM
May et 2319 1E00I
Hary et 2015
0.355
Mar et 2015
0:380
0.350
PEIMI
0.355
0.340
0.335
0.330
0.360
0.330
No. 11
Figure 5: Comparison between FERMI, Mary et al. Communities dataset. (Mary et al. 2019) outperforms (Baharlouei et al. 2020 Cho et al. 2020b), which believe could be attributed to the effectiveness of ERMI as a regularizer. FERMI outperforms Mary et al. (2019). This can be attributed to our empirical formulation Eq. FERMI obj.) and unbiased stochastic optimization algorithm.
2019 [aahrlouei et al. 20020), and Cho et al. 2020b) on
measured by the overall false positive rate of the trained network and fairness violation measured by the false positive gap between two sensitive groups (Christians and Jews-Muslims). The network structure is exactly same the one used by MinDiff (Prost al. 2019). We can observe that by decreasing the batch size, FERMI maintains the best fairness-accuracy tradeoff compared to other baselines.
3.3 Fair Non-binary Classification with Multiple Sensitive Attributes
In this section, we consider a non-binary classification problem with multiple binary sensitive attributes. In this case, we consider the Communtiies and Crime dataset, which has 18 binary sensitive attributes in total. For our experiments, we pick a subset of 1, 1,2,3.... 18 sensitive attributes, which corresponds to We discretize the target into three classes (high, medium, The only basclines that we are aware of that can handle non-binary classification with multiple sensitive attributes are (Mary) et al. 2019). (Baharlouei et al. 2020), (Cho et al. 2020b), (Cho et al. 2020a), and (Zhang et al. 2018). We used the publicly available implementations of (Baharlouei et al. 2020 and (Cho et al. 2020b) and extended their binary classification algorithms to the non-binary setting.
() E (2.,,8....,
The results are presented in Fig. 5 where we use conditional demographic parity Leo violation (Definition 10 and conditional equal opportunity Loo violation (Definition 11] as the fairness violation notions for the two experiments. In each panel, we compare the test error for different number of sensitive attributes for fixed value of DP violation. It is expected that test error increases with the number of sensitive attributes, as we
11

---

Published in Transactions on Machine Learning Research (11/2022)
0.9
0.8
Banariouel ct Cha al., 2020h
Ralurkkad at, 2020
Cho at 20206
0.7
Mary et al, 2019 FERMI
0.7
FTTN
MO'y ct 2013
0.6
0.5
Test Frror
0.6
0.4
0.5
0.3
Test Error Training Enor
0.2
0.1
0.1
0.3
0
200
400
600
300
1000
1200
Batch Size
Figure 6: Domain generalization on Color MNIST (Li & Vasconcelos 2019) using in-process fair algorithms for demographic parity. Left panel: The dashed line the training error and the solid line is test error. As increases, fairness regularization results in a learned representation that is less dependent on color; hence training error increases while test error decreases (all algorithms reach a platean around = 8). We use B = 512 for all baselines. Right panel: We plot test error vs. batch size using optimized value of À for each algorithm selected via a validation set. The performance of baselines drops 10-20% as batch size becomes small, whereas FERMI is less sensitive to batch size.
will have a more stringent fairness constraint to satisfy. As can be seen, compared to the baselines, FERMI offers the most favorable test error vs. fairness violation tradeoffs, particularly as the number of sensitive attributes increases and for the more stringent fairness violation levels, e.g., 0.02
3.4 Beyond Fairness: Domain Parity Regularization for Domain Generalization
In this section, we demonstrate that our approach may extend beyond fair empirical risk minimization to other problems such as domain generalization. In fact, Li & Vasconcelos 2019 Lahoti et al (2020); Creager ct al. (2021) have already established connections between fair ERM and domain generalization. We consider the Color MNIST dataset (Li & Vasconcelos 2019), where all 60,000 training digits are colored with different colors drawn from a class conditional Gaussian distribution with variance o² around a certain average color for each digit, while the test set remains black and white. Li & Vasconcelos (2019) show that as o² + O, a convolutional network model overfits significantly to each digit's color on the training set, and achicves vanishing training error. However, the learned representation does not generalize to the black and white test. set, due to the spurious correlation between digits and color.
Conceptually, the goal of the classifier in this problem is to achieve high classification accuracy with predictions that are independent of the color of the digit. We view color as the sensitive attribute in this experiment, and apply fairness baselines for the demographic parity notion of fairness. One would expect that by promoting such independence through a fairness regularizer, generalization would improve (i.e. lower test error on the black and white test set), at the cost of increased training error (on the colored training set). We compare against. Mary et al. (2019) Baharlouei et al. (2020), and Cho al. (2020b) baselines in this experiment.
The results of this experiment are illustrated in Fig. G In the left panel, we see that with no regularization (A = 0), the test error is around 80%. As À increases, all methods achieve smaller test error while training error increases. We also observe that FERMI offers the best test error in this setup. In the right panel, we obscrve that decreasing the batch size results in significantly worse generalization for the three basclines considered (due to their biased estimatooss for the regularizer). However, the negative impact of small batch size is much less severe for FERMI, since FERMI uses unbiased stochastic gradients. In particular, the performance gap between FERMI and other baselines is more than 20% for B 64. Morcover, FERMI with minibatch size [B| 64 still outperforms all other baselines with |B] > 1, 000. Finally, notice that the test error achieved by FFRMT when = O is ~ 30,, as compared to more than 50% obtained using REPAIR (Li
& Vasconcelos 2019 for < 0.05.
8 $Sec. 3.4 demonstrated that using smaller batch sizes results much more pronounced advantages of FERMI over these baselines.
12

---

Published in Transactions on Machine Learning Research (11/2022)
4 Discussion and Concluding Remarks
In this paper, we tackled the challenge of developing a fairness-promoting algorithm that is amenable to stochastic optimization. As discussed, algorithms for large-scale MT. problems are constrained to use stochastic optimization with (small) minibatches of data in each iteration. To this end, we formulated an empirical objective ( FERMI obj.) using ERMI as regularizer, and derived unbiased stochastic gradient estimators. We proposed the stochastic FERMI algorithm (Algorithm [ for solving this objective. We then provided the first theoretical convergence guarantees for stochastic in-processing fairness algorithm, by showing that FERMI converges to stationary points of the empirical and population-level objectives (Theorem ] Theorem 2). Further, these convergence results hold even for non- non-binary sensitive attributes and non- non-binary target variables, with any minibatch size.
From an experimental perspective, we showed that PERMI leads to better fairnes--cceurecy tradeoffs than all of the state-of-the-ate art baselines on wide variety of binary and non-binary classification tasks (for demographic parity, equalized odds, and equal opportunity). We also showed that these benefits are particularly significant when the number of sensitive attributes grows or the batch size is small. In particular, we observed that FERMI consistently oulperforms Mary el al. (2019) (which tries to solve the same objective Eq. {FRMI obj-) by up to 20% when the batch size is small. This is not surprising since FERMI is the only algorithm that is guarantced to find an approximate solution of the fair Iearning objective with any batch size [B] 2 1. Also, we show in Fig. 7 that the lack of convergence guarantee of Mary et al. (2019) is not just due to more limited analysis: in fact, their stochastic algorithm does not. converge. Even in full batch mode, FERMI outperforms all baselines, including (Mary et al. 2019) (Fig. [ Fig. 5). In full batch mode, all baselines should be expected to converge to an approximate solution of their respective empirical objectives, so this suggests that our empirical objective Eq. (FEM obj-) is fundamentally better, in some sense, than the empirical objectives proposed in prior works. In what sense is Eq. (FERMI obj.) a better empirical objective (apart from permitting stochastic optimization)? For one, it is an asymptotically unbiased estimator of Eq. (FRMI 2 and Theorem suggests that FERMI algorithm outputs an approximate solution of Eq.. (FRMI obj.) for large enough N. By contrast, the empirical objectives considered in prior works do not provably yield an approximate solution to the corresponding population-level objective.
obj.) (by Proposition
The superior fairness fairness-accuracy accuracy tradeoffs of FERMI algorithm over the (full batch) baselines also suggests that the underlying population-level objective Eq. FRMI obj.) has benefits over other fairness objectives. What might these benefits be? First, ERMI upper bounds all other fairness violations (c.g. Shannon mutual information, L.g Tox) used in the literature: see Appendix C This implies that ERMI-regularized training yields a model that has small fairness violation with respect these other notions. Could this also somchow help explain the superior fairness-uccuracy tradcoffs achieved by FERMI? Second, the objective function Eq. (RRM obj-) is easier to optimize than the objectives of competing in-processing methods: ERMI is smooth and is equal to the trace of a matrix (sce Lemma 5 in the Appendix). which is easy to compute. Contrast this with the larger computational overhead of Rényi correlation used by [Bharlouei et al. (2020), for example, which requires finding the second singular value of a matrix. Perhaps these computational benefits contribute to the observed performance gains? We leave it as future work to rigorously understand the factors that are most responsible for the favorable accuracy tradeoffs observed from FERMI.
Broader Impact and Limitations
This paper studied the important, problem of developing practical machine learning (ML) algorithms that are fair towards different demographic groups (o.g. race, gender, age). We hope that the socictal impacts of our work will be positive, as the deployment of our FERMI algorithm may enable/help companies, government. agencies, and other organizations to train large-scale ML models that are fair to all groups of users. On the other hand, any technology has its limitations, and our algorithm is no exception.
One important limitation of our work is that we have (implicitly) assumed that the data set at hand is labeled accurately and fairly. For example, if race is the sensitive attribute and "likelihood of default on a loan" is the target, then we assume that the training data based on past observational data accurately reflects the financial histories of all individuals (and in particular does not disproportionately inflate the financial histories of racial minorities). If this assumption is not satisfied in practice, then the outcomes promoted
13

---

Published in Transactions on Machine Learning Research (11/2022)
by our algorithm may not be fair (in the philosophical sense) as the computed level of fairness violation might suggest. For example, if racial minorities are identified as higher risk for default on loans, they may be extended loans with higher interest rates and payments, which may in turn increase their likelihood of a default. Hence, might be even possible that our mitigation strategy could result in more unfairness than unmitigated ERM in this case. More generally, conditional fairness notions like equalized odds suffer from a potential amplification of the inherent discrimination that may exist the training data. Tackling such issues is beyond the scope of this work; c.f. Kilbertus et al. (2020) and Bechavod et al. (2019).
Another consideration that not addressed in this paper is the interplay between fairness and other socially consequential AI metries, such as privacy and robustness (e.g. to data poisoning). It is possible that our algorithm could increase the impact of data from certain individuals to improve fairness at the risk of leaking private information about individuals in the training data set (e.g. via membership inference attacks or model
inversion attacks). even if the data is anonymous (Fredrikson et al. 2015 Shokri et al.) 2017 Faizullabhoy
& Korolova 2018 Nasr al. 2019 Carlini et al.) 2021). Differential privacy (Dwork et al.) 2006) ensures
that sensitive data cannot. be leaked (with high probability), and the interplay between fairness and privacy
has been explored (see e.g. Jagielski et al. (2019); Xu al. (2019); Cummmings et al. (2019); Mozannar
ct al. [2020]: : Tran ct al. (2021a b). Developing and analyzing a diffcrentially private version of FERMI could be an interesting direction for future work. Another potential threat to FERMIItrained models is data poisoning attacks. While our experiments demonstrated that FERMT is relatively effective with missing sensitive attributes, we did not investigate its performance in the presence of label flipping or other poisoning attacks. Exploring and improving the robustness of FERMI another avenue for future research.
References
Alekh Agarwal, Alina Beygelzimer Miroslav Dudík, John Langford, and Hanna Wallach. A reductions approach to fair classification. In International Conference on Machine Learning, pp. 60 69. PMLR, 2018.
Sina Aghaei, Mohammad Javad Azizi, and Phebe Vayanos. Learning optimal and fair decision trees for non-diseriminative decision-making In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 1418 1426, 2019.
Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. ProPublica, 2016.
Sina Baharlouei, Maher Nouiehed, Ahmad Beirami, and Meisam Razaviyayn. Rényi fair inference. In ICLR, 2020.
Yahav Bechavod and Katrina Ligett. Penalizing unfairness in binary classification, arXiv preprint DXXiv:1077.00444 2017.
Yahav Bechavod, Katrina Ligett, Aaron Roth, Bo Waggoner, and Zhiwei Steven Wu. Equal opportunity in online classification with partial feedback. arXiv preprint arXiv:1902. 02242, 2019.
Richard Berk, Hoda Heidari, Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamie Morgenstern, Seth Neel, and Aaron Roth. A convex framework for fair regression. arXiv preprint arXiv: 1706. 02400, 2017.
Alex Beutel, Jilin Chen, Tulsce Doshi, Hai Qian, Allison Woodruff, Christinc Luu, Pierre Kreitmann, Jonathan Bischof, and Ed H Chi. Putting fairness principles into practice: Challenges, metrics, and improvements. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 453 459, 2019.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in neural information processing systems, pp. 4349 4357, 2016.
Flavio Calmon, Ali Makhdoumi, Muriel Médard, Mayank Varia, Mark Christiansen, and Ken Duffy. Principal inertia components and applications. IEEE Transactions on Information Theory, 63(8):5011- 5038, 2017a.
14

---

Published in Transactions on Machine Learning Research (11/2022)
Flavio Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R. Varshney. Optimized pre-processing for discrimination prevention. In Advonces in Neural Information Processing Systems, pp. 3992 2001-401,1- 2017b.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulaar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633 2650, 2021.
Yair Carmon, John C Duchi, Oliver Hinder, and Aaron Sidford. Lower bounds for finding stationary points i. Mathematical Programming, 184(1):71-120, 2020.
Jaewoong Cho, Gyeongjo Hwang, and Changho Suh. A fair classifier using mutual information. In 2020 IEEE International Symposium on Information Theory (ISIT), pp. 2521-2526. IEEE, 2020a.
Jacwoong Cho, Gycongjo Hwang, and Changho Suh. A fair classifier using kernel density estimation. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neurol Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020b.
Evgenii Chzhen and Nicolas Schreuder. A minimax framework for quantifying risk-fairness trade-off in regression. arXiv preprint orXiv:2007.14263, 2020.
Amanda. Coston, Karthikeyan Natesan Ramamurthy, Dennis Wei, Kush R Varshney, Skyler Speakman, Zairah Mustahsan, and Supriyo Chakraborty. Fair transfer learning with missing protected attributes. In Proceedings of the 2019 AAAT/ACM Conference on AI, Ethics, and Sociely, pp. 91 98, 2019.
Thomas M Cover and Joy A Thomas. Information theory and statistics. Elements of Information Theory, (1):279-335, 1991.
Elliot Creager, Jörn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In International Conference on Machine Learning, pp. 2189-2200. PMLR, 2021.
Imre Csiszár and Paul C Shiclds. Information theory and statistics: A tutorial. Now Publishers Inc, 2004.
Rachel Cummings, Varun Gupta, Dhamma Kimpara, and Jamie Morgenstern. On the compatibility of privacy and fairness. In Adjunct Publication of the 27th Conference User Modeling, Adaptation and Personalization, pp. 309-311 315, 2019.
John M Danskin. The theory of max-min, with applications. SIAM Journal on Applied Mathematics, 14(4): 641-664, 1966.
Amit Datta, Michael Carl Tschantz, and Anupam Datta, Automated experiments on ad privacy settings. Proceedings on privacy enhancing technologies, 2015(1):92-112) 2015.
A. Dembo and O. Zeitouni. Large deviations techniques and applications. Springer Science & Business Media, 2009.
Michele Donini, Luca Oneto, Shai Ben-David, John S Shawe-Taylor, and Massimiliano Pontil. Empirical risk minimization under fairness constraints. In Advances in Neural Information Processing Systems, pp. 2791-2801, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words: Transformers for image recognition at scale. ar preprint arXi:2010.11929, 2020.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryplography conference, pp. 265 284. Springer, 2006.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pp. 214-226, 2012.
15

---

Published in Transactions on Machine Learning Research (11/2022)
Irfan Faizullabhoy and Aleksandra Korolova. Facebook's advertising platform: New attack vectors and the need for interventions. Xiv preprint orXin: 10099, 2018.
Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasuubamannaa. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and dala mining, pp. 259 268, 2015.
Benjamin Fish, Jeremy Kun, and Adám D Lelkes. Fair boosting: case study, In Workshop on Fairness, Accountability, and Transparency in Machine Learning. Citeseer, 2015.
Benjamin Fish, Jeremy Kun, and Ádám D Lelkes. A confidence- approach for balancing fairness and accuracy. In Proceedings of the 2016 SIAM International Conference on Data Mining, pp. 144-152. SIAM, 2016.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communécations Security, pp. 1322-1333, 2015.
Hans Gebelein. Das statistische problem der korrelation als variations-und eigenwertproblem und sein zusaum menhang mit der ausgleichsrechnung ZAMM-Journal of Applied Mathematics and Cecchanics/Zeitschhi für Angewandte Mathematik und Mechanik, 21 21(6):644-379, 1941.
Vincent Grari, Boris Ruf, Sylvain Lamprier, and Marcin Detyniecki. Fairness-aware neural Réyni minimization for continuous features. arXiv preprint orX1::1911.04929, 2019.
Vincent Grari, Oualid El Hajouji, Sylvain Lamprier, and Marcin Detyniecki. Learning unbiased representation via Rényi minimization. arXiv preprint orXiv:2009.0323 031183, 2020.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. In Advances in neural information processing systems, pp. 3315 3323, 2016.
Hermann O Hirschfeld. A connection between correlation and contingency. In Proceedings of the Cambridge Philosoppical Sociely, volume 31, pp. 520-50 00-,2,4, 1935.
Max Hort, Zhenpeng Chen, Jie M Zhang, Federica Sarro, and Mark Harman. Bia mitigation for machine learning classifiers: A comprehensive survey. ar preprint arXiv:2207.0 07068, 2022.
Matthew Jagielski, Michael Kearns, Jieming Mao, Alina Oprea, Aaron Roth, Saeed Sharifi- Malvajiii,,,jrrdi,, and Jonathan Ullman. Differentially private fair learning. In International Conference on Machine Learning, pp. 3000-3008. PMLR, 2019.
Ray Jiang, Aldo Pacchiano, Tom Stepleton, Heinrich Jiang, and Silvia Chiappa. Wasserstein fair classification. In Uncertainty in Artificial Intelligence, pp. 862 872. PMLR, 2020.
Faisal Kamiran, Toon Calders, and Mykola Pechenizkiy. Discrimination aware decision tree learning. In 2010 IEEE International Conference on Data Minning, pp. 869 874. IEEE, 2010.
Toshihiro Kamishima, Shotaro Akaho, and Jun Sakuma. Fairness-awar aware learning through regularization approach. In 2011 IEEF 11th International Conference on Data Mining Workshops, pp. 643-650. IEEE, 2011.
Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. In International Conference on Machine Learning, pp. 2564-22440 22722, 2018.
Niki Kilbertus, Manuel Gomez Rodriguez, Bernhard Schölkopf. Krikamol Muandet, and Isabel Valera. Fair decisions despite imperfect predictions. In Silvia Chiappa and Roberto Calandra (cds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 277 287-287. PMIR, 26 28 Aug 2020.
16

---

Published in Transactions on Machine Learning Research (11/2022)
Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed Chi. Fairness without demographics through adversarially reweighted learning. Advances in newral information processing systems, 33:728 740, 2020.
Y. Lecun, L.. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11))2278-233) 2324, 1998. doi: 10.1109/5.726791.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-soquenee pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019.
Yi Li and Nuno Vasconcelos. REPAIR: Removing representation bias by dataset resampling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9572-9581, 2019.
Tianyi Lin, Chi Jin, and Michael I. Jordan. On gradient descent ascent for monconvex-concnve minimax problems. Xiv: 1906.00331v0 2020.
Andrew Lowy and Mcisam Razaviyayn. Output perturbation for differentially private convex optimization with improved population loss bounds, runtimes and applications to private adversarial training. arXiv preprint ar Xiv:21002. 704, 2021.
Luo Luo, Haishan Ye, and Tony Zhang. Stochastic recursive gradient descent ascent for stochastic nonconver- strongly-concave minimax problems. In Advonces in Newral Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NearIPS 2020, December 6-12, 2020, virtual, 2020.
Jérémic Mary, Clément Calauzenes, and Noureddine El Karoui. Fairness-aware learning for continuous attributes and treatments. In International Conference on Machine Learning, pp. 4382-4391. PMLR, 2019.
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6):1-35, 2021.
Husscin Mozannar, Mesrob Ohannessian, and Nathan Srebro. Fair learning with private demographic data. In International Conference on Machine Learning, pp. 7066-7075. PMLR, 2020.
Katta G Murty and Santosh N Kabadi. Some np-complete problems in quadratic and nonlinear programming. 1985.
Milad Nasr, Reza Shokri, and Amir Hounansadr. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE symposium on security and privacy (SP), pp. 739-753. TEEE, 2019.
Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning, pp. 625 632, 2005.
Dmitrii M Ostrovskii, Andrew Lowy, and Meisam Razaviyayn. Efficient search of first-order nash equilibria in nonconvex-concave smooth min-max problems. arXiv preprint or arXiv:2002.000 20te:20000799,, 2020.
Adrián Pérez-Suay, Valero Laparra, Gonzalo Mateo-Garcia. Jordi Muñoz-Marí, Luis Gómez- Chova, and Gustan Camps-Valls Fair kernel learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 339-355. Springer, 2017.
John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
Gcoff Plciss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. On fairness and calibration. In Advances in Newral Information Processing Systems, pp. 5680 5689, 2017.
Flavien Prost, Hai Qian, Qiuwen Chen, Ed H Chi, Jilin Chen, and Alex Beutel. Toward better trado-off between performance and fairness with kernel- based distribution matching. ar preprint xrX 2019.
17

---

Published in Transactions on Machine Learning Research (11/2022)
Novi Quadrianto and Viktoriia Sharmanska. Recycling privileged learning and distribution matching for fairness. Advances in Nearal Information Processing Systems, 30, 2017.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Nya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Edward Raff, Jared Sylvester, and Steven Mills. Fair forests: Regularized tree induction minimize model bias. In Proceedings of the 2018 AAAI/ACM Conference AI, Ethics, and Society, pp. 243-250, 2018.
Alfréd Rényi. On measures of dependence. Acta Mathematica Academiae Scientiarum Hungarica, 10(3-4): 441 441-451, 1959.
Alfréd Rényi. On measures of entropy and information. Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume Contributions to the Theory of Statistics. The Regents of the University of California, 1961.
Ashkan Rezaci, Rizal Fathony, Omid Memarrast, and Brian D Ziebart. Fairness for robust log loss classification. AAAI, pp. 5511 1111-518, 2020.
Goce Ristanoski, Wei Liu, and James Bailey. Discrimination aware classification for imbalanced datasets. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pp- 1529 1532, 2013.
Yuji Roh, Kangwook Lee, Steven Whang, and Changho Suh. Fr-train: A mutual information-basse approach to fair and robust training. In International Conference on Machine Learning, pp. 8147-8157, PMLR, 2020.
Igal Sason and Sergio Verdú, f-divergence inequalities. IEEE Transactions on Information Theory, 62(11): 5973-6006, 2016.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on securily and pricucy (SP), pp. 3 18. IEEE, 2017.
Daniel Steinberg, Alistair Reid, Simon O'Callaghan, Finnian Lattimore, Lachlan McCalman, and Tiberio Caetano. Fast fair regression via efficient approximations of mutual information. arXiv preprint wrXiv:2002.0620, 2020.
Latanya Sweeney. Discrimination in online ad delivery. arXiv preprint arXiv:1301.6822, 2013.
Bahar Taskesen, Viet Anh Nguyen, Daniel Kuhn, and Jose Blanchet. A distributionally robust approach to fair classification. arXic preprint or 0rx: 09530, 2020.
Cuong Tran, My Diuh, and Ferdinando Fioretto. Differentially private empirical risk minimization under the fairness lens. Advances in Neural Information Processing Systems, 34:27555-27555 2021a.
Cuong Tran, Ferdinando Fioretto, and Pascal Van Hentenryck. Differentially private and fair deep learning: A lagrangian dual approach. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 35, pp. 9932-9939, 2021b.
Robert Williamson and Aditya Menon. Fairness risk measures. In International Conference on Machine Learning, pp. 6786 6797. PMLR, 2019.
Hans S Witsenhausen. On sequences of pairs of dependent. random variables. STAM Journal Applied Mathematics, 28(1):100 113, 1975.
Blake Woodworth, Suriya Gunasekar, Mesrob Ohannessian, and Nathan Srebro. Learning non-discriminatory 10X.::102000000 predictors. arXiu preprint arXiv: 702. 06080, 2017.
Depeng Xu, Shuhan Yuan, and Xintao Wu. Achieving differential privacy and fairness in logistic regression. In Companion proceedings of The 2019 world wide web conference, pp. 194-599, 2019.
18

---

Published in Transactions on Machine Learning Research (11/2022)
Muhammad Bilal Zafar, Isabel Valera, Mannel Gomez Rogriguez, and Krishna P Gummadi. Fairness constraints: Mechanisms for fair classification. In Artificial Intelligence and Stalistics, pp. 962-970. PMLR, 2017.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. Tn International Conference on Machine Learning, pp. 325-333, 2013.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAT/ACM Conference on AI, Ethics, and Society, pp. 335-340, 2018.
Siqi Zhang, Junchi Yang, Cristóbal Guzmán, Negar Kiyavash, and Niao He. The complexity of nonconvex- strongly-concave minimax oplimization. arXiv preprinl arXiv:2103. Siv:22103.5588. 2021.
Tianxiang Zhao, Enyan Dai, Kai Shu, and Suhang Wang. Towards fair classifiers without sensitive attributes: Exploring biases in related features. In Proceedings of the Fifleenth A ACM International Conference Web Search and Data Mining, pp. 1433 1442, 2022.
19

---

Published in Transactions on Machine Learning Research (11/2022)
Appendix
We provide simple table of contents below for casier navigation of the appendix.
CONTENTS
Section Notions of fairness
Section B
ERMI: General definition, properties, and special cases unraveled
Section Relations between ERMI and other fairness violation notions
Section
D
Precise Statement and Proofs of Theorem | and Theorem 2
Section Experiment details & additional results
Section
E.1
Model description
Section
E,2
More comparison (Mary et al. 2019
Section
E.3
Performance in the presence of outliers & class-imbalance
Section
E,
Effect of hyperparameter À on the sccuracy-firrness tradeoffs
Scction
E.5
Complete version of Figure (with pre-processing and post-processing basclines)
Section
F.6
Description of datasets
20

---

Published in Transactions on Machine Learning Research (11/2022)
A Notions of Fairness
Let (Y,Y, .A, S) denote the true target, prodicted target, the advantaged outcome class, and the sensitive attribute, respectively. We review three major notions of fairness.
Definition 2 (demographic parity (Dwork el al. 2012 We say that learning machine salisfies demographic parity if Y is independent of S.
Definition 3 (oqualizod odds (Hardt al. 2016)) ) We say that learning machine satisfies equalized odds, if Y is conditionally independent of S given Y.
Definition 4 (equal opportunity (Hardt et al. 2016)). We say that a learning machine satisfies equal
opportunity with respect to A, if Y is conditionally independent of S given Y y for all y E A.
Notice that the equal opportunity as defined here generalizes the definition in Hardt et al. 2016 It recovers equalized odds if A = Y, and it recovers equal opportunity of (Hardt et al. 2016 for A = (1) in binary classification.
21

---

Published in Transactions on Machine Learning Research (11/2022)
B ERMI: General Definition, Properties, and Special Cases Unraveled
We begin by stating notion of fairness that generalizes demographic parity, equalized odds, and equal opportunity fairness definitions (the three notions considered in this paper). This will be convenient for defining ERMI in its general form and presenting the results in Appendix [ Consider a learner who trains a model to make a prediction, Y, e.g., whether or not to extend a loan, supported on a set V. Here we allow Y to be either discrete or continuous. The prediction is made using a set of features, X, e.g., financial history features. We assume that there is a set of discrete sensitive attributes, S, c.g., race and sex, supported on S, associated with each sample. Further, let. C y denote advantaged outcome class, e-g-; the outcome where loan is extended.
Definition 5 ((Z, 2) fairnesss)) Given random variable Z, let Z be subset of values that Z can take. We say that a learning machine satisfies (Z, 2)-fainness if for every E 2, Y is conditionally independent of S given Z 2, Le. Vg E V E S, 2, Pp P (0 8)2) - P (0)))))
(Z, 2 - includes the popular demographic parity, equalized odds, and equal opportunity notions of fairness as special cases:
1. (Z, fairnesss recovers demographic parity (Dwork et al.) 2012) if Z 0 and Z {0}. In this case, conditioning on Z has no effect, and hence (0, (0}) fairness is equivalent to the independence between Ý and (see Definition Appendix A).
2. (Z, fairiiessss recovers equalized odds (Hardt et al. 2016 if Z Y and = y. In this case, Z E Z is trivially satisfied. Hence, conditioning Z is equivalent to conditioning on Y, which recovers the equalized odds notion of fairness, i.e., conditional independence of Y and S given Y (see Definition Appendix A
3. (Z, Z) fairness recovers equal opportunity (Hardt et al. 2016 if Z = Y and Z = A. This is also similar to the previous case with y replaced with A (see Definition Appendix A
Note that verifying (Z, 2a-fiinssss requires having access to the joint distribution of random variables (Z, P, S). This joint distribution unavailable to the learner in the context of machine learning, and hence the learner would resort to empirical estimation of the amount of violation of independence, measured through some divergence. See (Willimssnn & Menon 2019) for related discussion.
In this general context, here is the general definition of ERMT:
Definition 6 (ERMI- exponential Rényi mutual information). We define the erponential Réngý mutual information belween & and S given Z E Z
P2,sz (F,) (Piz ( ()psz((12))
D 2) E
ZEZ 1.
(ERMI)
Notice that ERMI is in fact the >dvivergene between the conditional joint distribution, PQ.s> and the Kronecker product of conditional marginals, Pp @ ps, where the conditioning is on Z E Z. Further, x² divergence is an f-divergence with f(t) = (t - 1) See (Csizáár & Shields 2004 Section 4) for a discussion. As an immediate result of this observation and well-known properties of ff f-divergences, we can state the following property of ERMI:
Remark D E 2) 2 with equality if and only if for all z 2, y and are conditionally
independent given Z
To further clarify the definition of ERMI, especially as it relates to demographic parity, equalized odds, and equal opportunity, will unravel the definition explicitly in few special cases.
22

---

Published in Transactions on Machine Learning Research (11/2022)
First, let Z 0 and Z {0}. In this case, Z Z trivially holds, and conditioning Z has no effect,
resulting in:
Dn(P:8)= Sn(P: E 2)
2=0,2=(0)
P (P,((,
=Bps
Pp (Y (s))
Py.s 10, 10.5(0.0 8) Pp (s)
Σ #ES BEY
PV, (0, s)dg.
(0.
(2)
Pp (8)
(P; the notion of ERMI that should be used when the desired notion of fairness demographic parity. In particular, DR(V:S) 0 implies that X² divergence between pp.g- and the Kronecker product of marginals, pp @ Ps is zero. This in turn implies that Y and S are independent, which is the definition of demographic parity. We note that when Ý and S are discrete, this special case (Z = 0 and Z = {0}) of ERMI is referred to as xiiinformation by Calmon et al. (2017a).
Next, we consider Z Y and 2 =. In this case, Z E is trivially satisfied, and hence,
DR(Y:SlY) = DR(P: 2)
--Y,Z=Y
Ps Pll (1,)
E YPP.9
PP ( (SIY)
(0,
(0, sly)
Ppix (11))s))
Σ BEY JEY
Pv. (y, 9, s)dÿdy
pyr (Wl))ss) (s|v)
PP PPSSY (0. s/y)
Σ
SES yEY JEE Pylx (1 (1b)ps (s|y) (v)djdy 1.
(3)
; should be used when the desired notion of fairness equalized odds. In particular, ; 0 directly implics the conditional independence of Y and S given Y.
Finally, we consider 7 Y and 2 = A. this case, we have
DA(Y = DR(Y e 2)
Z=Y,Z=A
Pr S(Y)
YEA
=Exp.
ppix (VIY)
ps PV (1,)
psly (Glu)ps(r
Σ
(u)dddy
pp ppiy (0ly))ss) (s|y)
DEA DEA BEY
P7 PQ.slY sly)
Σ
ES UEA Jey Pply (flv) (s) PI (0, (l)) 1,
(4)
where
pv(v) TyEAPy(y dyy)
(5)
This notion what should be used when the desired notion of fairness is equal opportunity. This can be further simplified when the advantaged class singleton (which the case in binary classification). If
23

---

Published in Transactions on Machine Learning Research (11/2022)
Z Y and Z {gy, then
DR(P;S = y = 1 (Y;))))
Py. ( sly) Ppp
Σ
(0.s))d
P (s|y)
#ES JEY
(ly)
Σ
dÿ- 1.
(6)
JEY Py y ( (s(g)
Finally, we note that we the notation PR; S|Y) and DR(Y; S y) to be consistent with the definition of conditional mutual information in (Cover & Thomas
1991
24

---

Published in Transactions on Machine Learning Research (11/2022)
C Relations Between ERMI and Other Fairness Violation Notions
Recall that most existing in-processing methods use some notion of fairness violation as regularizer to enforce fairness in the trained model. These notions of fairness violation typically take the form of some information divergence between the sensitive attributes and the predicted targets (e.g. Mary et al. (2019): ); Baharlouei et al. ) Cho et al. (2020a)). In this section, we show that ERMI provides an upper bound on all of the existing measures of fairness violations for demographic parity, equal opportunity, and equalizod odds. As mentioned in the main body, this insight might help explain the lavorable empirical performance of our algorithm compared to baselines even when full batch is used. In particular, the results in this section imply that FERMI algorithm leads to small fairness violation with respect to ERMI and all of these other measures.
We should mention that many of these properties of f divergences are well-known or easily derived from existing results, so we do not intend to claim great originality with any of these results. That said, we include proofs of all results for which we are not aware of any references with proofs. The results in this section also hold for continuous (or discrete) Y. We will now state and discuss these results before proving them.
Definition 7 (Rényi mutual information RRényi 1961 ). Let the Rényi matual information of order >> between rondom variables Y and given Z Z be defined as:
PD P9, (8 SIZ)
a-1
2E2))
1 a- log EgF.s 1a (f
(RMI)
PPiz (9 (8)))181111)))
which generulizes Shannon mustual information
PV,s PV, SYZ))
E : =2, log
(MI)
P9iz (Y ( (2)ps12((22))
and recovers it lim 1Y; 2) 11;8 2).
Note that la S 2) 2 with equality if and only if (Z, -airness is satisfied.
The following is minor change from results in Sason & Verdú 2016
Lemma 1 (ERMI provides an upper bound for Shannon mutual information). We have
0< e 2) s( e 2) (Riie) ; SIZE 2).
(7)
Lemma. also shows that ERMI is exponentially related to the Rényi mutual information of order 2. We include proof below for completeness.
Definition 8 (Rényi correlation (Hirschfeld, 1935 Gebelein 1941 Rényi 1959 Let F and g be the
set of measurable functions such that for random variables and S, Eg (f(:2))) Es 2)) = 0, Ey (1(;; = Es (9(S;2)²) =, for all 2 Z. Réngi correlation is:
S|Z E 2) == sup E 2 (18:2)9((:22) 2 P
(RC)
LooFxo
Rényi correlation generalizes Pearson correlation,
B (SS1))
, 0 2) 2): Ez
7 E
(PC)
B (Z=
capture nonlincar dependencies between the random variables by finding functions of random variables that maximize the Pearson correlation coefficient between the random variables. In fact, it true that pn(Y,St S Z) 2 0 with equality if and only if (Z, fa-rriss is satisfied. Rényi correlation has gained
25

---

Published in Transactions on Machine Learning Research (11/2022)
popularity as measure of fairness violation (Mary et al. 2019 Baharlouci et al 2020 Grari et al. 2020
Rényi correlation is also upper bounded by ERMI. The following result has already been shown by Mary et al. 2019) and we present it for completeness.
Lemma 2 (ERMI provides upper bound for Rényi correlation). We have
0 5 pp(P, 2)1 S pn(Y 2) s DR(Y: 2),
(8)
and (f = 2, DR(Y; E 2) = PR SIZ E Z).
Definition 9 (Ly fairness violation). We define the Lg fairness violation for g 2 1 by:
2) =E (1
#
Σ DE3O sESo
P9 (0.s1) p4iz (1112222)))
dy
Z ZEZ
(La)
Note that ( E) recovers demographic parity violation (Kearns et al.) 2018
0 if and only if (Z, -afirness satisfied. In particular, Loo fairness violation Definition 2.1) if we let = {0} and = 0. It also recovers equal opportunity violation (Hardt et al. 2016 if Z A and Z =YY.
Lemma 3 (FRMT provides an upper bound for T. fairness violation). Let Y be discrete or continuous random variable, and S be a discrete random variable supported on finite set. Then for any q 2 1,
0 L, SI 2) D
(9)
The above lemma says that if a method controls ERMT value for imposing fairness, then Tox violation is controlled. In particular, the variant of ERMI that specialized to demographic parity also controls Lo demographic parity violation (Kearns al. 2018). The variant of ERMI that is specialized to equal opportunity also controls the Les equal opportumity violation (Hardt et al. 2016). While algorithm uses ERMI as a regularizer, in our experiments, we measure fairness violation through the more commonly used Loo violation. Despite this, we show that our approach leads to better tradeoff curves between fairness violation and performance.
Remark. The bounds Lemmas 1-3 are not tight in general, but this not of practical concern. They show that bounding ERMI is sufficient because any model that achieves small ERMI guaranteed to satisfy any other fairness violation. This makes ERMI an effective regularizer for promoting fairness. In fact, in Sec. saw that our algorithm, FERMI, achieves the best tradeoffs between fairness violation and performance across ofatc-of-tho-art basclines.
Proof of Lemma
We proceed to prove all the (in)equalitics one by one:
0 < [1((Y8|1 E Z). This is well known and the proof can be found in any information theory textbook Cover & Thomas 1991).
[[; E 2) < T2(P; SIZ E 2). This a known property of Rényi mutual information, but we provide a proof for completeness Lemma below.
I(P E 3) df. - 1. This follows from the fact that x S e - 1.
(f(iss)zzzz DR(Y: s Z). This follows from simple algebraic manipulation.
Lemma 4. Let V,S, Z be discrete or continuous random variables. Then:
(a) For any a, B E [1, o0), Ts(9 2) 10 (Y; 2) 11 a.
26

---

Published in Transactions on Machine Learning Research (11/2022)
(b) lima-1++ la Ia(P E 2) 1))
= Ez DKu(py,sizllpp z @ ps|z) Z E z where I1(:)) de- notes the Shannon mutual information and DEL Kullback-Lete Leibler divergence (relative entropy).
(c) For all a E (0), TY S|Z 2) 2 with equality if and only if for all a E 2, Y and S are conditionally independent given z.
Proof. (a) First assume 0 < a < B < oo and that 3 + 1. Define a a 1, and b = B - 1. Then the function s(1) = 1b/a is convex for all 2 0, so by Jensen's 's inequality we have:
1 log
B {
((P,SIZ)
)
} 1)
1 log
(P. p( (((12))(S(2)
b/a
Z 2
E
2 E
b
((812)p(S12)
1 log
p(F,SIZ) p(Y(Z)p(SIZ)
a
Z ZEZ)
E
(10)
Now suppose a = 1. Then by the monotonicity for a # 1 proved above, we have 11 (Y; = lim La (a;P:)
=
(1,)e(0.1) La (Y: < infa>1 (Y;))) Also, Lco (Y; S) lima (Y: supa>0 (Y
(b) This is a standard property of the cumulant gencrating function (scc [embbo & Zeitouni, 2009
(e) It is straightforward to observe that independence implies that Rényi mutual information vanishes. On the other hand, if Rényi mutual information vanishes, then part (a) implies that Shannon mutual information also vanishes, which implics the desired conditional independence.
Proof of Lemma The proof is completed using the following pieces.
0s 10 3) PR( S|Z E 2). This obvious from the definition of p ,S E 2).
pR(P,SIZ E 2) < D ; S|ze E 3). This follows from Lemma below.
Notice that |S| = 2, Lemma implies that DR(Y:SIZ e Z) = SIY, 2).
Next, we rocall the following lemma, which stated Mary et al. 2019 and derives from Witsenhauseni 's characterization of Renyi correlation:
Lemma 5. Suppose that [k]. Let the k matrix P be defined as P (Pa)a.EEAAM))) where
/
(V.s(u.)pp ((V.5)
p=y:= = Vps(ips(i)
dy.
(11)
NEY
pp (v)
Lel 1 = 01 2 02 2 ... 2 oh 2 be the eigenvalues of Then,
PR(P,S)= =02
(12)
DR(P;S) -Tr-) - Σ oi
(13)
1=2
27

---

Published in Transactions on Machine Learning Research (11/2022)
Proof. Eq.
12 is proved in Witsenhausen 1975
Section 3). To prove Eq.
13
notice that
Tr(P)
Σ iE(R)
Pa
PF.s(u,0)2
E iE|ki
dy
ps(
JEE
PT(U) }
Ps (P..(Y.S))
E
(P)(ss()))
PG
11 (R({:S),
which completes the proof.
Proof of Lemma It suffices to prove the incquality for Iı, Lyg is bounded above by L₁ for all g 2 1. The proof for the case where Z = 0 and Z = {0} follows from the following set of inequalities:
LiV,SiZe SIZE 2)= 2) E/
- - pp()))s(s) dy
(14)
sES DEy DEY
Pv. Pp (s)ps(s)
Σ
SES BEY V (Pp())p(ss)
dy
(15)
pp (v)p(())
(pp (PP,s(U, 8) Pp (v)ps
Σ SES DEY
Pp PT(U)s((sddy)
E
(16)
pp pp(u)ss())
WES DEY
(Br.s(uis) (F(()s((s)) Pp(v)ss(s)
dy
(17)
SES wEY
(R(;
(18)
where Eq. 16 follows from Cauchy-Schwan inequality, and Eq. 18 follows from Lemma
The
extension to general Z and Z is immediate by observing that
Z) Bzz ((8
Z E 2
pR(P,S|Z 3) Ez (e((,s)2)) z E 2]. and D 2) Ez 1Ra(,,S12)) S|1) 2 E
Lemma 6. We have
(pp (Pp.s(u.s) pp (u)ps(s)) dy. (p(U)ss(s)
DR(P:S)= .ES NEY
(19)
28

---

Published in Transactions on Machine Learning Research (11/2022)
Proof. The proof follows from the following set of identities:
(Pg,s(U,s) 8) PPp (Pp())s(())
(P
Σ/ NES vEy
dy Σ
py (y)pp((s) dy
pp pV(U)p((s)
#ES
NEY
2 Σ
ps.s P9.s(u.81) (y, s)dy
#ES
NEY
Σ
pp (c())ps(5)dg
(20)
#ES' VEY
Pp P
E
(21)
PG (P)
DR({;
(22)
Next, we present some alternative fairness definitions and show that they are also upper bounded by ERMI
Definition 10 (conditional demographic parity Los violation). Given a predictor Ý supported on y and a discrete sensitive attribute S supported on a finite set S, we define the conditional demographic parity violation by:
áp(\1s) == sup max Jey SES pris(0ls) pp Pp(
(23)
First, we show that dp({))) is a reasonable notion of fairness violation. Lemma 7. dp(Pis) 0 iff (if and only if) Y and S are independent.
Proof- By definition, Ap(Vis) 0 iff for all 7 V,8 E (0,s)) = (G) iff & and S are independent. (since we always assunee p(s)= > for all S).
Lemma 8 (ERMI provides an upper bound for conditional demographic parily Lo violation). Let & be a discrete or continuous random variable supported on V, and S be a discrete random variable supported on a fonite set S. Denote p := min.es (s()) > 0. Then,
0 S dp(\)s) < min V DR(P:S).
(24)
PS
Proof. The proof follows from the following set of (in)oqualitics:
(To(P1s))²
sup max wEy #ES
(piis py (7)
(25)
sup max Dey #ES pp () 1p0)ppploo -Pp Pp(Dps()))
(26)
(pminys
Pp. (0.)) pp (0)p((s))
(27)
(pmin
BEY
Dn (Y:55))
(28)
(pgin)z
where Eq. 28 follows from Lemma
3
29

---

Published in Transactions on Machine Learning Research (11/2022)
Definition 11 (conditional equal opportunity Lo violation (Hardt al. 2016)). ). Let Y,Y take values in y and let A y be compact subset denoling the advantaged oulcomes (For example, the decision "to interview" on individual or classify an individual as a "low risk" for financial purposes). We define the conditional equal opportunity Los violation of Y with respect to the sensitive attribute S and the advantaged outcome A by
sup max (vyy RE8 Pf ( pyy (11)) (
( = Ey
(29)
Lemma 9 (ERMI provides an upper bound for conditional equal opportunity Loe violation). Let Y, Y, be discrete or continuous random variables supported on V, and let S be a discrete random variable supported on a finite set S. Let A be a compact subset of y.
Denote PS P (sly). Then,
0 S E
Dn(Y: e A).
(30)
piin
Proof. Notice that the same proof for Lemma g would give that for all E A:
0 S sup max
JEY #ES PV.sy (0)s,)) 2) pvir (1v) = 6 = y)
DR(P; S/Y = y)
Pglv
Dn(P:SVV = v).
min Psc
Hence,
corris, A) = Ey { co(V)) Y YEA)
min By
1
{ (DDR(F:S)r)
Y YEA
1
Ey (DD((: ( YEA
min
PSA
Dn(P:S A),
PS
where the last inequality follows from Jensen's inequality. This completes the proof.
30

---

Published in Transactions on Machine Learning Research (11/2022)
D Precise Statement and Proofs of Theorem I and Theorem 2
To begin, we provide the proof of Proposition
Proposition 3 (Re-statement of Proposition
1. For random variables Y and S with joint distributionn pv ss
where Y E [m]. S E [k], we have
DR(P;S)=
max WER
{- TrWP
-1},
if P. diag(07(1)) (1)....... (0())...(((())) Ps = (.. and (Pgi))ss = si,3( with pp(1), > for i E [m].j [k]]
Proof. Let W E arg max - TT(WP, + + + D. 112) sion on the RHS equal to zero leads to:
Setting the derivative of the expres-
=0 = W* P-1/2pp P -2 P, + 22
1/211/p1 v.s
U.
Plugging this expression for W have
Tr(W NIWPPTT WT + TTT P
max W E
= -TT((P,-1/23 1/2 p-1 P pr. Pr 1-1 + 2TT(( P..PPiPP Po.PP.19 1/2)
0.
U.a
-Tr(P_1/2p/ p-p Po.s D-1/2 0.
-Tr(P--PH P. P..s).
Writing out the matrix multiplication explicitly in the last expression, we have
P-- pr p- Pg. = UV. 0.0
where = = ps(()) ((. and Vr, = Pr (0)0))11p Dbp.syssss.. ((,i),, for i [k), E [m]. Hence
Tr -Tr(WP,WT) P/W 2Tr 2T. P,..
max WERkXm
Tr(UVT)
Pr. si,01 (0)
Σ iE[l) jE(m]
= (R;) +1,
which completes the proof.
Corollary 2 (Re-statement of Corollary Let (xi, 85, vi) be a random draw from D. Then, Eq. FERMI
obj.
is equivalent to
(F(O.W)== 200) x0
min
Haa We
(31)
where () Tr(WP,WT) Tr P. P. 1/4 ΣN (0, W) and
(0 == (xx,, (x) 0)T WT) 2T 0)s x s; 1-122
- TT (0, x.). Fm (0,x)) WT) 2 1x, sp
1-1122
Proof. The proof simply follows the fact that
max E (0.
(") P-1/22 -1) =DR(P;S).
max WeRkxm
WER
where the last equality is due to Proposition
31

---

Published in Transactions on Machine Learning Research (11/2022)
Next, will state and prove the precise form of Theorem 1 We first recall some basic definitions: Definition 12. function f is L-Lipschitz if for all u, u' E domain( we have ((u)-f(w)) S u u'll Definition 13. A differentiable function f B-smooth if for all u, u' domain f) we have [IV
Vf(u))l - B| u'll.
Definition 14. A differentiable function is p-strongly concove if for all x.y domain(f), have ((x + fxx)= (y x) lly -x12 2 f(y)
Definition 15. A point 0" = A(D) output by a randomized algorithm A is an e-slalionary point of differentiable function & if V S e. We say 0r is an e-stationary point of the nonconvez-stren strongly concave min-max problem ming maxw F(O, W) if it an c-stationary point of the differentiable function (0)) = maxw F(O,W) ().
Recall that Eq. FERMI obj.) is equivalent to
1
Σ [(x,, Vii 0) X m.
min WER
(32)
N
where (0, W) TWW P/1/3 - 1= Σ (0, W) and
(0, W) == T 0)5 0)² x; ) + TTr (x;; 0)s?"
x, 8; P. P-1/2)
- --aggWi xx) (0: x: W 2T 0)s Xx, s;] P-11/2) 1/2)
where f(x,; 0) and s; are the one-hot encodings 7(x; 1) and 8s, respectively. Assumption 1.
x() is G-Lipscthiz, and Be-smooth for all x, y-
x(, is L- L-Lisschitz and b-smooth for all
Pgin = inf (or (0r, SE(T]) min F,
F, x4) 2 >0 0.
pg = N E. (x-=)) >0.
Remark 3. As mentioned remark 1 the third bullet in Assumption
1 convenient and allows for faster convergence rale, but not strictly necessary for convergence of Algorithm
Theorem 3 (Precise statement of Theorem I Let (i be any given data set of features, labels, and sensitive altributes and grant Assumplion Let W = Br (0, D) Rexm (Frobenius norm boll of radius VOg" in Algorithm Denote Ag :- (0o)) - info (0), where D(0) == maxw F(0, W).
D), where = A
In Algorithm choose the step-sizes as Tl θ (max (1, 11,oo -2)).. Then under Assumption e-slalionary point of is bounded by
0(1kx²3) and 1w 0(1/3) and mini-batch size as |Ba] the iteration complerity of Algorithm to return
3A + x22 Ke
22
which gives the total stochastic gradient complerity of
122 122943
B3 D2 2
0
max (1, (1,10?²
where
D+ 1
3 2
B1 3
+ + + 8L
D
imin
# 2Xp amin
K = 3/1,,
à = 162 +1) + 4G 32X 2 (1
mk
pmin
32

---

Published in Transactions on Machine Learning Research (11/2022)
Remark 4. The larger minibatch size is necessory to obtain the faster O(c=*) converyence rate vio two- timesoole SGDA. However, as noted in (Lin et al. 2020, p.8), their proof readily erlends to any batch size [B,] 2 1, showing that two-timescale SGDA still converges. But with [Ba] = 1, the iteration complerity becomes slower: (b. This is the informal Theorem that was stated in the main body.
In light Corollary 1 Theorem follows from [in et al. 2020 Theorem 4.5) combined with the following technical lemmas. We assume Assumption [ holds for the remainder of the proof of Theorem 3
Lemma 10. If x,, Ut, are drown uniformly at random from data set D, then the gradients of ((x,, ye. 0) + ( W) are unbiased estimators of the gradients of F(e, W) for all 0, W.
V Vi, 0) AV o0, w VoF(e, W), and
B[AV (0, w|] Fw W..
Purthermore, if Wile & D, then the variance of the stochastic gradients is bounded as follows:
VL Vr. 0) + AV - w) 3 O,W
sup
(33)
where o² 16X 1) + 4G 32
mi
Proof.
Unbiascdness is obvious. For the variance bound, will show that
sup V W) Vw ( S o. 0.W
(34)
and
sup Vol(x,, 81, + w s oz. 0.W
(35)
where 0² oz + o²
First,
20 (r) -12 -/ ",. 0)(,;
(36)
Thus, for any 0, W, A, we have
412
E A (0. W)
Vw F(O,
WE 0)8(x, </ |x;
N
"x. 1)7ix;, sil)
N
412
|| 1) P/12
2
N
P
(Bis:)(x:; 0)² xx, si] -Pg.. >/,))) l1.
4X2
N Σ 2 2D2 +
1--122 (E(s,5(x, (),
Xx, ss] P/..)) 113
4X2 N
N Σ 2 [2D2 +2]
$ ( +1,,
where we used Young's incquality, the Frobenius norm incquality ||A]] < |||||~|| B||r, the
facts that | (x,, (x) 0) 1x;11 (00,,,) < and ||s, 0) |x; s:]1.
33

---

Published in Transactions on Machine Learning Research (11/2022)
Σ s Σi= Fi(x;, 0)2 1 for all i E [M] (since for every i E [N], only one of the Sij is non-zero and equal to 1, and Ei= Fi x;) = 1). Next,
AV AVoys (0, W) A V vec (ec(EEy((;) ()) (); xx1)" vec(WTT) + 2Vo vec(Els.5(x,, 0)" |x, sil) ve
1/3)
(37)
Hence, for any 0, W, we have
E| Vol vz 0) + AVo 1
< 2 2 sup Vol(x, Vc. 0112
Xiilli
+ X
sup
Ve (xx 0)² veo(W>"
Xi
2V vec vec(E(ss 0)²² xx, s:) P |?
< 4 G + +21
sup xi
Vo vec(B (x5(x; 0)² (x,1)² vec(WT w)ll
Vo vec( 0) |x,, s.]) veo(WT P. 1/) 121113)1))) }
+ sup X1.6;
by Young's and Jensen's incqualitics and the assumption that E(x;, 1). G-Lipschitz. Now,
Ve (x, xx))) vec(W W) = (F 0) k
#=1
which implies
Ve (ec(EE)( ())(xxx 0) Ix:)) W) Σ
W3, sup
|VFi(x S D²L?
(38)
3.1
by L-Lipschitzness of x(, Also,
Si. W (0,
Ve vec(B)s,ú(x, 1x" vec 1-1122 =
VDs(r)
r=1
which implics
Ve vxc(E(s,((xx,0)) x,, s:l) vee P-11/3 12
k
Wi
mk pmin
< mkk
sup
[F,
L²D².
(ps(r))
x.00
Thus,
mk
o < 402 32X
1
P3
Combining the 0- and W-variance bounds yields the lemma.
Lemma 11. Let
F(O.W) N
Σ
E(x;, M6; 0) +0,(00
NE[N]
where
() Tr(WE|5(x,, 0)5((, 0) (x + +2 s,p--11
Then:
34

---

Published in Transactions on Machine Learning Research (11/2022)
F B-smooth, where 3 = 2 Br 2ADmb (D+
(D+ )
+ 8L
//g
2. F(0,-) is n -strongly concuve for all 0
3. If W D) with D
then Eg. 1 mino maxwew F(O, W).
Buin P
Proof. We shall freely use the expressions for the derivatives of vis obtained in the proof of Lemma 10
1. First,
I [(,F( V.. V., suu (), - 0)* |x:]|rr
2 Wr,
since F,(0,x,) x,) S for all j E [m]. Next,
|v.F(0, W) - v F,',
S 8 sup 8.
D2 [ 0){xx, 0) [x.] E[ 0')² [xi]
p 1-1/2 1/2
(x,, 0)¹ x, s: Elss (x, (') xx,, 8s s1))
< 8 sup
D2 - x,)
172 k
1, - -=)((0")))))7)())) -1 -'s?,
j=1
I
< 8 sup
D² 0'112 + imin
10 0'11
x, :3
which implies
IV (V,
- V v.. (1', () S8L D+
1
110 |10-00 0'
v p
Lastly,
|Vo -V) Will
sup x. 81
V yi, 0) Vi(x;; yi, 0')l
+
vec(EEyy(x (x) 0)' xx,))= Vo (ec( (')( 0')² ( vec(W""W)]
Ve vea (E) 0)² Xx,, st)) V vec(Els: 0') x., sc])] vec( W P 1/2/|
+ 2\
S 33:e 0'll XD sup
[F, (=F(0'x))))
m
22 sup x.rE(K]
Σ
(f,(0,)) VF, ())(()))s)))111010
Db
81 + 21
1Pb+
00 0',0' 0'1
35

---

Published in Transactions on Machine Learning Research (11/2022)
by Assumption
1 Combining the above inequalities yields part 1.
2. We have V www F(O.W) W) -2AP3. which is a diagonal matrix with F(0, W))ss
- N
N Fy=1F 0) f -Xp7"" by Assumption i Thus, F(..0) is 2 2npmin- -strongly concave for all 0.
3. Our choice of D ensures that (*"0") int(W), since
||>((0)]le 1| (0)""""))))
(39)
(40)
p p
Therefore, maxwew F(0, w) = maxw F(0, W), which implies part of the lemma.
By Assumption I and Lemma 11
our choice of W implies that (0*) E W and hence that the solution
of Eq. FERMI obj.) solves
F(O,W): = N
Σ 200 + M
min max WEW
This enables us to establish the convergence of Algorithm 1 (which involves projection) to a stationary point for the unconstrained min-max optimization problem Eq. that we consider. The W projection step in Algorithm 1 is necessary to cnsure that the itcrates W remain bounded, and hence that the smoothness and bounded variance conditions of F are satisfied at every iteration.
D.1 Proof of Theorem 2
Now we turn to the proof of Theorem 2 We first re-state and prove Proposition 2
Proposition 4 (Restatement of Proposition 2 Let {a) 81. be drawn i. d. from an unknown joint distribution D. Denote ((m) W) = -Tr(WEly(x, (x) ()Txx + (P!
2Tr WE 0)s Xx: s:]
(m.=))) Denote Po.. 1, where P3 diag(EEF (0,x), EF, x)), (Py.))jrr (0, () TTWPP (I
)- )
-1/2)
--1, -, where ((m)
1 n Ei diag((((;;=)))
Ex.. (6. for j [m].r E [k], and P. diag( Ps()), ... Ps(k)). Assume ps(r) for all r E [k].
Then,
max (0,, W) D (
and
lim E ==0
(m)) [)] ().
Proof. The first claim, that maxw ( W) = replacing the empirical probabilities with D-probabilities everywhere. For the second claim, clearly have
= DR (0) is immediate from Proposition and its proof, by
(p(n)) -1/2) }
- (11)
El (0, E (x., (x;, 0)² x][TT)] + 2E Tr WE)[(x;; xx, si]
=TT(WP,W") + 2F Tr [ 0)s xx, s.] (P()) -1/2) -12)
] -1.
for any n 2 1. Now, p!n) (r) converges almost surely (and in probability) to ps(r) by the strong law of large numbers, and E[P!")(r) (r)] (r)).. Thus, ((n) (r) is consistent estimator of ps(r). Then by the continuous mapping theorem and the assumption that ps(r) 2 C for some C'> 0, we have that (p!") (r)
-1/2
converges
almost surely (and in probability) to 11/2 Moreover, claim that there exists N N such that
36

---

Published in Transactions on Machine Learning Research (11/2022)
for any n 2 No, Var (AT7 -1/2)
S < cc. To see why this claim holds, note that the definition of almost sure convergence of p! (r) pr( implies that, with probability 1, there exists N such that for all n 2 N
min P! z
min ps(r) - C/2 C/2. rE(l]
rE(R]
Thus, Var (1T 1/22
(r)
-1
S a. Therefore (P! -1/2
is a consistent estimator with uniformly bounded variance, hence it is asymptotically unbiased: limn E
I E
( (P! (r)
-1/2
p/( Furthermore,
[18)(x; 0)s" xx, (P!.))
1/2
s for all n 2
N and
J.n
[l 0)s? [x sis (for any j E [m], r E [k]]..
(
1/2
converges almost surely to [x][(x; s)s? X1, 8i] (Ps) -1/2)
as
for
jr
Thus, by Lebesgue's dominated convergence theorem, we have
Bly 0)5 xx, (p!..) )"1") j.r
E. [[ 0)5 [x, 8il lim (p!))())
1//2
lim
E
lim (r) -1/2
EE Fs (0,xx)
11000 (Pr.))).ps(()-11 1/2
(Pa..P. )
(42)
for j j E [m],r E [k]. Combining Eq. 41 proves the second claim.
with Eq.
42
(and using linearity of trace and matrix multiplication)
We are now ready to prove Theorem
Proof of Theorem 2 Denote (()) maxw F(O, W) for the population-level objective F(O, W) = L() (0, W) (using the notation in Proposition 2 Let 0" denote the output of the ooc-ppss//ammple-withot
replacement version of Algorithm run on the modified empirical objective where (p!".) 1/2 is replaced by
the true sensitive attribute matrix Ps 1/2 That is, 0 ~ Unif(Ø 07), where 0 denotes the t-th iterate of the modified FERMI algorithm just described. Then, given i.idd samples, the stochastic gradients are unbiased (with respect the population distribution D) for any minibatch size, by Corollary [
and its proof. Further, the without-replacemmet sampling strategy ensures that the stochastic gradients independent
across iterations. Additionally, the proof of Proposition 2 showed that (P!m)) -1/2
converges almost surely to
P5 P-1/2, Thus, there exists N such that n N 2 T
2e (()--),, then minreej) p(m) (r) (by almost sure
convergence of Pe, see proof of Proposition 2 and
E \ 6 4'
(43)
by Theorem I and its proof. Let @(m) denote the t-th iteration of the one-pass version of Algorithm run on the empirical objective (with (p()) 1/2
Now,
Vo, W) Vo (ec (x) 0)T (x.)" wec(W vec 0)7 x., s:)) vec (T (pl)) 11/2)
which shows that @(m)) continuous (indeed, lincar) function of (P!!m) - 112
for every t. Thus, the continuous
mapping theorem implies that 0.n) converges almost surely to 0: as oo for every E (T)- Hence, if
37

---

Published in Transactions on Machine Learning Research (11/2022)
@(m) Unif 00"h. 0 then @(m) converges almost surely to 0*. Now, for any 0, let us denote W(O)
arg max F(O,W) Recall that by Danskin's theorem (Danskin 1966). we have V Ve (())), Then,
|V((0()) - 2
Vol VoF(Om) (2F(0) (( - VoF(O", voF(0,,((""))
+ (o0(0rr, - -Vor(0,
< 2 B (10mm) - 1-2 B (I(W((())) - [1?
2
2 310 1-2 + 4 10 0-11]
where L denotes the Lipschitz parameter of F, B is the Lipschitz parameter of VF, and is the strong concavity parameter of F(O,-): see Lemma 11 and its proof (in Appendix D) for the explicit 3, L,, and We used Danskin's ss theorem and Young's inequality in the first line, B-Lipschitz contimuity of VF in the second line, and 21 -ipiscchit continuity of the arg maxw F(0, W) function for p-strongly concave and L-Lipschitz
F(O,-) (see e.g. (Lowy Razaviyayn 2021 Lemnaa B.2)). Letting n + oo makes 10(0) 0*
I| + 0 almost
surely, and hence ((())) v (0" ||2 - 0 almost surely. Furthermore, Danskin's 's theorem and Lipschitz continuity of V&F implies that || (
v < C' almost surely for some absolute constant C > 0 and all n sufficiently large. Therefore, we may apply Lebesgue's dominated convergence theorem to get. - El V (@())) - (0(0")12 = lim [(()) v = 0. In particular, there exists such that 2 N completes the proof.
E ())))) (0(0")| S 1- Combining this with Eq. 443) and Young's inequality
38

---

Published in Transactions on Machine Learning Research (11/2022)
E Experiment Details and Additional Results
E.1 Model description
For all the experiments, the model's output is of the form = softmax(Wx + 6). The model outputs are treated as conditional probabilities p = 1|2) = O, which are then used to estimate the ERMI regularizer. We encode the true class label Y and sensitive attribute S using one-hot encoding. We define {(-) as the cross-entropy measure between the one-hot encoded class label Y and the predicted output vector O.
We use logistic regression as the base classification model for all experiments in Fig. 1 The choice of logistic regression is due to the fact that all of the existing approaches demonstrated in Fig. [ use the same classification model. The model parameters are estimated using the algorithm described in Algorithm [I The trade-off curves for FERMI are generated by sweeping across different values for À E [o, 10000]. The learning rates T7. nw is constant during the optimization process and is chosen from the interval [0.0005, 0.01] for all datasets. Moreover, the number of iterations T for experiments in Fig. D is fixed to 2000. Since the training and test data for the Adult dataset are separated and fixed, do not consider confidence intervals for the test accuracy. We generate ten distinct train/ train/testian sets for each one of the German and COMPAS datasets by randomly sampling 80% of data points as the training data and the rest 20% as the test data. For a given method in Fig. [ the corresponding curve is generated by taking the average test accuracy on training/tert datasets. Furthermore, the confidence intervals are estimated based on the test accuracy's standard deviation on these 10 datasets.
To perform the experiments in Sec. 3.3 we use a linear model with softmax activation. The model parameters are estimated using the algorithm described in Sec. 3 The data set is cleaned and processed as described in (Kaars et al., 2018). The trade-off curves for FERMI are generated by sweeping across different values for À in [0, 100] interval, learning rate 17 in [0.0005, 0.01], and number of iterations T in [50, 200). The data set is cleaned and processed as described in (Kaarn et al. 2018
For the experiments Sec. 3.4 we create the synthetic color MNIST as described by Li & Vasconcelos (2019). We set the value 0. In Fig. 6 we compare the performance of stochastic solver (Algorithml [ against the basclines. We use mini-batch of size 512 when using the stochastic solver. The color MNIST data has 60000 training samples, so using the stochastic solver gives speedup of around 100x for each iteration, and an overall speedup of around 40x. We present our results on two neural network architectures; namely, LeNet-5 (Lecun et al. 1998) and a Multi-layer perceptron (MLP). We set the MLP with two hidden layers (with 300 and 100 nodes) and an output layer with ten nodes. A ReLU activation follows each hidden layer, and a softmax activation follows the output layer.
Larger value for generally translates to better fairness, but one must be careful to not use very large value for À as it could lead to poor generalization performance of the model. The optimal values for X, 7, and T largely depend on the data and intended application. We recommend starting with x x 10. In Appendix E.4 we can observe the effect of changing À on the model accuracy and fairness for the COMPAS dataset.
Some general advice for tuning A:
E.2
More comparison to (Mary et al.
2019)
The algorithm proposed by Mary et al. 2019 backpropagates the batch estimate of ERMI, which is biased especially for small minibatches. Our work uses correct and unbiased implementation of a stochastic ERMI estimator. Furthermore, Mary et al.) algorithm does not converge. See Fig. for the evolution of training loss (i.c. value of the objective function) and test accuracy. For this experiment, we follow the same setup used in (Mary et al. 2019 Table 1); the minibatch size for this experiment is 128.
(2019) does not establish ony convergence guarantees, and in fact their
E.3 Performance in the presence of outliers & imbalan
We also performed an additional experiment on Adult (setup of Fig with a random 10% of sensitive attributes in training forced to 0. FERMI offers the most favorable tradeoffs clean test data, however, all
39

---

Published in Transactions on Machine Learning Research (11/2022)
6
Mary et al.
0.9
FERMI
Test Accuracy
0.8
Train Loss
MAm ant Appni y
4
WW
0.7
2
0.6
Mary FERMI
250
500 Epochs
750
1000
250
500 Epochs
750
1000
Figure 7:
Mary et al. 2019 fails to converge stationary point whereas stochastic algorithm easily converges.
methods reach higher plateau (see Fig The interplay between fairness, robustness, and generalization is important future direction. With respect to imbalanced sensitive groups, the experiments in Fig naturally imbalanced dataset, where max.es p(s)/ minses p(s) > 100 for 3-18 sensitive attrib, and F FERMI offers the favorable tradeoffs.
0.11
Hardt et al., 2016
Equal Opportunity
Zafar et al., 2017
0.10
Donini et al., 2018
Rezaei et al., 2020
0.09
FERMI
0.08
0.07
0.06
0.16
0.18
0.22 Test Error
0.20
0.24 0.26
Figure 8: Comparing FERMI with other methods in the presence of outliers (random 10% of sensitive attributes in training forced to O. FERMI still achieves a better trade-off compared to all other baselines.
E.4 Effect hyperparameter on the accuracy-fairness tradeoffs
We run ERMI algorithm for the binary case to COMPAS dataset investigate the effect of hyper-parameter tuning on the accuracy-fairnes trade-off of the algorithm. As it can be observed in Fig, g by increasing X from 0 to 1000, test error (left axis, red curves) is slightly increased. On the other hand, the fairness violation (right axis, green curves) is decreased as we increase À to 1000. Moreover, for both notions of fairness (demographic parity with the solid curves and equality opportunity with the dashed curves) the trado-off between test error and fairness follows the similar pattern. To mcasure the fairness violation, we use demographic parity violation and equality of opportunity violation defined in Section equation for the solid and dashed curves respectively.
40

---

Published in Transactions on Machine Learning Research (11/2022)
COMPAS
Adult
0.31
0.225
0.205
0.175
aimass Violation
0.30
0.175
0.150
0.150
0.125
0.29
0.125
Test
0.100
0.100
Gemoggaphie Party
0.075
0.180
Demogophic Party
0.075
Equal Opportunity on
0.175
Eauil Opontnity
0.27
0.025
0.170
0.000
100
Lambda
Lambda
Figure 9: Tradeoff of fairness violation vs test error for FERMI algorithm on COMPAS and Adult datasets. The solid and dashed curves correspond to FERMI algorithm under the demographic parity and equality of opportunity notions accordingly. The left axis demonstrates the effect changing on the test error (red curves), while the right axis shows how the fairness of the model (measured by equality of opportunity or demographic parily violations) depends on changing X.
E.5 Complete version of Figure 1 (with pre- pre-processing and post-processing baselines)
In Figure 1 we compared FERMI with several stato-of-tho-art in-processing approaches. In the next three following ligures compare the in-processing approaches depicted in Figure with pre-processing and
post-processing methods including (Hardt et al. 2016 Kamiran et al. 2010 Feldman et al. 2015).
E.6 Description of datasets
All of the following datasets are publicly available at UCI repository.
German Credit Dataset. g German Credit. dataset consists of 20 features (13 categorical and 7 numerical) regarding to social, and economic status of 1000 customers. The assigned task is to classify customers as good or bad credit risks. Without imposing fairness, the DP violation of the trained model is larger than 20%. We choose 80% of customers the train data and the remaining 20% customers as the test data. The sensitive attributes are gender, and marital-status.
Adult Dataset, 10 Adult dataset contains the census information of individuals including education, gender, and capital gain. The assigned classification task is to predict whether a person earns over 50k anmually. The train and test sets are two separated files consisting of 32, 000 and 16, 000 samples respectively. We consider gender and race as the sensitive attributes (For the experiments involving one sensitive attribute, we have chosen gender). Learning a logistic regression model on the training dataset (without imposing fairness) shows that only 3 features out of 14 have larger weights than the gender attribute. Note that removing the sensitive attribute (gender), and retraining the model does not climinate the bias of the classifier. the optimal logistic regression classifier in this case is still highly biased. For the clustering lask, we have chosen contimuuus features (Capital-gain, age, fnlwgt, capital-loss, hours-per-week). and 10, 000 samples to cluster. The sensitive attribute of each individual is gender.
Communities and Crime Dataset The dataset is cleaned and processed as described in (Kearns et al. 2018). Briefly, each record in this dataset summarizes aggregate socioeconomic information about both the citizens and police force in particular U.S. community, and the problem is to predict whether the community has high rate of violent crime.
COMPAS Dataset Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) is a famous algorithm which widely used by judges for the estimation of likelihood of reoffending crimes. is
https://arciive.ic.
41

---

Published in Transactions on Machine Learning Research (11/2022)
German Dataset
Adult Dataset
Demographic Parity Violation
Demographic Parity Violation
0.20
Kamiran et al.,
0.20
Kamiran et al., 2012
Feldman al., 2015
0.18
Feldman al., 2015
0.18
Zafar, et al., 2017
Zafar, et al., 2017
0.15
0.15
Jiang al., 2020
Jiang et al., 2020
0.12
Cho et al., 2020
0.13
Cho et al., 2020
0.10
Baharlouei et al., 2020
0.10
FERMI
Baharlouei al., 2020
FERMI
0.08
0.08
0.05
0.05
0.03
0.03
0.00
0.20
0.22 0.24 0.26 0.28 0.30 0.32
2..60.1...20..22224226.....22222
Test Error
Test Error
COMPAS Dataset
Demographic Parity Violation
Kamiran et al., 2012
0.20
Feldman al., 2015
Zafar, et al., 2017
0.15
Jiang et al., 2020
Cho al., 2020
Baharlouei et al., 2020
0.10
FERMI
0.05
0.26 0.28 0.30 0.32 0.34 0.36 0.38
Test Error
Figure 10: Tradeoff of demographic parity violation vs test error for FERMI algorithm on COMPAS, German, and
Adult datasets.
observed that the algorithm is highly biased against the black defendants. The dataset contains features used by COMPAS algorithm alongside with the assigned score by the algorithm within two years of the decision.
Colored MNIST Dataset We use the code by Li Vasconcelos (2019 to create a Colored MNIST dataset with σ We use the provided LeNet-ō model trained on the colored datasct for all bascline models of [Baharloui et al. 20020); Mary el al. (019)) Cho et al. (2020b) and FERMI, where we further apply the corresponding regularizer in the training process.
42

---

Published in Transactions on Machine Learning Research (11/2022)
Adult Dataset
German Dataset
Equalized Odds Violation
Hardt et al., 2016
Equalized Odds Viola Violation
0.16
Hardt et al., 2016
0.10
Zafar al., 2017
0.14
Zafar et al., 2017
Donini et al., 2018
Donini et al., 2018
0.08
Rezaei al., 2020
0.12
Rezaei et al., 2020
Baharlouei et al., 2020
0.10
Baharlouei et al., 2020
0.06
FERMI
0.08
FERMI
0.04
0.06
0.02
0.04
0.02
0.18
0.20
0.22
0.24
0.26
0.20
0.22
0.24
0.26 0.28 0.30
Test Error
Test Error
COMPAS Dataset
Equalized Odds Violation
0.25
Hardt et al., 2016
Zafar et al., 2017
0.20
Donini al., 2018
Rezaei al., 2020
0.15
Baharlouei et al., 2020
FERMI
0.10
0.05
0.00
0.26
0.28
0.30
0.32
0.34
Test Error
Figure 11: Tradeoff of equalized odds violation vs test error for FERMI algorithm on COMPAS, German, and Adult datasets.
43

---

Published in Transactions on Machine Learning Research (11/2022)
Adult Dataset
German Dataset
Equality of Opportunity Violation
Equality of Opportunity Violation
0.10
Hardt et al., 2016
0.14
Hardt et al., 2016
Zafar al., 2017
Zafar et al., 2017
Donini et al., 2018
0.12
Donini et al., 2018
0.08
Rezsei 2020
0.10
Rezaei et al., 2020
Baharlouei et al., 2020
Baharlouei al., 2020
0.06
0.08
FERMI
0.06
0.04
0.04
0.02
0.02
0.17
0.18
0.19
0.20
0.21
0.22
0.20
0.22
0.24
0.26 0.28 0.30 Test Error
Test Error
COMPAS Dataset
Equality of Opportunity Violation
0.14
Hardt 2016
0.12
Zafar et al., 2017
Donini 2018
0.10
Rezael et al. 2020
0.08
Baharlouei et al., 2020
FERMI
0.06
0.04
0.02
0.26
0.28
0.30
0.32
0.34
0.36
Test Error
Figure 12: Tradeoff of equality of opportunity violation vs test error for FERMT algorithm COMPAS, German, and Adult datasets.
44


Reviews:
[{"review_id": "J9soZQKik7", "reviewer": "Reviewer_NTvn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper is quite strong. Theoretically, the idea of re-expressing ERMI as a max term that is an average of data points allows them to turn the problem into a min-max problem and allows the empirical estimate to be an unbiased estimate. As a result, they can use stochastic optimization problem and hence allowing them to be scalable to even very large datasets. Empirically, it’s quite impressive that they beat all other baselines that are state-of-art methods in all the settings they considered. \n\nOnly one question/complaint I have is regarding the generalization aspect of the paper. It’s claimed that this is the first paper to ‘provide fairness/accuracy tradeoff that can be achieved by a non-binary classification algorithm unseen “test data”’. However, correct me if I’m wrong, but I don’t think there’s anything in prior work’s generalization results that leveraged the binary aspect (e.g. Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness by Kearns et al.). That is, I think their generalization proofs would have worked even if the classifier considered there is not non-binary; instead of VC-dimension, other appropriate hypothesis complexity could have been used. If this is not true, I think it would be helpful to discuss not just the techniques used in this paper but the hurdles that are faced if one were to just extend other previous fairness generalization results. \n\nIn addition, is there a reason why one couldn’t have used epsilon-net argument to provide uniform convergence for the generalization result? Algorithm 1 is projecting both theta and W in some norm ball with bounded radius. So one can consider a finite cover of these two balls, apply union bound to show that for each of the cover the empirical estimate is close to the true value via Chernoff bound, and then show that every parameters in each cover must have similar due to Lipschitzness. Once again, if this approach doesn't work, I think it would be helpful to discuss why this approach (as third-person, this is the first thing that I would have tried to use to show generalization) doesn't work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes adding ERMI (exponential Renyi mutual information) into the objective as a regularizer. Different forms of ERMI can be used to encourage different fairness fairness constraints such as demographic parity, equalized odds, equalized opportunity, and so on. \n\nBy re-expressing ERMI as a max term (proposition 1), one can re-write the regularized optimization problem (Loss + ERMI) as a min-max problem. And the term inside the max can be written as an average of data points. Therefore, the average over any iid sample is an unbiased estimator of the true ERMI. This allows them to employ stochastic min-max optimization approach (i.e. stochastic gradient descent ascent) to solve the regularized optimization problem.\n\n\nThey empirically demonstrate how their algorithm outperforms all other state-of-art methods in many different settings. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I don’t have any immediate change requests. But of course, it would be possibly helpful to include some of the answers that the authors provide to the reviewers in the future version of the paper, if those discussions add clarity and further show novelty of the paper. ", "broader_impact_concerns": "I don’t have any concern on the ethical implication of the work. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Jh9P8PJMlF", "reviewer": "Reviewer_B8JB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "On the positive side, the novel contributions made by the authors are presented clearly, with clear reference to the closely related literature. While the significance of the findings is arguably not particularly high, the approach is general and clear, and the initial empirical results look to be rather encouraging. As such, I think that as a record of these insights, the paper has value and can be considered worth publication.\n\nOn the negative side, I felt there were a few points that readers might trip up on, and that hurt the overall presentation.\n\n- The comparison with Mary et al. (2019) seems rather incomplete to me. The authors say that \"their (Mary et al.'s) algorithm\" is kernel-based, i.e., that the probabilities used in ERMI are estimated using kernel density techniques, I assume. In contrast, in the authors' setup, they assume that the conditional probabilities of $\\widehat{Y}$ are known via some function $\\mathcal{F}(\\theta,X)$. While this is natural for logistic regression (and indeed most classification methods), one expects this would preclude traditional SVM-type classifiers which bypass conditional probability estimation. Is this not the case? As such, since the KDE-based approach of Mary et al. can presumably work with any underlying classifier, this is a limitation on the proposed methodology.\n\n- After Corollary 1, the authors say *\"Corollary 1 implies that for any given data set [...] is an unbiased estimator of $\\widehat{F}(\\theta,W)$ for all $i \\in [N]$ (with respect to the empirical distribution).\"* I find this puzzling; isn't the quantity mentioned an unbiased estimator with respect to the uniform random draw of $i$ from $[N]$?\n\n- The exposition feels a bit bloated in section two; are Theorems 1 and 2 really both necessary? One would expect the authors' technique to work for the population FRMI objective under iid data, and that the empirical analogue would trivially follow. Unless there is some technical reason for separating these two results (I don't see it), if it were me, I would just have shown how the min-max form yields unbiased estimators of the population objective (joint in $\\theta$ and $W$) gradient, and plugged that into a standard congerence argument.\n\n- Statements like *\"first guarantee on the fairness/accuracy tradeoff\"* are in my opinion over-selling the results. The tradeoff depends in a very deep way on the nature of the function $\\theta \\mapsto \\mathcal{F}(\\theta,X)$, and this is left completely abstract, so we really do not learn anything (in theory) about accuracy/fairness tradeoffs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The broad context of this work is machine learning under fairness constraints, where fairness is captured (in theory) using ERMI (exponential Renyi mutual information), a quantity which measures the degree of dependence that the learned predictor's outputs ($\\widehat{Y}$ in their notation) have on certain sensitive attributes $S$. Imporant previous work is that of [Mary et al., 2019], who use the same objective function (empirical risk + ERMI) and estimate the relevant joint probabilities using a standard kernel-based method. In this work, the authors leave these probabilities as a black box, assuming they are provided by a sufficiently \"nice\" function (in terms of the key parameter $\\theta$; required to be Lipschitz and smooth). They show that since the ERMI objective can be written as the maximum of a second order polynomial, the full objective (what they call \"FRMI\") can be written in a min-max form that admits easy computation of unbiased gradient estimators, and with the aforementioned \"niceness\" one can leverage existing insights for gradient-based optimizers to get guarantees of convergence to a stationary point in the FRMI objective. Of course this strategy can also can be applied to get analogous results for the empirical objective (what they call FERMI) using a stochastic mini-batch algorithm. They complement this formal proposal and analysis with empirical tests (using L-infinity checks of fairness violations), and show several settings in which their proposed approach achieves an improved accuracy-fairness tradeoff when compared with benchmark methods.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I ask that the authors consider the negative points raised above as potential avenues for refining the current manuscript.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OgLTiqB55j", "reviewer": "Reviewer_kp4G", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the paper is clear for the most part, and the theoretical analysis and experiments are novel. In what follows I include a list of points that I feel require clarification:\n- Theorem 1 is central to the paper and shouldn't be informal. Please find a way to formalize it. Also, you have conditions that are local around a solution theta^*. Do these conditions have to hold around *all* stationary points theta^*? Is the convergence then local, i.e. initialization has to be close enough to theta^*?\n- On page 4, you refer to model outputs as random, i.e. you write P(Y-hat = j|X) ... These are not actual probabilities (i.e., model output is not random) but softmax outputs. Please clarify.\n- I'd suggest including formal definitions of demographic parity, equal opportunity, equalized odds, since these are key to the paper.\n- Usually I think of fairness notions as imposing a *constraint*, and not as being a desideratum that can be added via regularization. I think it would be helpful to state explicitly that your solution does not (approximately) satisfy demographic parity, equal opportunity, equalized odds (which are typically constraints), but these only serve as regularizers. Is it possible to adapt the methods in the paper to the constrained setting? For example, we optimize the loss subject to having low empirical ERMI?\n- In my experience at least, exponential Rényi mutual information doesn't seem to be a very common notion. Is there a standard reference for this definition?\n- In Algorithm 1, please reformat the latex source so that step 6 doesn't have the awkward spacing. (Maybe define some relevant quantities outside of step 6.)\n- I don't understand how the curves in the experimental section are obtained. For example, in Figure 1, how do you obtain the points that are interpolated? Do you vary lambda used in the FERMI objective on a grid of values and you plot the resulting test error and ERMI? \n- Given that Proposition 1 is a deterministic statement, I don't see the need to refer to random variables and distributions.\n- In Theorem 2, you refer to a \"one-pass sample-without-replacement variant of Algorithm 1.\" Please be precise.\n- The middle plot in Figure 1 looks strange, like it was cut off; there is a lot of white space on the right.\n- ERMI upper bounding other fairness notions (which is discussed in the paper) sounds only relevant if we're in a constrained setting. Why is it relevant in the studied regularized setting?\n- In Proposition 2, it's a bit strange that the maximum over W is taken outside of the limit over n. I don't see how this means that the estimator is asymptotically unbiased. I would imagine the natural notion of asymptotic unbiasedness being lim_n E [D_n] =D, which is not what's happening here.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies a formulation of fair empirical risk minimization where the fairness objective can be captured via exponential Rényi mutual information and provides the first algorithm with guaranteed convergence even with stochastic gradients. The studied fairness notions include demographic parity, equalized odds, and equal opportunity. The key technical ingredient is an asymptotically unbiased estimate of the gradient of the objective function, which in turn allows showing convergence to an approximate solution of the corresponding population-level objective. Numerical experiments demonstrate that the studied algorithm achieves a superior fairness-accuracy tradeoff compared to baselines for all studied fairness notions.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please clarify the points above.", "broader_impact_concerns": "Please find a way to move the broader impact section from the appendix to the main body.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were all satisfied with the authors' responses to their question and the consequent paper revisions, recommending the paper for acceptance to TMLR. I concur, and recommend acceptance to the manuscript as is.
Decision: Accept (poster)