Title: Efficient Post-Processing for Equal Opportunity in Fair Multi-Class Classification
Venue: NeurIPS 2023
Accepted: no
Keywords: fairness

equal opportunity

multi-class

classification
Abstract: Fairness in machine learning is of growing concern as more instances of biased model behavior are documented while their adoption continues to rise.  The majority of studies have focused on binary classification settings, despite the fact that many real-world problems are inherently multi-class.  This paper considers fairness in multi-class classification under the notion of parity of true positive rates—an extension of binary class equalized odds—which ensures equal opportunity to qualified individuals regardless of their demographics.  We focus on algorithm design and provide a post-processing method that derives fair classifiers from pre-trained score functions.  The method is developed by analyzing the representation of the optimal fair classifier, and is efficient in both sample and time complexity, as it is implemented by linear programs on finite samples.  We demonstrate its effectiveness at reducing disparity on benchmark datasets, particularly under large numbers of classes, where existing methods fall short.

Abstract

Fairness in machine learning is of growing concern as more instances of biased model behavior are documented while their adoption continues to rise. The majority of studies have focused on binary classification settings, despite the fact that many real-world problems are inherently multi-class. This paper considers fairness in multi-class classification under the notion of parity of true positive rates-an extension of binary class equalized odds [23]-which ensures equal opportunity to qualified individuals regardless of their demographics. We focus on algorithm design and provide a post-processing method that derives fair classifiers from pretrained score functions. The method is developed by analyzing the representation of the optimal fair classifier, and is efficient in both sample and time complexity, as it is implemented by linear programs on finite samples. We demonstrate its effectiveness at reducing disparity on benchmark datasets, particularly under large numbers of classes, where existing methods fall short.

1 Introduction

Algorithmic fairness has emerged as a topic of significant concern in the field of machine learning, due to the potential for models to exhibit discriminatory behavior towards historically disadvantaged demographics [9, 4, 6], all while their adoption continues to rise in domains including high-stakes areas such as criminal justice, healthcare, and finance [3, 7]. To address the concern, a variety of fairness criteria have been proposed (e.g., demographic parity, equalized odds) along with mitigation methods [10, 19, 23, 26]. On classification problems, the majority of work focuses on the binary class setting [2, Table 1], where one class is typically considered to be more favorable (e.g., the approval vs. rejection of a credit card application).

Yet, many real-world problems are multi-class in nature. In the case of credit card applications, issuers may opt to assigning higher-tier interest rates to high-risk applicants rather than outright rejecting them, which creates opportunities to applicants who would otherwise be denied credit and also generates returns for the banks. Similarly, in online advertising, recruiting platforms can employ machine learning models to match users to relevant job postings across multiple occupation categories. There are evidences, however, for such systems to exhibit gender bias [8, 13, 44]; for instance, models that are trained to identify occupation from biography tend to show higher accuracy (recall) on male biographies than on their female counterparts in occupations that are historically male-dominated [14].

In the example above, unfairness is manifested in a disparity of true positive rates (TPRs) across demographic groups A (generalizing the true positive and negative rates in binary classification),

A classifier satisfying parity of TPRs, i.e., TPR a = TPR a ′ for all a, a ′ , ensures that individuals with 34

the same qualification ( Y ) will have equal opportunity of receiving their favorable outcome ( ̂ Y = Y ) 35

Boundary of feasible

region (group 1)

Boundary of feasible

region (group 2)

TPR parity feasible

region

TPR parity optimum

TPR curves (2 classes)

TPR of class 1

TPR of class 2

TPR of class 3

0

1

TPR surfaces (upper half; 3 classes)

1

0

0

1

TPR of class 2

0

1

TPR of class 1

1

0

regardless of demographics [20], e.g., being shown job postings on recruiting platforms for which the 36 user is qualified. When the classes are binary, this fairness notion recovers equalized odds [23]. 37

In this paper, we focus on the design of algorithm for mitigating TPR disparity and provide an efficient 38 post-processing method that derives attribute-aware fair classifiers from (pre-trained) scoring models. 39 Our method works on multi-class and multi-group classification problems, guarantees fairness by a 40 sample complexity bound, can be implemented by linear programs, and achieves higher reductions in 41 disparity compared to existing algorithms that are applicable to multi-class-a recently proposed post42 processing method based on model projection [2], and adversarial debiasing [41], an in-processing 43 method-especially when the number of classes is large. 44

Organization. We introduce the problem setup and objectives in Section 2, then describe our 45 post-processing method for TPR parity in Section 3, along with suboptimality analyses; in particular, 46 our method yields the optimal fair classifier when applied to the Bayes optimal score function. 47 Our method is instantiated for finite sample estimation in Section 4, and we also provide sample 48 complexity bounds to complete the analysis. Finally, in Section 5, we compare our algorithm with 49 existing methods for disparity reduction on benchmark datasets. 1 Ahigh-level summary of our results 50 is provided in Section 1.1. 51

52

1.1 Summary of Results

One way to interpret and understand TPR parity is through visualizing the feasible regions of TPRs. 53 In Fig. 1, we plot the feasible regions (achievable by probabilistic classifiers) of two groups on a 54 (hypothetical) binary classification problem on the left, and those on a three-class problem on the 55 right, where each axis represents the TPR of a class. Achieving optimal TPR parity amounts to 56 first finding the TPR that maximizes the overall utility (e.g., accuracy) in the intersection of feasible 57 regions, and subsequently an (attribute-aware) classifier attaining that target TPR on all groups. Note 58 that the left figure is equivalent to the ROC curve (with a flip of the horizontal axis, because the TPR 59 of class 1 equals one minus the false negative rate by treating class1 as the negative class), which 60 was used by Hardt et al. [23] for studying equalized odds. And thus, the TPR (hyper)surface plots in 61 higher dimensions are a natural generalization of the ROC curve to multi-class settings. 62

Step one of finding the optimal fair TPR can be formulated as a linear program when estimating from 63 finite samples. For the second step, our method derives a classifier attaining the target TPR from the 64 score function; in particular, it yields the optimal fair classifier when the score is Bayes optimal: 65

Theorem 1.1. Let f ∗ 1 , · · · , f ∗ m : X → ∆ k denote the Bayes score function on each group, f ∗ a ( x ) := E [ Y | X = x, A = a ] , and q 1 , · · · , q m ∈ ∆ k be arbitrary. Then under a continuity assumption (2.3), ∃ β 1 , · · · , β m ∈ [0 , 1] and λ 1 , · · · , λ m ∈ R k s.t. the probabilistic attribute-aware classifier

achieves the maximum utility subject to TPR parity. 66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

The post-processed classifier returned by our method is a mixture of two models (weighted by β ). Eq. (1a) returns the class with the highest likelihood after a class-wise rescaling, called a tilting [2], which generalizes the concept of thresholding in binary classifiers. Eq. (1b) makes random assignments sampled from a Multinoulli ( q ) distribution, which handles situations where the fair TPR lies in the interior of the feasible region (see Fig. 1, where the optimum is located within the interior of group 2 feasible region). To alleviate potential ethical concerns regarding this randomization, we point out that the parameter q a 's used in class sampling can be specified per-group by the practitioner responsibly, e.g., uniform 1 / k , or e y ′ with y ′ being an advantaged outcome.

Among the possibly infinitely many fair classifiers derived from the score function f , our method specifically seeks the simplistic representation in Eq. (1) because it can be estimated via linear programs from finite samples. More importantly, it immediately extrapolates to unseen examples, and provides good generalization performance at the rate of ˜ O ( √ k / n ) thanks to its low function complexity (Theorem 4.2).

When the score function being post-processed is not Bayes optimal, our method is still applicable, but the resulting classifier may not be optimal nor exactly achieve TPR parity without access to labeled data (the method itself only needs unlabeled data with the sensitive attribute) or additional knowledge of the model. But these suboptimalities are minimized if the model is calibrated (Theorem 3.5); this answers the question raised in [2] about the effects of base model inaccuracies on downstream post-processing.

1.2 Related Work

Fairness Critetia. The notion of TPR parity has appeared in the literature as conditional procedure accuracy equality [7], avoiding disparate mistreatment [39], and (multi-class) equal opportunity [14, 29, 31] (to be distinguished from the fairness criterion with the same name in [23]). Other group fairness notions that extend to multi-class include (but not limited to) equalized odds [23] (of which TPR parity is a necessary condition), and demographic parity (DP) [10] (where Xian et al. [35] recently proposed an optimal post-processing method). However, DP may be less desirable than TPR parity in some use cases because the perfect classifier is not permitted under DP when the base rates differ [42]. It is worth noting that TPR parity implies accuracy parity [9]. In addition to group fairness, there are notions defined on the individual level [19].

Mitigation Methods. Our method is based on post-processing [25, 23]. There are also in-processing methods via fair representation learning [40, 41, 43, 30] or solving zero-sum games [1, 36], and pre-processing methods that debias the data prior to model training [11, 44]; see [4, 12] for a survey.

For multi-class TPR parity, the only applicable post-processing method to date, to our knowledge, is due to Alghamdi et al. [2] (which is the primary baseline for our method in our experiments). It is a general-purpose method that transforms the scores to satisfy fairness while minimizing the distributional divergence (e.g., KL) between the transformed scores and the original. However, the tradeoff between model performance and fairness is unclear as they did not relate the divergence to utility. Furthermore, while the authors provided a sample complexity bound for their optimization objective, it is not explicitly related to the violation of the fairness criteria.

2 Preliminaries

A k -class classification problem is defined by a joint distribution µ of input X ∈ X , demographic group membership A ∈ [ m ] := { 1 , · · · , m } (a.k.a. the sensitive attribute), and class label Y ∈ [ k ] . We denote the joint distribution of ( X,A ) by µ X,A , and, the ( k -1) -dimensional probability simplex by ∆ k := { z ∈ R k ≥ 0 : ‖ z ‖ 1 = 1 } .

Let f : X ×A → ∆ k be an attribute-aware (pre-trained) score function, whose outputs are probability vectors that estimate the class probabilities as in f ( x, a ) y ≈ P µ ( Y = y | X = x, A = a ) . We will write f a : X → ∆ k to denote the component of f associated with group a , i.e., f a ( x ) ≡ f ( x, a ) . Our goal is to find fair (probabilistic) post-processing maps g 1 , · · · , g m : ∆ k →Y to derive a classifier ( x, a ) ↦→ g a ◦ f a ( x ) that satisfies TPR parity while maximizing utility (e.g., classification accuracy).

We allow for controllable tradeoffs between utility and fairness through the following relaxation of TPR parity, and call a classifier α -fair if it satisfies α -TPR parity:

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

Definition 2.1 (Approximate TPR Parity) . Let α ∈ [0 , 1] . A predictor ̂ Y is said to satisfy α -TPR parity if ∆ TPR ( ̂ Y ) ≤ α , where

and TPR a ( ̂ Y ) := P ( ̂ Y | Y = y, A = a ) ∈ [0 , 1] k ; P includes the randomness of the predictor.

Beyond classification accuracy, we also allow for any utility functions that depend only on the TPRs: 2 Definition 2.2 (Utility) . The utility function u : [ k ] × [ k ] → R is defined for some υ ∈ R k by

E.g., accuracy, 1 [ y = ˆ y ] , is obtained by setting υ = 1 k . The term υ will appear in our analyses, and the significance of considering utilities of this form is that we could evaluate a classifier by a weighted sum of its TPRs. Define p ay := P µ ( A = a, Y = y ) , then

Finally, we make the following continuity assumption on the distributions of score to avoid technical complexities related to tie-breaking (on the atoms). This assumption has also appeared in prior work on fair post-processing [16, 21, 35]; it holds when the input distributions are continuous and the score function is injective, or can be satisfied by adding small random perturbations to the scores.

Assumption 2.3. The conditional distribution of score, P ( f a ( X ) | A = a ) , is (Lebesgue absolutely) continuous, ∀ a ∈ [ m ] .

3 TPR Parity via Post-Processing

Given a score function f : X × A → ∆ k , and access to the (unlabeled) joint distribution µ X,A (i.e., no estimation error), we describe a method for deriving an attribute-aware α -fair classifier while maximizing utility, in the form of ( x, a ) ↦→ g a ◦ f a ( x ) , where the g a 's are (probabilistic) fair post-processing maps for each group. That is, we want to solve

Although the method only returns classifiers derived from f as opposed to searching over the space of all classifiers h : X × A → Y , it would yield the optimal fair classifier provided that the information of ( A,Y ) is preserved in the output of f ; this is the case when the score function is Bayes optimal.

3.1 Deriving Optimal Fair Classifier From Bayes Score Function

In this section, we explain how to obtain an optimal fair classifier by deriving from the Bayes score function f ∗ , thereby providing a proof of Theorem 1.1 (omitted proof are deferred to the appendix).

Step 1 (Finding Utility-Maximizing Fair TPRs) . Let D a ⊆ [0 , 1] k denote the set of feasible TPRs on group a achieved by probabilistic classifiers. The first step is to find utility-maximizing fair TPRs contained in an glyph[lscript] ∞ -ball of diameter α per Definition 2.1 of α -TPR parity (left figure of Fig. 2):

When α = 0 , this reduces to finding a single t ∈ ⋂ a D a , and because each D a is convex (since probabilistic classifiers are allowed), it can be found with ternary search as suggested in [23]. If instead the t a 's are to be estimated from finite samples, then the empirical ̂ D a 's are described by polytopes and the problem can be formulated as a linear program (Section 4).

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

TPR of class 1

1

0

TPR of class 2

0

1

β

1 -

β

q

∆

2

α

TPR of class 2

0

1

TPR of class 2

0

1

TPR of class 1

1

0

TPR of class 1

1

0

The feasible regions of TPR generally differ across groups, due to uncertainties that are inherent to each group in the task of interest, or to inadequate and biased collection or sourcing of data. The more the D a 's differ, the greater the tradeoff between fairness and utility; hence TPR parity incentivizes the practitioner to improve data collection and aspects of modeling that induces a balanced predictive capability on all groups [23].

Because f ∗ ( X,A ) is sufficient statistic for Y , the fair TPRs we found above are always achievable by classifiers derived from f ∗ . Or more concretely,

Proposition 3.1. Let f ∗ : X → ∆ k denote the Bayes score function, then D := { TPR ( h ) ∈ [0 , 1] k | h : X → Y (probabilistic) } = { TPR ( g ◦ f ∗ ) ∈ [0 , 1] k | g : ∆ k →Y (probabilistic) } .

Step 2 (Obtaining Fair Classifier of Desired Form) . Having found the utility-maximizing fair TPR t a 's, the next step is to derive a classifier that attains t a on each group. This is provided by the following theorem:

Theorem 3.2. Let f ∗ : X → ∆ k denote the Bayes score function, and q ∈ ∆ k be arbitrary. Then under Assumption 2.3, ∀ t ∈ D , there exists β ∈ [0 , 1] and λ ∈ R k s.t. TPR ( h ) = t , where

The construction uses the observation that the boundary of D , denoted by ∂D , is given by the set of TPRs attained by tiltings of the Bayes score:

glyph[negationslash]

Proposition 3.3. Let f ∗ : X → ∆ k denote the Bayes score function, then h : X → Y (probabilistic) satisfies TPR ( h ) ∈ ∂D if and only if ∃ λ ∈ R k , λ = 0 s.t. h ( x ) ∈ arg max y λ y f ∗ ( x ) y .

Proof of Theorem 3.2. If the target TPR lies on the boundary of D , then by Proposition 3.3, it is achieved by a tilting of the Bayes score without any randomization (i.e., β = 0 ; center figure of Fig. 2). This holds due to Assumption 2.3, because we may break ties arbitrarily without affecting TPR, since the set of tied scores (finite union of ( k -2) -d subspaces) has (Lebesgue) measure zero.

Otherwise, and generally, there must exists t ′ ∈ ∂D and β ∈ [0 , 1] s.t. t can be written as a linear combination of t = βq +(1 -β ) t ′ . This is simply because q ∈ ∆ k ⊆ D , and the line connecting q and t must intersect ∂D at some point t ′ (right figure of Fig. 2). Since the TPR of the input-agnostic randomization according to Multinoulli ( q ) equals q , and t ′ is achieved by a tilting of the score per Proposition 3.3, their β -mixture achieves the target TPR t by linearity.

3.2 Deriving From Any Score Function

The post-processing method described in the previous section, which only requires unlabeled data ( X,A ) , yields the optimal α -fair classifier when applied to Bayes scores f ∗ . Yet, in practice, there

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

is the concern that Bayes score functions could be arbitrarily complex and are often not exactly learnable due to limited data or computational constraints [34].

Nonetheless, our method is still applicable to arbitrary (approximations to the Bayes) score functions f : X × A → ∆ k for deriving classifiers that are approximately fair and optimal, by treating them as if they were Bayes optimal (Algorithm 1). Where, the only tweak we made is replacing the ground-truth TPRs and feasible regions (which are unknown without access to the Bayes score) by approximations induced by f , i.e.,

where

It is not hard to show that they are equal to their ground-truth counterparts when f = f ∗ .

We may control and minimize the suboptimalities of the classifier returned from Algorithm 1 by performing group-wise distribution calibration to the score function f (using labeled data ( X,A,Y ) ): Definition 3.4 (Distribution Calibration) . A score R is said to be (group-wise) distribution calibrated if P ( Y = y | R = s ) = s y , ∀ s ∈ ∆ k , y ∈ [ k ] (resp. P ( Y = y | R = s, A = a ) = s y , ∀ a ∈ [ m ] ).

Distribution calibration is a multi-class generalization of the original definition of calibration for binary predictors [15, 32], requiring the predicted score to match the underlying class distribution conditioned on the score across all classes, not just the most confident one [22]. Although this definition is convenient to work with mathematically, it could be difficult to achieve in practice. In the proof of Theorem 3.5, we relax it to a recently proposed notion of decision calibration [45] (w.r.t. the set of all tiltings; derived from multicalibration [24]), which could be achieved in polynomial time.

Theorem 3.5. Let f : X × A → ∆ k be a score function, and h : X × A → Y the (probabilistic) classifier derived from f using Algorithm 1. Then under Assumption 2.3, for any group-wise calibrated reference score function ¯ f : X × A → ∆ k ,

where p ay := P µ ( A = a, Y = y ) , υ is from the utility function in Definition 2.2, U denotes the utility achieved by the optimal α -fair classifier derived from the calibrated reference ¯ f , and

is the L 1 ( µ ) difference between f and the calibrated reference ¯ f on group a and class y .

glyph[negationslash]

We draw two conclusions from this result. First, by using the Bayes score function f ∗ as the reference, it states that the suboptimality of the derived classifier when f = f ∗ is upper bounded by the difference between the approximate scores and the ground-truth; this answers the question raised in [2] regarding the impact of base model inaccuracies. Second, if f satisfies calibration, then by using itself as the reference, the result guarantees that the classifier derived using Algorithm 1 exactly achieves the desired level of fairness, and is optimal among all fair classifiers derived from f (which cannot be improved without labeled data).

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

4 Finite-Sample Algorithm and Guarantees

We instantiate the post-processing method above for TPR parity to the case where we do not have access to the distribution µ X,A but only samples drawn from it (i.e., to perform estimation), and analyze the sample complexity.

Assumption 4.1. We have n i.i.d. (unlabeled) samples of ( X,A ) , which are independent of the score function f being post-processed.

Denote the number of samples from group a by n a , and the samples themselves by ( x a,i ) i ∈ [ n a ] .

4.1 Algorithm

Weadapt Algorithm 1 to handle finite samples by replacing ˜ D a and U with their empirical counterparts (essentially calling it with the empirical distribution ˆ µ X,A formed by the samples as the argument), and implement the optimization problems on Lines 3, 5 and 6 using linear programs.

Step 1 (Finding Utility-Maximizing Fair TPRs) . The empirical induced feasible region of TPRs, ̂ D a , can be computed via evaluating the TPRs of all (probabilistic) classifiers acting on the samples-by representing them using n a × k lookup tables (each row gives the probabilities of the random class assignment on the corresponding sample):

where

(cf. Line 2 and Eqs. (5) and (6)). Note that ̂ D a is a polygon, as it is specified by linear constraints.

To obtain the utility-maximizing fair TPR ˆ t a 's, we take the empirical maximizer subject to the α -TPR constraint via solving a linear program (cf. Line 3 and Eqs. (3) and (4)):

where ̂ U ( ˆ t 1 , · · · , ˆ t m ) := ∑ a,y υ y ˆ p ay ( ˆ t a ) y is the empirical utility.

Step 2 (Obtaining Fair Classifier of Desired Form) . The next step is finding a classifier that achieves ˜ t a 's on the empirical distribution, i.e., Lines 5 and 6. To implement Line 5, note that another way of approaching this problem is to realize that among all eligible ( β a , h a ) -pairs, the h a associated with the maximum β a value must satisfy ˜ TPR a ( h a ) ∈ ∂ ˜ D a (otherwise, a contradiction can be reached using the fact that ˜ D a ⊆ [0 , 1] k is compact; also see the right figure of Fig. 2). Combined with the strategy above of representing classifiers using lookup tables, we get the following linear program:

Finally, on Line 6, we find a tilting λ a s.t. after coordinate-wise multiplied by the scores, the argmax class assignment has nonzero probability according to the classifier γ a found in the preceding step:

The feasible set of this problem is nonempty by Proposition 3.1, because we are treating f as if it were the Bayes score function, and the empirical distribution ˆ µ X,A as the population.

All combined, our algorithm involves solving (2 m +1) linear programs, where LP1 is the dominating one with O ( nk ) variables and constraints; solving which (to near-optimality) takes, e.g., ˜ O ( poly ( nk )) time using interior point methods [33].

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

4.2 Sample Complexity

Thanks to the low function complexity of post-processing maps used in our algorithm to derive classifiers (Eq. (1)), it enjoys the following efficient sample complexity:

Theorem 4.2. Let f : X × A → ∆ k be a score function, and h : X × A → Y the (probabilistic) classifier derived from f using Algorithm 1 with the empirical distribution formed by samples from Assumption 4.1 as the argument. Then under Assumption 2.3, for any group-wise calibrated (Definition 3.4) reference score function ¯ f : X × A → ∆ k , and n ≥ Ω( max a,y ln ( mk / δ )/ p ay ) ,

where U denotes the utility achieved by the optimal α -fair classifier derived from the calibrated reference ¯ f , and glyph[epsilon1] ay := E [ | ¯ f a ( X ) y -f a ( X ) y | · 1 [ A = a ]] .

The bound consists of a calibration error glyph[epsilon1] ay as discussed in the remarks of Theorem 3.5, an estimation error from applying uniform convergence (the Natarajan dimension of the set of tiltings is O ( k ) ), and a k / n term that comes from the disagreement over class assignments on the samples between the (deterministic) tilting found on Line 6 and the (probabilistic) classifier on Line 5 due to tie-breaking.

5 Experiments

We evaluate Algorithm 1 for reducing TPR disparity on benchmark datasets, and demonstrate its effectiveness compared to existing post-processing as well as in-processing bias mitigation methods.

Datasets. The first task is income prediction, for which, we use the ACSIncome dataset [18]-an extension of the UCI Adult dataset [27] with much more examples (1.6 million vs. 30,162), allowing us to compare methods confidently. We consider a binary setting where the sensitive attribute is gender and the target is whether the income is over $50k, as well as a multi-group multi-class setting with five race categories and five income buckets. The second is text classification, of identifying occupations (28 in total) from biographies in the BiasBios dataset [14]; sensitive attribute is gender.

Baselines and Setup. The main baseline is FairProjection [2]-the only post-processing algorithm applicable for multi-class TPR parity to our knowledge. 3 In the binary setting, we also compare to RejectOption [25]. To demonstrate the deficiencies of existing methods at reducing TPR disparity, we additionally include in-processing results using Reductions [1] and Adversarial [41]. 45

On each task, we first create a pre-training split from the dataset and train a linear logistic regression scoring model (with isotonic calibration and five-fold cross-validation as implemented in scikitlearn [37, 38, 28]), then randomly split the remaining data for post-processing and testing with 10 different seeds and aggregate the results (the pre-trained model remains the same). For inprocessing, we use the same splits but merge the pre-training and post-processing data for training. On BiasBios , linear logistic regression is performed on the embeddings of the biographies computed by a previously fine-tuned BERT model [17] (in other words, head-tuning). Additional details including hyperparameters are included in the appendix.

Results. In Fig. 3, we plot the tradeoff curves from varying the fairness tolerance ( α for our method). Our method is consistently the most effective at minimizing TPR disparity, particularly under multi-class settings, where existing algorithms only manage to partially reduce ∆ TPR (and at a greater cost to accuracy when using FairProject and RejectOption ). It also outperforms

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

0.772

0.774

0.776

0.778

ACSIncome

0.00

0.05

0.10

0.15

0.725

0.750

0.472

0.478

ACSIncome

0.2

0.4

0.6

0.42

0.43

0.44

0.855

0.856

BiasBios

0.2

0.4

0.82

0.83

(2 genders, 2 classes)

(5 races, 5 classes)

(2 genders, 28 occupations)

Violation of TPR parity,

TPR

Classification accuracy

Ours

FairProjection-KL

RejectOption

Reductions

Adversarial (MLP)

the in-processing Reductions on binary ACSIncome , and Adversarial in terms of ∆ TPR , which, although enjoys higher accuracies because of the use of the more expressive feedforward networks as the prediction model, fails to reduce TPR parity. Sharper drops in accuracies are observed when applying our method with small α settings, e.g., 0 . 001 to 0 . 0001 . We saw this happen when the randomized component in Eq. (1b) is activated (i.e., β > 0 ), meaning that Line 3 finds a fair TPR that lies in the interior of the feasible region of the better-performing group in order to match the feasible TPR on the worse-performing one(s). Hence the drop is expected because utility is being sacrificed to achieve TPR parity.

Although our method greatly reduces TPR disparity, there remains a gap to reaching ∆ TPR = 0 , especially on tasks with more classes (i.e., BiasBios , where a higher variance is also observed). While this could be due to miscalibration, or potentially a violation of Assumption 2.3, the main reason is suspected to be insufficient sample size. Recall from Theorem 4.2 that the sample complexity for ∆ TPR scales as ˜ O ( √ k / np ay ) in the worse-case ( a, y ) , which is itself at least ˜ O ( √ mk 2 / n ) . Thus, learning generalizable classifiers that satisfy TPR parity under more groups and classes is much harder in terms of data requirement (and by extension, computing resource).

Lastly, we emphasize the necessity of group-wise calibration for achieving low ∆ TPR , as the definition of the criterion involves conditioning on the true label (it is also reflected by the calibration error term glyph[epsilon1] ay in Theorem 4.2). In an ablation study in the appendix, a larger (minimum achievable) ∆ TPR is observed when no efforts are made to calibrate the scoring model. It is therefore necessary for model vendors to provide accurate uncertainty quantifications, and for practitioners building fair classifiers to verify and improve calibration.

6 Conclusions and Limitations

Wedescribed a post-processing method for reducing TPR disparity for equal opportunity in multi-class classification, and demonstrated its performance in comparison to existing algorithms on benchmarks datasets, especially when the number of classes is large. We analyzed the sample complexity of our method, and established its optimality under model calibration.

The effectiveness of our method at reducing TPR disparity is largely contributed to the tailored analysis, although it limits our method to this fairness notion only. Some use cases may demand equalized odds ( ̂ Y ⊥ A | Y ) beyond TPR parity ( 1 [ ̂ Y = Y ] ⊥ A | Y ), which is a more stringent criterion: TPR parity only needs to match the main diagonal of the (conditional) confusion matrix across groups, whereas equalized odds requires matching all k 2 entries. The design of efficient algorithms for achieving equalized odds remains an open problem. 6

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

References

Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A Reductions Approach to Fair Classification. In Proceedings of the 35th International Conference on Machine Learning , pages 60-69, 2018.

Wael Alghamdi, Hsiang Hsu, Haewon Jeong, Hao Wang, P. Winston Michalak, Shahab Asoodeh, and Flavio P. Calmon. Beyond Adult and COMPAS: Fair Multi-Class Prediction via Information Projection. In Advances in Neural Information Processing Systems , 2022.

Solon Barocas and Andrew D. Selbst. Big Data's Disparate Impact. California Law Review , 104(3):671-732, 2016.

Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and Machine Learning: Limitations and Opportunities . MIT Press, 2023.

Rachel K. E. Bellamy, Kuntal Dey, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, Seema Nagar, Karthikeyan Natesan Ramamurthy, John Richards, Diptikalyan Saha, Prasanna Sattigeri, Moninder Singh, Kush R. Varshney, and Yunfeng Zhang. AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias, 2018. arxiv:1810.01943 [cs.AI] .

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? . In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , pages 610-623, 2021.

Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. Fairness in Criminal Justice Risk Assessments: The State of the Art. Sociological Methods & Research , 50 (1):3-44, 2021.

Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. In Advances in Neural Information Processing Systems , volume 29, 2016.

Joy Buolamwini and Timnit Gebru. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 2018 Conference on Fairness, Accountability, and Transparency , pages 77-91, 2018.

Toon Calders, Faisal Kamiran, and Mykola Pechenizkiy. Building Classifiers with Independency Constraints. In 2009 IEEE International Conference on Data Mining Workshops , pages 13-18, 2009.

Flavio P. Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and 348 Kush R. Varshney. Optimized Pre-Processing for Discrimination Prevention. In Advances in 349 Neural Information Processing Systems , volume 30, 2017. 350

Simon Caton and Christian Haas. Fairness in Machine Learning: A Survey, 2020. 351 arxiv:2010.04053 [cs.LG] . 352

353

354

355

Jeffrey Dastin. Amazon scraps secret AI recruiting tool that showed bias against women. Reuters , oct 2018. URL https://www.reuters.com/article/us-amazon-com-jobsautomation-insight-idUSKCN1MK08G .

356

357

358

359

360

Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting. In Proceedings of the 2019 ACM Conference on Fairness, Accountability, and Transparency , pages 120-128, 2019.

Morris H. DeGroot and Stephen E. Fienberg. The Comparison and Evaluation of Forecasters. 361 Journal of the Royal Statistical Society. Series D (The Statistician) , 32(1/2):12-22, 1983. 362

Christophe Denis, Romuald Elie, Mohamed Hebiri, and François Hu. Fairness guarantee in 363 multi-class classification, 2023. arxiv:2109.13642 [math.ST] . 364

365

366

367

368

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, 2019.

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring Adult: New Datasets for Fair Machine Learning. In Advances in Neural Information Processing Systems , 2021.

Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness Through Awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference , pages 214-226, 2012.

Executive Office of the President. Big Data: A Report on Algorithmic Systems, Opportunity, and Civil Rights. The White House, 2016. URL https://purl.fdlp.gov/GPO/gpo90618 .

Solenne Gaucher, Nicolas Schreuder, and Evgenii Chzhen. Fair learning with Wasserstein barycenters for non-decomposable performance measures. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , pages 2436-2459, 2023.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning , pages 1321-1330, 2017.

Moritz Hardt, Eric Price, and Nathan Srebro. Equality of Opportunity in Supervised Learning. In Advances in Neural Information Processing Systems , volume 29, 2016.

Úrsula Hébert-Johnson, Michael P. Kim, Omer Reingold, and Guy N. Rothblum. Multicalibration: Calibration for the (Computationally-Identifiable) Masses. In Proceedings of the 35th International Conference on Machine Learning , pages 1939-1948, 2018.

Faisal Kamiran, Asim Karim, and Xiangliang Zhang. Decision Theory for DiscriminationAware Classification. In 2012 IEEE 12th International Conference on Data Mining , pages 924-929, 2012.

Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness. In Proceedings of the 35th International Conference on Machine Learning , pages 2564-2572, 2018.

Ron Kohavi. Scaling Up the Accuracy of Naive-Bayes Classifiers: A Decision-Tree Hybrid. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining , pages 202-207, 1996.

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research , 12(85):2825-2830, 2011.

Preston Putzel and Scott Lee. Blackbox Post-Processing for Multiclass Fairness. In Proceedings of the Workshop on Artificial Intelligence Safety 2022 , volume 3087, 2022.

Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 7237-7256, 2020.

Julien Rouzot, Julien Ferry, and Marie-José Huguet. Learning Optimal Fair Scoring Systems for Multi-Class Classification. In 2022 IEEE 34th International Conference on Tools with Artificial Intelligence , 2022.

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

Hao Song, Tom Diethe, Meelis Kull, and Peter Flach. Distribution Calibration for Regression. In Proceedings of the 36th International Conference on Machine Learning , pages 5897-5906, 2019.

Pravin M. Vaidya. Speeding-up linear programming using fast matrix multiplication. In 30th Annual Symposium on Foundations of Computer Science , pages 332-337, 1989.

Blake Woodworth, Suriya Gunasekar, Mesrob I. Ohannessian, and Nathan Srebro. Learning Non-Discriminatory Predictors. In Proceedings of the 2017 Conference on Learning Theory , pages 1920-1953, 2017.

Ruicheng Xian, Lang Yin, and Han Zhao. Fair and Optimal Classification via Post-Processing Predictors. In Proceedings of the 40th International Conference on Machine Learning , 2023.

Forest Yang, Mouhamadou Cisse, and Sanmi Koyejo. Fairness with Overlapping Groups. In Advances in Neural Information Processing Systems , volume 33, pages 4067-4078, 2020.

Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In Proceedings of the Eighteenth International Conference on Machine Learning , pages 609-616, 2001.

Bianca Zadrozny and Charles Elkan. Transforming Classifier Scores into Accurate Multiclass Probability Estimates. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 694-699, 2002.

Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P. Gummadi. Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment. In Proceedings of the 26th International Conference on World Wide Web , pages 1171-1180, 2017.

Richard Zemel, Yu Ledell Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning Fair Representations. In Proceedings of the 30th International Conference on Machine Learning , pages 325-333, 2013.

Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating Unwanted Biases with Adversarial Learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society , pages 335-340, 2018.

Han Zhao and Geoffrey J. Gordon. Inherent Tradeoffs in Learning Fair Representations. Journal of Machine Learning Research , 23(57):1-26, 2022.

Han Zhao, Amanda Coston, Tameem Adel, and Geoffrey J. Gordon. Conditional Learning of Fair Representations. In International Conference on Learning Representations , 2020.

Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) , pages 15-20, 2018.

Shengjia Zhao, Michael P. Kim, Roshni Sahoo, Tengyu Ma, and Stefano Ermon. Calibrating 445 Predictions to Decisions: A Novel Approach to Multi-Class Calibration. In Advances in Neural 446 Information Processing Systems , 2021. 447

Peer reviews:
comments: SUMMARY:
This paper considers fairness in multi-class classification under the notion of parity of true positive rates - an extension of binary class equalized odds - which ensures equal opportunity to qualified individuals regardless of their demographics. We focus on algorithm design and provide a post-processing method that derives fair classifiers from pretrained score functions.

STRENGTHS:
- paper deals with an interesting problem
- paper is technically sound

WEAKNESSES:
- paper does not properly compare w.r.t. the state of the art 
- novelty of the proposal is not clear
- paper is hard to read and follow

QUESTIONS:
Paper does not properly compare w.r.t. the state of the art, there are plenty of work (even in NeurIPS) that are not even mentioned in the paper, why these work are not mentioned? 
Novelty of the proposal is not clear because of a proper state-of-the-art review and then it is not clear the contribution. Can you please elaborate and descrive the technical challenges of the contribution?

LIMITATIONS:
Paper is very hard to read, follow, and fully understand.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel post-processing approach to reduce the true positive rate parity for multi-class classification problems. It is shown on two real world data to outperform an existing baseline in terms of accuracy and true positive rate parity.\n\nSTRENGTHS:\n1. The proposed approach is novel and technically sound.\n\n2. The proposed approach guarantees fairness by a sample complexity bound.\n\n3. The proposed approach outperforms an existing baseline post-processing approach in terms of reducing TPR parity.\n\n4. The presentation is clear.\n\nWEAKNESSES:\n1. Some related work, such as [45], is mentioned in the paper but not compared in the experiments.\n\n2. Fairness in multi-class problems seems to be a rather trivial/incremental problem when there are plenty of approaches solving the fairness problems in binary classification and regression problems. Approaches from fairness in regression (e.g. Narasimhan et al., \"Pairwise fairness for ranking and regression\", 2020) should be easily applied to solve the fairness problems in multi-class classification. Please justify more for why Fairness in multi-class classification is a problem worth studying. Some approaches from fairness in binary-classification (e.g. Kamiran and Calders, \"Data preprocessing techniques for classification without discrimination\", 2012; Yu et al., \"FairBalance: How to Achieve Equalized Odds With Data Pre-processing\", 2023) can also be easily adapted to multi-class problems.\n\nQUESTIONS:\nMy main concern is about the significance and contribution of fairness in multi-class classification. To me, this is a relatively trivial problem compared to fairness in binary-classification and regression. Many approaches from fairness in binary-classification and regression can be easily adapted to solve this problem.\n\nQ1. Why not compare this approach to [45]?\n\nQ2. Why is fairness in multi-class classification is an important problem? Can existing work from fairness in binary-classification or regression be applied to solve this problem?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work proposes a post-processing algorithm to achieve the equal opportunity constraint in multi class classification. The proposed algorithm takes arbitrary Bayes rule estimate and only requires additional unlabelled data. The authors derive finite-sample guarantees and perform empirical evaluation to support their claims.\n\nI did not check the math, but, having prior experience in this area it looks believable.\n\nSTRENGTHS:\nThe paper is well written and the proposed methodology is sound. It extends a rather long line of works on post-processing with unlabelled data.\n\nWEAKNESSES:\nIn the context of the paper, I do not see major weaknesses from the methodological side, but rather remarks that are presented in the next part. \n\nFrom the theoretical part, I could mention that the devision by p_{a, y} with large number of protected attribute and classes can make the bound non-informative. \n\nI think that the expectation in the fairness guarantee is not well placed, I expected to have  $E[\\Delta(h)]$.\n\nQUESTIONS:\n1. Could the authors formulate a theorem, stating explicit form of the optimal fair classifier in their setup? \n2. In the very end of conclusion, the authors say that designing post-processing algorithm for equal odds remain open. While formally correct, isn't it true that one can do exactly the same analysis?\n3. While assumption 2.3 did appear in many previous works, all of them were dealing purely with deterministic classifiers (regressors). Actually, the only purpose of this assumption is to avoid randomization on some tie-breaks. Since you still do randomise, why do you need this assumption?\n4. Many works explicitly present the form of the optimal fair classifier (e.g., [16, 21, A, B] ...), which is not the case here (it is more opaque here). Could you sate the result unpacking the exact form?\n5. What's the issue to generalise it to attribute unaware case? One only needs to perform a post-processing on top of the Bayes rule and A | X, Y or A | X.\n6. Could the authors discuss computation complexity of their method, explicating the dependency on all the parameters?\n\nTo the best of my knowledge the first use of unlabelled data for post-processing in binary classification with equal opportunity was done in [A]. In [A], authors only consider the binary case and exact fairness, while targeting the optimal fair classifier.\n\n\n[A] E. Chzhen, C. Denis, M. Hebiri, L. Oneto, M. Pontil, Leveraging Labeled and Unlabeled Data for Consistent Fair Binary Classification, 2019\n[B] N. Schreuder, E. Chzhen, Classification with abstention but without disparities\n\nLIMITATIONS:\n---", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies algorithmic fairness in multiclass classification setting. The fairness notion considered is parity of true positive rates (TPR) which is the multi class analog to equalized odds. The paper gives a post-processing algorithm which, given a score function, outputs a fair classifier. The paper then gives sample and time complexity guarantees and experimental evaluation on benchmark datasets. \n\nThis paper furthers work studied Alghmadi et al., which according to the authors, is the only other post-processing method available for multi-class TPR parity. \n\nThe paper gives a general post-processing algorithm that takes as input a score function and outputs a classifier that satisfies approx TPR party. The authors show that if you begin with the Bayes score function, their post-processing returns an optimal, fair classifier. They also give results showing that if the initial score function is not Bayes optimal but instead satisfies a decision calibration condition, then the post-processing is optimal among all classifiers that can be derived from the initial score function. \n\nThe post-processing method consists of two parts. The first part of the process estimates the feasible region of TPRs and then finds the utility-maximizing TPRs that satisfy fairness constraints (either exactly via search if we know the distribution, or via a linear program if we are estimating the TPRs from data). The second step involves finding the hypothesis that achieves these TPRs, which either corresponds to a tilting - essentially a threshold - of the score function without randomization or a mixture of two models that lie on the boundary. \n\nFinally, the paper gives some experimental evaluation on three benchmark datasets that serves as a proof of concept of the post-processing and showed that it is competitive with other standard techniques (notably reductions).\n\nSTRENGTHS:\nFairness via post-processing in multiclass classification settings is not a focus on most prior work on algorithmic fairness, which makes this an interesting and novel contribution. \n\nAlthough restricted to TPR parity, the paper does a complete analysis of the topic under this fairness definition. The paper addresses both not knowing the underlying distribution and so possibly starting from a non-Bayes score function and also discuss estimating the parameters from finite samples.\n\nWEAKNESSES:\nThe paper gets a little notationally and technically dense in Section 3. While the presentation is still fairly clear, I think additional higher level exposition to describe in particular Step 2 of the algorithm could be beneficial to readers - maybe including additional description of a tilting.\n\nQUESTIONS:\nWhat are the assumptions on group structure? Must they be disjoint? This might be useful to clarify.\n\nLIMITATIONS:\nThe authors address one of the main limitations of this work that their method is confined to only reducing TPR disparity and not other fairness notions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}