Title: Individual Arbitrariness and Group Fairness
Venue: NeurIPS 2023
Accepted: yes
Keywords: predictive multiplicity

fairness in machine learning

Rashomon effect
Abstract: Machine learning tasks may admit multiple competing models that achieve similar performance yet produce conflicting outputs for individual samples---a phenomenon known as predictive multiplicity. We demonstrate that fairness interventions in machine learning optimized solely for group fairness and accuracy can exacerbate predictive multiplicity. Consequently, state-of-the-art fairness interventions can mask high predictive multiplicity behind favorable group fairness and accuracy metrics. We argue that a third axis of ``arbitrariness'' should be considered  when deploying models to aid decision-making in applications of individual-level impact.
To address this challenge, we propose an ensemble  algorithm applicable to any fairness intervention that provably ensures  more consistent predictions.

Individual Arbitrariness and Group Fairness

Abstract

Machine learning tasks may admit multiple competing models that achieve similar performance yet produce arbitrary outputs for individual samples-a phenomenon known as predictive multiplicity. We demonstrate that fairness interventions in machine learning optimized solely for group fairness and accuracy can exacerbate predictive multiplicity. Consequently, state-of-the-art fairness interventions can mask high predictive multiplicity behind favorable group fairness and accuracy metrics. We argue that a third axis of 'arbitrariness' should be considered when deploying models to aid decision-making in applications of individual-level impact. To address this challenge, we propose an ensemble algorithm applicable to any fairness intervention that provably ensures more consistent predictions.

1 Introduction

Non-arbitrariness is an important facet of non-discriminatory decision-making. Substantial arbitrariness exists in the training and selection of machine learning (ML) models. By simply varying hyperparameters of the training process (e.g., random seeds in model training), we can produce models with arbitrary outputs on individual input samples [9, 13, 26, 35]. The phenomenon where distinct models exhibit similar accuracy but arbitrary individual predictions is called predictive multiplicity 1 [35]. The arbitrary variation of outputs due to unjustified choices made during training can disparately impact individual samples, i.e., predictive multiplicity is not equally distributed across inputs of a model. When deployed in high-stakes domains (e.g., medicine, education, resume screening), the arbitrariness in the ML pipeline may target and cause systemic harm to specific individuals by excluding them from favorable outcomes [7, 15, 40].

Popular fairness metrics in the ML literature do not explicitly capture non-arbitrariness. A widely recognized notion of non-discrimination is group fairness . Group fairness is quantified in terms of, for example, statistical parity [18], equal opportunity, equalized odds [23], and variations such as multi-accuracy [30] and multi-calibration [24]. Broadly speaking, methods that control for group fairness aim to guarantee comparable performance of a model across population groups in the data. The pursuit of group fairness has led to hundreds of fairness interventions that seek to control for performance disparities while preserving accuracy [25].

The central question we tackle in this paper is: Do models corrected for group fairness exhibit less arbitrariness in their outputs? We answer this question in the negative . We demonstrate that stateof-the-art fairness interventions may improve group fairness metrics at the expense of exacerbating arbitrariness. The harm is silent: the increase in arbitrariness is masked by favorable group fairness and accuracy metrics. Our results show that arbitrariness lies beyond the fairness-accuracy frontier:

ENEM(RandomForest)

1.0

ps

0.5

0.66

Score

0.8

ScoreS

0.4

0.6

Post-Intervention

0.3

0.64

EqOdds

Reduction

Rejection

6

0.4

Reduction High Fair

(Acc:0.65,Mean EO:0.04)

0.2

0.62

LevEqOpp

Fair Projection

0.2

Reduction Low Fair

(Acc:0.66,Mean EO:0.15)

0.1

Base

0.0

(Acc:0.66,MeanEO:0.28)

Baseline

0.0

0.00

0.05

MeanEqualizedOdds

0.10

0.15

0.20

0.25

0.0

0.1

0.2

0.3

0.4

0.5

0.0

0.1

0.2

0.3

0.4

0.5

ClassifierScoreStd.

predictive multiplicity should be accounted for in addition to usual group-fairness and accuracy metrics during model development.

Figure 1 illustrates how fairness interventions can increase predictive multiplicity. Here, state-ofthe-art fairness interventions are applied 4 to a baseline random forest classifier to ensure group fairness (mean equalized odds [23], see Definition 6) in a student performance binary prediction task. We produce multiple baseline classifiers by varying the random seed used to initialize the training algorithm. Each baseline classifier achieves comparable accuracy and fairness violation. They also mostly agree in their predictions: for each input sample, the standard deviation of output scores across classifiers is small (see Definition 3). After applying a fairness intervention to each randomly initialized baseline classifier, we consistently reduce group fairness violations at a small accuracy cost, as expected. However, predictive multiplicity changes significantly post-intervention: for roughly half of the students, predictions are consistent across seeds, whereas for 20% of the students, predictions are comparable to a coin flip. For the latter group, the classifier output depends on the choice of a random seed instead of any specific input feature. The increase in predictive multiplicity is masked by the fairness-accuracy curve, does not impact all samples equally, and is consistent across datasets and learning tasks.

At first, the increase in predictive multiplicity may seem counter-intuitive: adding fairness constraints to a learning task should reduce the solution space, leading to less disagreement across similarlyperforming classifiers relative to an unconstrained baseline. We demonstrate that, in general, this is not the case. For a given hypothesis class, the non-convex nature of group fairness constraints can in fact increase the number of feasible classifiers at a given fairness and accuracy level. We show that this phenomenon occurs even in the simple case where the hypothesis space is comprised of threshold classifiers over one-dimensional input features, and the optimal baseline classifier is unique. To address this challenge, we demonstrate - both theoretically and through experiments that ensembling classifiers is an effective strategy to counteract this multiplicity increase.

The main contributions of this work include 5 :

We demonstrate that the usual 'fairness-accuracy' curves can systematically mask an increase of predictive multiplicity. Notably, applying state-of-the-art fairness interventions can incur higher arbitrariness in the ML pipeline.

We show that multiplicity can be arbitrarily high even if group fairness and accuracy are controlled, when we do not have perfect classifiers. Hence, fairness interventions optimized solely for fairness and accuracy cannot, in general, control predictive multiplicity. We also provide examples of why fairness constraints may exacerbate arbitrariness.

We propose an ensemble algorithm that reduces multiplicity while maintaining fairness and accuracy. We derive convergence rate results to show that the probability of models disagreeing drops exponentially as more models are added to the ensemble.

We demonstrate the multiplicity phenomena and benchmark our ensemble method through comprehensive experiments using state-of-the-art fairness interventions across real-world datasets.

1.1 Related Works

Multiplicity, its implications, and promises. Recent works have investigated various factors that give rise to multiplicity. D'Amour et al. [17] studied how under-specification presents challenges to the credibility of modern machine learning algorithms. More precisely, under-specified optimization problems in machine learning admit a plethora of models that all attain similar performance, and which model to deploy in practice may ultimately depend on arbitrary choices of the randomization made during training procedure [3]. The arbitrariness of the model could potentially harm the reproducibility of model predictions [5], and hence the credibility of the conclusion made thereof.

Creel and Hellman [15] thoroughly explored the notion of arbitrariness in machine learning and discuss how high-multiplicity predictions can lead to systematized discrimination in society through 'algorithmic leviathans.' Multiplicity in prediction and classification can also have beneficial effects. Black et al. [7], Semenova et al. [39], and Fisher et al. [20] view multiplicity of equally-performing models as an opportunity to optimize for additional criteria such as generalizability, interpretability, and fairness. Coston et al. [14] develop a framework to search over the models in the Rashomon set for a better operation point on the accuracy-fairness frontier. However, they do not discuss the potential predictive multiplicity cost of existing fairness interventions nor propose algorithms to reduce this cost.

The work most similar to ours is [13]. Cooper et al. [13] consider the problem of predictive multiplicity as a result of using different splits of the training data. Therein, they quantify predictive multiplicity by prediction variance, and they propose a bagging strategy [8] to combine models. Our work considers a different problem where predictive multiplicity is exacerbated by group-fairness interventions. Our work is also different from Cooper et al. [13] as we fix the dataset when training models and consider multiplicity due to randomness used during training. In this sense, our ensemble algorithm is actually a voting ensemble [41] (see Section 4); see also ensembling and reconciliation strategies proposed by Black et al. [6] and Roth et al. [38] that aim to create more consistent predictions among competing models. To the best of the authors' knowledge, we are the first to measure and report the arbitrariness cost of fairness interventions.

Hidden costs of randomized algorithms. Recent works [21, 31, 32] examine the potential detrimental consequences of randomization in the ML pipeline. In their empirical study, Ganesh et al.[21] observe that group fairness metrics exhibit high variance across models at different training epochs of Stochastic Gradient Descent (SGD). The authors point out that random data reshuffling in SGD makes empirical evaluation of fairness (on a test set) unreliable, and they attribute this phenomenon to the volatility of predictions in minority groups. Importantly, they do not incorporate fairness interventions in their experiments. In contrast, we apply fairness interventions to baseline models. Specifically, we examine the variance in predictions among models with similar fairness and accuracy performances. In addition to the observations made by Ganesh et al., our theoretically-grounded study reveals the different paths that lead to group-fairness, i.e., that arbitrariness can be an unwanted byproduct of imposing fairness constraints. Krco et al. [31] empirically study if fairness interventions reduce bias equally across groups, and examine whether affected groups overlap across different fairness interventions. In contrast, our work examines the multiplicity cost of group fairness and its tension with individual-level prediction consistency, rather than the fairness cost of randomness in the ML pipeline. Another work on the hidden cost of randomized algorithms is given by Kulynych et al. [32], who report that well-known differentially-private training mechanisms can exacerbate predictive multiplicity.

In an early work [34], Lipton et al. indirectly points to the potential arbitrary decision on individuals as a result of imposing group fairness constraints. They give an illustrative example using synthetic hiring data to show that a fair model resorts to using irrelevant attribute (hair length) to make hiring decision in order to achieve near-equal hiring rate for men and women.

2 Problem Formulation and Related Works

We explain the setup and relevant definitions in this section.

Prediction tasks. We consider a binary classification setting with training examples being triplets ( X , S, Y ) with joint distribution P X ,S,Y . Here, X is an R d -valued feature vector, S is a discrete random variable supported on [ K ] ≜ { 1 , · · · , K } representing K (potentially overlapping) group memberships, and Y is a binary (i.e., { 0 , 1 } -valued) random variable denoting class membership. 6 We consider probabilistic classifiers in a hypothesis space H , where each h ∈ H is a mapping h : R d → [0 , 1] . Each value of a classifier h ( x ) aims to approximate P Y | X = x (1) . The predicted labels ̂ y can be obtained by thresholding the scores, e.g., ̂ y = 1 { h ( x ) ≥ 0 . 5 } , where 1 { · } is the indicator function. Finally, we denote by ∆ c the probability simplex over c dimensions.

Randomized training procedures and the Rashomon set. We assume access to the following:

a training dataset of n i.i.d samples D ≜ { ( x i , s i , y i ) } n i =1 drawn from P X ,S,Y ;

a randomized training procedure T ; and

an induced distribution T ( D ) on the hypothesis class of predictors H .

We denote a sampled classifier by h ∼ T ( D ) , which can be sampled, for example, using different random seeds at the beginning of the execution of procedure T on D . For concreteness, the above data may for example correspond to the following practical setting.

Example 1. The dataset D can be comprised of resumes of individuals applying for a job, and the training procedure T is an algorithm to predict whether to extend an interview opportunity for an applicant. For example, T can be a neural network with unspecified hyperparameters (e.g., random seed that needs to be chosen at the outset); alternatively, T can be the same pre-trained neural network composed with a fairness intervention method. The classifiers considered will be the last layer of the neural network (or the classifier after fairness enhancement), which will belong to a hypothesis class H determined by the chosen neural network architecture. By varying the random seed, say, m times, we would obtain

independent classifiers, denoted by

h

1

,

· · ·

, h

m

∼ T

(

D

)

.

♦

We are interested in detecting whether competing classifiers (i.e., deployed for, and performing similarly in the same prediction task) have conflicting predictions non-uniformly across individuals. Next, we define the set of competing models obtained from the randomized training procedure T .

For a loss function ℓ : [0 , 1] × { 0 , 1 } → R + , finite dataset D ⊂ R d × [ K ] × { 0 , 1 } , and classifier h : R d → [0 , 1] , we let the empirical loss incurred by h on D be denoted by ℓ ( h ; D ) ≜ |D| -1 ∑ ( x ,s,y ) ∈D ℓ ( h ( x ) , y ) . The ( empirical ) Rashomon set ( ϵ -level set) of competing models is defined as the set of models with loss lower than ϵ [26], i.e., R ( H , D , ϵ ) ≜ { h ∈ H : ℓ ( h ; D ) ≤ ϵ } . We extend the definition of the Rashomon set to take into consideration the effect of the randomized algorithm T , as follows.

Definition 1 (Empirical Rashomon Set of Randomized Training Procedure) . Fix a finite dataset D , a hypothesis class H , and a randomized training procedure T inducing the distribution T ( D ) on H . Given a loss function ℓ : [0 , 1] ×{ 0 , 1 } → R + and a parameter ϵ > 0 , we define the empirical Rashomon set with m models induced by T as the collection of m classifiers independently sampled from T ( D ) and having empirical loss less than ϵ :

Here, ϵ is an approximation parameter that determines the size of the set. The set ̂ R m ( T , D , ϵ ) can be viewed as an approximation of the Rashomon set of 'good' models [9, 35, 40], and indeed we have the inclusion ̂ R m ( T , D , ϵ ) ⊂ R ( H , D , ϵ ) where H = supp ( T ( D )) . Note that the set ̂ R m ( T , D , ϵ ) is itself random even for a fixed dataset D , where the source of randomness is coming from the distribution T ( D ) . In the sequel, we omit the arguments of ̂ R m ( T , D , ϵ ) when they are clearly implied from context.

i.i.d.

There are various metrics to quantify predictive multiplicity across models in ̂ R m by either considering their output scores [40] or thresholded predictions [35]. e focus on two metrics: 1) ambiguity for evaluating the predictive multiplicity of thresholded predictions, and 2) cumulative distribution function (CDF) of standard deviation (std.) of output scores when model outputs are in the interval [0 , 1] (interpreted as the probability of the positive class). Those two metrics are defined as follows.

Definition 2 (Ambiguity [35]) . Fix a dataset D = { ( x i , s i , y i ) } i ∈ [ n ] ⊂ R d × [ K ] ×{ 0 , 1 } and a finite set of models R⊂H . Let f ( r ) ≜ 1 { r ≥ 0 . 5 } be the thresholding function. The ambiguity of a dataset over the set of models R is the proportion of points in the dataset that can be assigned a conflicting prediction by a competing classifier within R :

̸

To define the CDF of std. of scores, we first delineate what we mean by empirical std. of scores. Definition 3 (Std. of Scores) . Fix a finite set of models R = { h j } j ∈ [ m ] ⊂ H . The empirical standard deviation (std.) of scores for a sample x ∈ R d relative to R is defined by

where ¯ µ x ≜ 1 m ∑ j ∈ [ m ] h j ( x ) denotes the empirical mean (over R ) of the scores.

Further, to understand the std. of scores of the population, we consider the empirical cumulative distribution function of the std. of the scores, defined as follows.

Definition 4 (Quantiles of std. of Scores) . Fix a dataset D = { ( x i , s i , y i ) } i ∈ [ n ] ⊂ R d × [ K ] ×{ 0 , 1 } and a finite set of models R⊂H . We define the empirical cumulative distribution function of the std. of the scores by (where s ( x , R ) is the empirical std. as in Definition 3)

Example 2. Consider a resume screening task where the algorithm decides whether to extend an interview opportunity. If ̂ F D , R (0 . 5) = 90% , then for 10% of the individuals in the dataset, the predictions produced by the competing models are arbitrary and conflicting: regardless of the mean scores, with an std. of at least 0.5, there would exist models with scores falling above and below the one-half threshold, so the thresholded output can be both 0 (no interview) and 1 (offer interview). ♦

A note on related metrics. An alternative measurement of score variation is Viable Prediction Range as defined in [40], which measures the difference in max and min scores among competing models on each individual. For thresholded scores, the original definition of Ambiguity [35] considers the proportion of a flip in prediction with respect to a baseline model (from empirical risk minimization with fixed hyperparameters and randomness). Since we consider randomized training procedures with no clear baseline model, the definition for ambiguity above is a variation of the original.

Group fairness. We consider three group fairness definitions for classification tasks-statistical parity (SP), equalized odds (EO), and overall accuracy equality (OAE) [11, 18, 23, 37]. OAE and Mean Equalized Odds (MEO) are defined below as they are used in the next sections, and we refer the reader to Appendix D for the remaining definitions.

Definition 5 (Overall Accuracy Equality, OAE) . Let ̂ Y be the predicted label obtained, e.g., from thresholding the scores of a classifier h : R d → [0 , 1] . The predictor ̂ Y satisfies overall accuracy equality (OAE) if its accuracy is independent of the group attribute: for all groups s, s ′ ∈ [ K ] ,

For binary classification, SP boils down to requiring the average predictions to be equal across groups, while EO requires true positive rates (TPR) and false positive rates (FPR) to be calibrated. In this paper, we consider mean equalized odds (MEO): the average of absolute difference in FPR and TPR for binary groups S ∈ { 0 , 1 } . We consider binary group since this is the setup for most fairness intervention methods.

Definition 6 (Mean Equalized Odds, MEO [4, 23]) . Let ̂ Y be the predicted label, S ∈ { 0 , 1 } denotes binary group membership. Mean Equalized Odds is the average odds difference for binary groups:

To examine whether current fairness intervention methods lead to an exacerbation of multiplicity, we survey state-of-the-art intervention methods, including Reductions [1], Fair Projection [2], Reject Options [29], and EqOdds [23]. We offer a brief discussion of their mechanism in Appendix D.

3 Orthogonality of Fairness and Arbitrariness

We discuss next why arbitrariness is a third axis not captured by fairness and accuracy. Models with similar fairness and accuracy metrics can differ significantly in predictions. Moreover, a set of fair and approximately accurate models can attain maximal predictive multiplicity. We also explore through an example one fundamental reason why adding fairness constraints can lead to more arbitrariness.

̸

Example 3 (Ambiguity = OAE) . Overall Accuracy Equality (OAE, Definition 5) does not capture the ambiguity of model outputs (Definition 2). Consider two hypothetical models that are fair/unfair and exhibit high/low predictive multiplicity in Figure 2. Here, in each panel, the rectangle represents the input feature space and the shaded regions represent the error region of each model.

In the top left panel, both Model 1 and 2 have equal accuracy for both groups, since the proportion of the error regions (red stripes and pink shade) for both groups are the same. Hence, both models are considered group-fair in terms of OAE. However, the error regions of the two models are disjoint. Since ambiguity is measured by the percentage of the samples that receive conflicting predictions from either models, samples from the union of the two error regions contribute to ambiguity. Hence, Model 1 and 2 bear high predictive multiplicity despite being group-fair.

In the lower right panel, Model 1 and 2 attain low fairness and low predictive multiplicity. Both models have higher accuracy for Group 2 than Group 1, so they are both unfair. The error regions completely overlap, which means that the two models are making the same error-ambiguity is 0. ♦

The schematic diagram in Figure 2 shows that predictive multiplicity is not captured by OAE. Indeed, ambiguity of a collection of models is a global property (i.e., verified at the collection level), whereas OAE is a local property (i.e., verified at the classifier level). Hence, one should not a priori expect that a set of competing models each satisfying OAE would necessarily comprise a Rashomon set with favorable ambiguity.

We prove the orthogonality of OAE and Statistical Parity (SP) from ambiguity formally in the proposition below, where we show that it is possible to construct classifiers with very stringent accuracy and perfect fairness constraint, albeit with maximal ambiguity. We determine the Rashomon set using the 0-1 loss:

̸

Fair

Unfair

High Predictive

Multiplicity

Low Predictive

Multiplicity

Group

1

Positive

Label

Y = 1

Negative

Label

Y = 0

Group

2

Group

1

Positive

Label

Y = 1

Negative

Label

Y = 0

Group

2

Positive

Label

Y = 1

Group

1

Negative

Label

Y = 0

Group

2

Negative

Label

Y = 0

Group

1

Positive

Label

Y = 1

Group

2

Model 1's

Error Region

Model 2's

Error Region

where ̂ y i ∈ { 0 , 1 } is the class membership of x i predicted by h . We prove the following orthogonality in Appendix A.

Proposition 3.1 (Orthogonality of OAE/SP and Ambiguity) . Fix any empirical loss value 0 < ϵ ≤ 1 2 and any number of models m> 1 ϵ . Then, for some finite dataset D ⊂ R d × [ K ] ×{ 0 , 1 } , there is a realization of the empirical Rashomon set ̂ R m = { h j } j ∈ [ m ] satisfying the following simultaneously:

DataDistribution

Probof Error and EOofThresholdClassifiers

0.5

ProbabilityofError

0.8

Pr[X|Y=1,S=0]

0.4

Mean EO

Pr[X|Y=0,S=0]

Mean EO=0.1

0.6

Pr[X|Y=1,S=1]

Pr[X|Y=0,S=1]

0.3

PDF

0.4

0.2

0.2

0.0

-3

-2

-1

0

1

2

3

=1

-2

1

1

2

X

Each h j has 0-1 loss upper bounded by ℓ 0-1 ( h j ; D ) ≤ ϵ ;

Each h j satisfies OAE perfectly, or each h j satisfies SP perfectly;

The collection ̂ R m has the worst ambiguity, i.e., α ( D , ̂ R m ) = 100% .

Remark 1. For OAE, such Rashomon set ̂ R m exists for any dataset D satisfying the two conditions:

with n k denoted the number of samples in D belonging to group k ∈ [ K ] , the greatest common divisor of the n k is at least ( m -1) / ( mϵ -1) .

if ( x , s, y ) , ( x , s ′ , y ′ ) ∈ D share the same feature vector x , then y = y ′ too.

The requirement that the n k share a large enough common divisor is used in the proof to guarantee perfect OAE. One could relax this requirement at the cost of nonzero OAE violation.

Proposition 3.1 implies that there exists a dataset and competing models for which all samples receive conflicting predictions. Specifically, we can construct a large enough set of competing models ( m> 1 ϵ ) such that 100% of the samples in the dataset can receive conflicting predictions from this set of perfectly fair models with respect to OAE.

In the next example, we demonstrate that, counter-intuitively, adding a fairness constraint can enlarge the set of optimal models, thereby increasing predictive multiplicity. This points to a fundamental reason why adding fairness constraints can lead to more arbitrariness in model decisions.

Example 4 (Arbitrariness of Threshold Classifiers with Fairness Constraint) . Given a data distribution of a population with two groups (Figure 3 Left ), consider the task of selecting a threshold classifier that predicts the true label. Without fairness considerations, the optimal threshold is 0 - i.e., assigning positive predictions to samples with x > 0 and negative predictions to x ≤ 0 minimizes the probability of error (Figure 3 Right ). This optimal model is unique. Adding a fairness constraint that requires Mean EO ≤ 0 . 1 , the previously optimal classifier at 0 (with Mean EO = 0 . 15 , Right ) does not meet the fairness criteria. Searching over the threshold classifiers that minimize the probability of error while satisfying Mean EO constraint yields two equally optimal models (red and blue dots Right ) with distinct decision regions (red and blue arrows Left ). Even in this simple hypothesis class, the addition of fairness constraints yields multiple models with indistinguishable fairness and accuracy but with distinct decision regions. The arbitrary selection between these points can lead to arbitrary outputs to points near the boundary. ♦

4 Ensemble Algorithm for Arbitrariness Reduction

To tackle the potential arbitrariness cost of fairness intervention algorithms, we present a disparityreduction mechanism through ensembling. We provide theoretical guarantees and numerical benchmarks to demonstrate that this method significantly reduces the predictive multiplicity of fair and accurate models.

In a nutshell, given competing models h 1 , · · · , h m , we argue that the disparity in their score assignment can be reduced by considering a convex combination of them, defined as follows.

Definition 7 (Ensemble Classifier) . Given m classifiers { h 1 , · · · , h m : R d → [0 , 1] } and a vector λ ∈ ∆ m , we define the λ -ensembling of the h j to be the convex combination h ens , λ ≜ ∑ j ∈ [ m ] λ j h j .

4.1 Concentration of Ensembled Scores

We prove in the following result that any two different ensembling methods agree for fixed individuals with high probability. Recall that we fix a dataset D and a set of competing models T ( D ) coming from a stochastic training algorithm T (see Section 2). All proofs are provided in Appendix B.

Theorem 4.1 (Concentration of Ensembles' Scores) . Let h 1 , . . . , h m ; ˜ h 1 , . . . , ˜ h m iid ∼ T ( D ) be 2 m models drawn from T ( D ) , and h ens , λ , ˜ h ens , γ be the ensembled models (constructed with { h 1 , . . . , h m } and { ˜ h 1 , . . . , ˜ h m } respectively) for λ , γ ∈ ∆ m (see Definition 7) satisfying ∥ λ ∥ 2 2 , ∥ γ ∥ 2 2 ≤ c/m for an absolute constant c . For every x ∈ R d and ν ≥ 0 , we have the exponentially-decaying (in m ) bound

In particular, for any validation set D valid. ⊂ R d of size |D valid. | = n , we have the uniform bound

4.2 Concentration of Predictions Under Ensembling

The above theorem implies that we can have a dataset of size that is exponential in the number of accessible competing models and still obtain similar scoring for any two ensembled models (uniformly across the dataset).

In practice, one cares more about the agreement of the final prediction of the classifiers. The following result extends Theorem 4.1 to the concentration of thresholded classifiers. For this, we need to define the notion of confident classifiers .

Definition 8 (Confident Classifier) . Fix a probability measure P X over R d and constants δ, θ ∈ [0 , 1] . We say that a classifier h : R d → [0 , 1] is ( P X , δ, θ ) -confident if P (∣ ∣ h ( X ) -1 2 ∣ ∣ < δ ) < θ .

In other words, h is a confident classifier if it is 'more sure' of its predictions. We observe in experiments that models corrected by fairness interventions have scores concentrated around 0 and 1.

Using confident classifiers, we are able to extend Theorem 4.1 to thresholded ensembles, as follows.

Theorem 4.2. Let h ens , λ , ˜ h ens , γ be as in Theorem 4.1, and assume that both ensembled classifiers are ( P X , δ, θ ) -confident in the sense of Definition 8. Let f ( t ) ≜ 1 { t ≥ 0 . 5 } be the thresholding function. For any set D valid. ⊂ R d of size |D valid. | = n , we may guarantee the probability of agreement in the predictions for all samples under the two ensembles to be at least

We note that in the fairness-intervention setting, the set D valid. in the above theorem would be chosen as the subset of samples having the same group attribute. Thus, the size n 0 of D valid. would be significantly smaller than the total size of the dataset, and the parameter θ then can be required to be moderately small.

Remark 2. In Appendix C, we discuss how to optimize the ensembling parameters λ . In the next section, we will stick to the uniform ensembling: h ens , λ = 1 m ∑ j ∈ [ m ] h j , i.e., λ = 1 m 1 . This simple uniform ensemble suffices to illustrate the main goal of this paper: that arbitrariness can be a by-product of fairness intervention methods, and ensembling can mitigate this unwanted effect.

5 Experimental Results

We present empirical results to show that arbitrariness is masked by favorable group-fairness and accuracy metrics for multiple fairness intervention methods, baseline models, and datasets 7 . We also demonstrate the effectiveness of the ensemble in reducing the predictive multiplicity of fair models.

Setup and Metrics. We consider three baseline classifiers (BASE): random forest (RF), gradient boosting (GBM), and logistic regression (LR), implemented by Scikit-learn [36]. By varying the random seed, we obtain 10 baseline models with comparable performance. Then, we apply various state-of-the-art fairness methods (details in Appendix D) on the baseline models to get competing fair models.

On the test set, we compute mean accuracy, Mean EO (Definition 6), and predictive multiplicity levels on competing models before and after fairness interventions. We use ambiguity (Definition 2) and score standard deviations (Definition 3) as metrics for predictive multiplicity.

Datasets. We report predictive multiplicity and benchmark the ensemble method on three datasets two datasets in the education domain: the high-school longitudinal study (HSLS) dataset [27, 28] and the ENEM dataset [16] (see Alghamdi et al. [2] Appendix B.1), and the UCI Adult dataset[33] which is based on the US census income data. The ENEM dataset contains Brazilian college entrance exam scores along with student demographic information and socio-economic questionnaire answers (e.g. if they own a computer). After pre-processing, the dataset contains 1.4 million samples with 139 features. Race is used as the group attribute S , and Humanities exam score is used as the label Y . Scores are quantized into two classes for binary classification. The race feature S is binarized into White and Asian ( S = 1 ) and others ( S = 0 ). The experiments are run with a smaller version of the dataset with 50k samples. Complete experimental results can be found in Appendix E.

Results that Reveal Arbitrariness. We juxtapose the fairness-accuracy frontier and metrics for predictive ambiguity to reveal arbitrariness masked by favorable group-fairness and accuracy metrics in Figure 1 and 4. Starting with 10 baseline classifiers by varying the random seed used to initialize the training algorithm, we apply the fair interventions REDUCTION [1], REJECTION [29], LEVERAGING [12] to obtain point clouds of models with comparable fairness and accuracy metrics. In Figure 4, we take models that achieve very favorable accuracy and MEO metrics (in blue rectangle in Left ) and plot the std. of scores to illustrate predictive multiplicity Right . Group fairness violations are greatly reduced (from 0.28 in baseline to 0.04 in fair models) at a small accuracy cost (from 67% in baseline to 66% in fair models). However, there is higher arbitrariness.

Compared to baseline (red curve), fair models corrected by REDUCTION and ROC produce lower score arbitrariness for the bottom 50% but much higher arbitrariness for the top 50% of samples; importantly, the induced arbitrariness becomes highly nonuniform across different individuals after applying the two fairness intervention. We observe that LEVERAGING produce models that agree on 90% of the samples, thereby not inducing concerns of arbitrariness.

Remarkably, arbitrariness does not vary significantly among models with different fairness levels. We consider two sets of models trained with high and low fairness constraints using REDUCTION in Figure 1.

Results on the Effectiveness of Ensembling. We pair our proofs in Section 4 with experiments that demonstrate the concentration of scores of ensembled models. In Figure 5 Left , taking the competing

HSLS

Ave EO violation of

Ensemble: 0.06

Ave EO violation of

Ensemble: 0.04

models in the high-fairness bins corrected with REDUCTION that achieve an Mean EO violation of 0 . 04 but very high score std. for half of the samples (blue rectangle in Figure 4), we ensemble the models with increasing number of models per ensemble ( m ) ranging from 1 to 30. For each m , we measure std. of scores in 10 such ensembles. The top percentile std. of the ensembled fair models drops to baseline with 30 models. Similar convergence occur on the HSLS dataset. Importantly, the ensembled models are still fair, the Mean EO violations of the ensembled models remain low.

6 Final Remarks

We demonstrate in this paper that arbitrariness is a facet of responsible machine learning that is orthogonal to existing fairness-accuracy analyses. Specifically, fairness-vs-accuracy frontiers are insufficient for detecting arbitrariness in the predictions of group-fair models: two models can have the same fairness-accuracy curve while at the same time giving widely different predictions for subsets of individuals. We demonstrate this undesirable phenomenon both theoretically and experimentally on state-of-the-art fairness intervention methods. Furthermore, towards mitigating this arbitrariness issue, we propose an ensemble algorithm, where a convex combination of several competing models is used for decision-making instead of any of the constituent models. We prove that the scores of the ensemble classifier concentrate, and that the ensuing predictions can be made to concentrate under mild assumptions. Importantly, we exhibit via real-world experiments that our proposed ensemble algorithm can reduce arbitrariness while maintaining fairness and accuracy.

Limitations. The proposed framework for estimating the predictive multiplicity of fairness interventions requires re-training multiple times, limiting its applicability to large models. We consider model variation due to randomness used during training. In practice, competing models may exist due to inherent uncertainty (i.e., a non-zero confidence interval) when evaluating model performance on a finite test set. In this regard, models with comparable average performance can be produced by searching over this Rashomon set even if training is deterministic (e.g., equivalent to solving a convex optimization problem).

Future directions. An interesting future direction is to explore the multiplicity cost of fairness interventions in such deterministic settings. Furthermore, our ensembling strategy may not guarantee that the ensemble classifier ensures fairness constraints due to the non-convex nature of such constraints. Though we empirically observe that fairness constraints are indeed satisfied by the ensemble model, proving such guarantees theoretically would be valuable.

Societal impacts. While fairness intervention algorithms can effectively reduce the disparate impact among population groups, they can induce predictive multiplicity in individual samples. The increase in predictive multiplicity does not impact all individuals equally. If predictive multiplicity caused by fairness interventions is not accounted for, some individuals will bear the brunt of arbitrary decision-making-their predictions could be arbitrary upon re-training the classifier using different random initializations, leading to another level of disparate treatment to certain population groups.

Acknowledgements. The authors would like to thank Jamelle Watson-Daniels, Arpita Biswas and Bogdan Kulynych for hepful discussions on initial ideas. This material is based upon work supported by the National Science Foundation under grants CAREER 1845852, CIF 1900750, CIF 2312667 and FAI 2040880, and by Meta Ph.D. fellowship.

Disclaimer. This paper was prepared by Hsiang Hsu prior to his employment at JPMorgan Chase & Co.. Therefore, this paper is not a product of the Research Department of JPMorgan Chase & Co. or its affiliates. Neither JPMorgan Chase & Co. nor any of its affiliates makes any explicit or implied representation or warranty and none of them accept any liability in connection with this paper, including, without limitation, with respect to the completeness, accuracy, or reliability of the information contained herein and the potential legal, compliance, tax, or accounting effects thereof. This document is not intended as investment research or investment advice, or as a recommendation, offer, or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction.

References

Agarwal, A., Beygelzimer, A., Dudík, M., Langford, J., and Wallach, H. (2018). A reductions approach to fair classification. In International Conference on Machine Learning , pages 60-69. PMLR.

Alghamdi, W., Hsu, H., Jeong, H., Wang, H., Michalak, P., Asoodeh, S., and Calmon, F. (2022). Beyond adult and compas: Fair multi-class prediction via information projection. Advances in Neural Information Processing Systems , 35:38747-38760.

Bahri, D. and Jiang, H. (2021). Locally adaptive label smoothing for predictive churn. arXiv preprint arXiv:2102.05140 .

Bellamy, R. K., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovi´ c, A., et al. (2019). Ai fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias. IBM Journal of Research and Development , 63(4/5):4-1.

Bhojanapalli, S., Wilber, K., Veit, A., Rawat, A. S., Kim, S., Menon, A., and Kumar, S. (2021). On the reproducibility of neural network predictions. arXiv preprint arXiv:2102.03349 .

Black, E., Leino, K., and Fredrikson, M. (2021). Selective ensembles for consistent predictions. arXiv preprint arXiv:2111.08230 .

Black, E., Raghavan, M., and Barocas, S. (2022). Model multiplicity: Opportunities, concerns, and solutions. In 2022 ACM Conference on Fairness, Accountability, and Transparency , pages 850-863.

Breiman, L. (1996). Bagging predictors. Machine learning , 24:123-140.

Breiman, L. (2001). Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical science , 16(3):199-231.

Caton, S. and Haas, C. (2020). Fairness in machine learning: A survey. arXiv preprint arXiv:2010.04053 .

Chouldechova, A. (2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data , 5(2):153-163.

Chzhen, E., Denis, C., Hebiri, M., Oneto, L., and Pontil, M. (2019). Leveraging labeled and unlabeled data for consistent fair binary classification. Advances in Neural Information Processing Systems , 32.

Cooper, A. F., Barocas, S., De Sa, C., and Sen, S. (2023). Variance, self-consistency, and arbitrariness in fair classification. arXiv preprint arXiv:2301.11562 .

Coston, A., Rambachan, A., and Chouldechova, A. (2021). Characterizing fairness over the set of good models under selective labels. In International Conference on Machine Learning , pages 2144-2155. PMLR.

Creel, K. and Hellman, D. (2022). The algorithmic leviathan: arbitrariness, fairness, and opportunity in algorithmic decision-making systems. Canadian Journal of Philosophy , 52(1):2643.

Cury, C. R. J. (2022). Instituto nacional de estudos e pesquisas educacionais anísio teixeira: uma trajetória em busca de uma educação de qualidade.

D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., et al. (2022). Underspecification presents challenges for credibility in modern machine learning. The Journal of Machine Learning Research , 23(1):1023710297.

Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., and Roth, A. L. (2015). Preserving statistical validity in adaptive data analysis. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing , pages 117-126.

Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. (2012). Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference , ITCS '12, page 214-226, New York, NY, USA. Association for Computing Machinery.

Fisher, A., Rudin, C., and Dominici, F. (2019). All models are wrong, but many are useful: Learning a variable's importance by studying an entire class of prediction models simultaneously. J. Mach. Learn. Res. , 20(177):1-81.

Ganesh, P., Chang, H., Strobel, M., and Shokri, R. (2023). On the impact of machine learning randomness on group fairness. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency , FAccT '23, pages 1789-1800, New York, NY, USA. Association for Computing Machinery.

Hajek, B. and Raginsky, M. (2019). Statistical learning theory. Lecture Notes , 387.

Hardt, M., Price, E., and Srebro, N. (2016). Equality of opportunity in supervised learning. Advances in neural information processing systems , 29.

Hébert-Johnson, U., Kim, M., Reingold, O., and Rothblum, G. (2018). Multicalibration: Calibration for the (computationally-identifiable) masses. In International Conference on Machine Learning , pages 1939-1948. PMLR.

Hort, M., Chen, Z., Zhang, J. M., Sarro, F., and Harman, M. (2022). Bia mitigation for machine learning classifiers: A comprehensive survey. arXiv preprint arXiv:2207.07068 .

Hsu, H. and Calmon, F. d. P. (2022). Rashomon capacity: A metric for predictive multiplicity in probabilistic classification. arXiv preprint arXiv:2206.01295 .

Ingels, S. J., Pratt, D. J., Herget, D. R., Burns, L. J., Dever, J. A., Ottem, R., Rogers, J. E., Jin, Y., and Leinwand, S. (2011). High school longitudinal study of 2009 (hsls: 09): Base-year data file documentation. nces 2011-328. National Center for Education Statistics .

Jeong, H., Wang, H., and Calmon, F. P. (2022). Fairness without imputation: A decision tree approach for fair prediction with missing values. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 9558-9566.

Kamiran, F., Karim, A., and Zhang, X. (2012). Decision theory for discrimination-aware classification. In 2012 IEEE 12th international conference on data mining , pages 924-929. IEEE.

Kim, M. P., Ghorbani, A., and Zou, J. (2019). Multiaccuracy: Black-box post-processing for fairness in classification. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society , pages 247-254.

Krco, N., Laugel, T., Loubes, J.-M., and Detyniecki, M. (2023). When mitigating bias is unfair: A comprehensive study on the impact of bias mitigation algorithms. arXiv preprint arXiv:2302.07185 .

Kulynych, B., Hsu, H., Troncoso, C., and Calmon, F. P. (2023). Arbitrary decisions are a hidden cost of differentially-private training. In Conference on Fairness, Accountability, and Transparency (FAccT) . ACM.

Lichman, M. et al. (2013). Uci machine learning repository, 2013.

Lipton, Z., McAuley, J., and Chouldechova, A. (2018). Does mitigating ml's impact disparity require treatment disparity? Advances in neural information processing systems , 31.

Marx, C., Calmon, F., and Ustun, B. (2020). Predictive multiplicity in classification. In International Conference on Machine Learning , pages 6765-6774. PMLR.

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. (2011). Scikit-learn: Machine learning in python. the Journal of machine Learning research , 12:2825-2830.

Pleiss, G., Raghavan, M., Wu, F., Kleinberg, J., and Weinberger, K. Q. (2017). On fairness and calibration. Advances in neural information processing systems , 30.

Roth, A., Tolbert, A., and Weinstein, S. (2022). Reconciling individual probability forecasts. arXiv preprint arXiv:2209.01687 .

Semenova, L., Rudin, C., and Parr, R. (2019). A study in rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning. arXiv preprint arXiv:1908.01755 .

Watson-Daniels, J., Parkes, D. C., and Ustun, B. (2022). Predictive multiplicity in probabilistic classification. arXiv preprint arXiv:2206.01131 .

Witten, I. H. and Frank, E. (2002). Data mining: practical machine learning tools and techniques with java implementations. Acm Sigmod Record , 31(1):76-77.

Appendix

The appendix is divided into the following five parts. Appendix A: proof of the orthogonality of OAE and SP from ambiguity (Proposition 3.1); Appendix B: proofs from Section 4; Appendix C: discussion on optimizing ensemble parameters; Appendix D: additional discussions on group fairness and fairness interventions; and Appendix E: additional experiments and details on the experimental setup.

A Proof of Proposition 3.1: Orthogonality of OAE and SP from Ambiguity

We divide the proof into two cases according to the group-fairness metric considered (OAE or SP).

A.1 Proof for the OAE Metric

Let D ⊂ R d × [ K ] ×{ 0 , 1 } be a dataset satisfying the two conditions listed in Remark 1, namely,

with n k denoting the number of samples in D belonging to group k ∈ [ K ] , the greatest common divisor of the n k is at least ( m -1) / ( mϵ -1) .

if ( x , s, y ) , ( x , s ′ , y ′ ) ∈ D share the same feature vector x , then y = y ′ too.

Consider the partition along group membership D = ⋃ k ∈ [ K ] D k , so we may write D k = { ( x k,i , k, y k,i ) } i ∈ [ n k ] for each k ∈ [ K ] , where n k ≜ |D k | . By assumption on the n k , we have

Now, let θ ≜ ⌊ ϵg ⌋ /g , and note that we obtain the integers µ k ≜ θ · n k for k ∈ [ K ] , because g divides each n k by definition of the gcd. Let λ ≜ ⌈ 1 -ϵ θ ⌉ . Using ⌊ t ⌋ > t -1 , we note that

where the last inequality follows by assumption on g being large enough (10). Thus, taking the ceiling of both sides above, we obtain λ +1 ≤ m . Hence, it suffices to prove that there are λ +1 classifiers R = { h j } j ∈ [ λ +1] satisfying the claim in the proposition (i.e., with λ +1 replacing m ).

It is straightforward to check that λµ k ≤ n k for each k . We divide each D k into λ sets of size µ k each, and collect the remainder into a separate subset. Thus, for each k ∈ [ K ] , consider any partition D k = ⋃ j ∈ [ λ +1] D j k where |D j k | = µ k for each j ∈ [ λ ] and |D λ +1 k | = n k -λµ k .

Next, we define the classifiers based on the partitions of the D k . For each j ∈ [ λ +1] , define the classifier h j over D as follows. Fix ( x , k, y ) ∈ D . We set the value h j ( x ) to be

Note that this makes h j well-defined by the second property assumed on D at the beginning of this proof. We show that the set R = { h j } j ∈ [ λ +1] satisfies the desired properties in the proposition.

Accuracy. First, we show that each h j incurs a 0-1 loss less than ϵ . Indeed, we have that, for each j ∈ [ λ ] ,

whereas the error for the last classifier is

Thus, it suffices to check that θ, 1 -λθ ≤ ϵ . Recall that we set θ = ⌊ ϵg ⌋ /g , hence θ ≤ ϵ is immediate. Further, as we take λ = ⌈ 1 -ϵ θ ⌉ ≥ 1 -ϵ θ , the inequality 1 -λθ ≤ ϵ follows immediately too. Hence, we have that ℓ ( h j ; D ) ≤ ϵ for all j ∈ [ λ +1] .

Fairness. Next, we check that each h j satisfies Overall Accuracy Equality (OAE) perfectly. Let ̂ Y j be the prediction of the classifier h j , so P ̂ Y j | X = x (1) = h j ( x ) . Then, for j ∈ [ λ ] the predictions of the j -th classifier satisfy

̸

and similarly the last classifier satisfies

̸

Hence, the h j , for j ∈ [ λ +1] , all satisfy OAE perfectly.

̸

Ambiguity. Finally, we show that the set R = { h j } j ∈ [ λ +1] exhibits full ambiguity. Note that we have a partition D = ⋃ ( k,j ) ∈ [ K ] × [ λ +1] D j k . Fix a sample ( x , s, y ) ∈ D and consider the index ( k, j ) ∈ [ K ] × [ λ + 1] of the unique part including it in the partition, i.e., ( x , s, y ) ∈ D j k . By construction of R , we have that h j ( x ) = 1 -y (so ̂ Y j = 1 -y ) but h j ′ ( x ) = y (so ̂ Y j ′ = y ) for any j ′ = j . In other words, for every fixed sample in D , there is a pair of classifiers in R assigning it conflicting predictions. Thus, the ambiguity is α ( D , R ) = 100% .

A.2 Proof for the SP Metric

Consider now the SP group-fairness constraint instead. In this case, we define we construct the following alternative dataset D ⊂ R d × [ K ] × { 0 , 1 } . Let t ≜ ⌈ 1 /ϵ ⌉ ≥ 2 . For each ( k, y ) ∈ [ K ] × { 0 , 1 } , fix any dataset D k,y = { ( x k,y,i , k, y ) } i ∈ [ t ] ⊂ R d × { k } × { y } , let D ≜ ⋃ ( k,y ) ∈ [ K ] ×{ 0 , 1 } D k,y ⊂ R d × [ K ] × { 0 , 1 } , and assume that if x k,y,i = x k ′ ,y ′ ,i ′ then y = y ′ . Denote n ≜ |D| = 2 Kt .

For each ( k, y ) ∈ [ K ] ×{ 0 , 1 } , let σ k,y be a permutation on [ t ] , i.e., { σ k,y ( j ) } j ∈ [ t ] = [ t ] . Define the set of classifiers R = { h j } j ∈ [ t ] on D as follows. For each j ∈ [ t ] and each ( k, y, i ) ∈ [ K ] ×{ 0 , 1 }× [ t ] , we set

We show that the set R = { h j } j ∈ [ t ] satisfies the desired properties in the proposition.

Accuracy. Consider first the 0-1 loss incurred by the h j . We have that, for each j ∈ [ t ] ,

Therefore, ℓ ( h j ; D ) = 1 /t = 1 / ⌈ 1 /ϵ ⌉ ≤ ϵ . In other words, R is a realization of the ϵ -level empirical Rashomon set of size t .

Fairness. Next, we check that each h j satisfies SP perfectly. In other words, we check that h j assigns to the class y = 1 the same percentage of samples across the groups k ∈ [ K ] . Indeed, this is true as we are switching the class memberships of exactly one sample from each of the D k,y . In particular, h j assigns the class membership 1 to exactly 1 + ( |D k, 1 |-1) = |D k, 1 | = t samples out of the total |D k, 0 ∪ D k, 1 | = 2 t samples belonging group k . As the ratio t/ (2 t ) = 1 / 2 is independent of the group k , we obtain the desired result that h j satisfies SP.

Ambiguity. Finally, we check that the set R = { h j } j ∈ [ t ] suffers from 100% ambiguity. Fix a sample ( x k,y,i , k, y ) ∈ D , and we will show that there are two classifiers in R assigning conflicting predictions to it. Indeed, let j = σ -1 k,y ( i ) and let j ′ ∈ [ t ] \ { j } be any other index. Then, h j ( x k,y,i ) = 1 -y , whereas h j ′ ( x k,y,i ) = y . Therefore, the sample ( x k,y,i , k, y ) contributes to the overall ambiguity α ( D , R ) . As this is true for all samples in D , we conclude that α ( D , R ) = 100% , as desired. This completes the proof of the proposition.

B Proofs of Section 4

B.1 Proof of Theorem 4.1

We assume that ∥ λ ∥ 2 2 , ∥ γ ∥ 2 2 ≤ c/m for an absolute constant c , e.g., we have c = 1 for the uniform ensembling λ = (1 /m, · · · , 1 /m ) as then ∥ λ ∥ 2 2 = 1 /m . Fix x , and denote the mean of the classifiers µ x = E h ∼T ( D ) [ h ( x )] . The mapping ( h 1 , · · · , h m ) ↦→ h ens , λ satisfies the bounded-difference condition in the McDiarmid inequality. Indeed, changing h i can change h ens , λ by at most λ i . Furthermore, h ens , λ ( x ) has the mean

Hence, by Mcdiarmid's inequality, we have the bound

The same inequality holds for γ in place of λ :

Therefore, we obtain the bound

where the first inequality comes from triangle inequality, the following from probability of subset of events ( P ( A ) ≥ P ( B ) if A ⊇ B ), the equality from taking complement, and the last line from applying McDiarmid's inequality along with the union bound.

Finally, applying the union bound on D valid. with |D valid. | = n , we obtain the bound

and the proof is complete.

B.2 Proof of Theorem 4.2

The main idea is as follows: first observe that for the ensembled labels to disagree on a sample (given that the scores are bounded away from 1 2 with high probability), the two models need to produce scores in the range [0 , 1 2 -δ ] ∪ [ 1 2 + δ, 1] . This means that the scores need to deviate at least 2 δ which has an exponentially low probability given Theorem 4.1.

We will show that

̸

Indeed, for each fixed x ∈ D 0 , we may reduce the failure probability to the case of separation of scores:

̸

Finally, applying the union bound, we obtain that

̸

and the proof is complete.

C Discussion on optimizing ensemble parameters

We have taken the weights λ ∈ ∆ m which determines the ensembled model h ens , λ to be fixed. We explain here how λ can be optimized according to a given cost. Specifically, given a loss function ℓ : [0 , 1] ×{ 0 , 1 } → R + , we can search for the optimal λ ∈ ∆ m that minimizes the total cost

For the above optimization problem, we think of the constituent models h 1 , · · · , h m as being fixed and the randomness is from that of ( X,Y ) .

However, in practice, we have access to only samples ( x i , y i ) ∼ P X ,Y . Thus, we consider minimizing the regularized sample mean (for fixed β > 0 )

The 2 -norm regularization is added to facilitate proving convergence. This convergence result can be obtained via known results from statistical learning theory, e.g., using Theorem 13.2 in [22]. Specifically, consider the following two restrictions:

Consider only λ ∈ ∆ m satisfying ∥ λ ∥ 2 ≤ α for a prescribed α . Note that we may take α = 1 to encapsulate the whole probability simplex. However, we may choose λ to be a slight modification of the uniform ensembling, in which case we would have α of order 1 / √ m .

Assume that the function λ ↦→ ℓ ( h ens , λ ( x ) , y ) is convex and A -Lipschitz for each fixed ( x , y ) .

In this case, choosing β = A/α and denoting the optimizers

we can bound the utility of these minimizers by

for any δ ∈ (0 , 1) .

D Additional discussion on group fairness and fairness interventions

In addition to OAE (Definition 5), two other important fairness criteria are Statistical Parity [18] and Equalized Odds [23].

Definition 10 (EO) . Pr( ̂ Y = 1 | S = s, Y = b ) = Pr( ̂ Y = 1 | S = s ′ , Y = y ) for all groups s, s ′ ∈ [ K ] , and binary labels y ∈ { 0 , 1 } .

Essentially, SP requires the predicted label ̂ Y ≜ arg max h ( X ) to be independent of the group membership S [19]. In comparison, EO conditions on both group and the true label [23]. EO improves upon SP in the sense that it does not rule out the perfect classifiers whenever the true label Y is correlated with the group membership S [1]. In practice, we quantify EO violation by measuring Mean EO as in Equation 6 (for two groups) and, more generally, in Equation 42 below (beyond two groups). Similarly, we can measure SP violation as in Equation 43.

Next, we offer a brief discussion of various intervention mechanisms used in this paper. The fairness interventions can be categorized into two categories: in-processing and post-processing. In-processing mechanisms incorporate fairness constraints during training. It usually add the fairness constraint to the loss function and outputs a fair classifier. Post-processing mechanisms treat the model as a black box and update its predictions to achieve the desirable fairness constraints [10].

REDUCTION [1], short for exponentiated gradient reduction, is an in-processing technique that reduces fair classification to a sequence of cost-sensitive classification problems, and yields a randomized classifier with the lowest empirical error subject to the desired constraints. This technique achieves fairness with a minimal decrease in accuracy, but it is computationally expensive since it requires re-training multiple models.

REJECT OPTION CLASSIFIER [29] is a postprocessing technique that achieves fairness constraints by modifying outcomes of samples in a confidence band of the decision boundary with the highest uncertainty. It gives favorable outcomes to unprivileged groups and unfavorable outcomes to privileged groups. It outputs a thresholded prediction rather than a probability over the binary labels.

EQODDS [23] is a post-processing technique that formulates empirical risk minimization with fairness constraint as a linear program and modifies predictions according to the derived probabilities to achieve equalized odds.

FAIR PROJECTION [2] is a post-processing technique that can accommodate fairness constraints in a setting with multiple labels and multiple groups. The fair model is obtained from 'projecting' a pre-trained (and potentially unfair) classifier onto the set of models that satisfy target group-fairness requirements.

E Additional experiments and details on the experimental setup

Our proposed methodology can be summarized in the pipeline in Figure 6.

Data

Fairness,

Predictive

Multiplicity

Ensembled

Fair Models

Competing

Baseline

Models

Competing

Fair Models

vary

random seeds

apply fairness

interventions

apply optimal

ensemble

Fairness,

Predictive

Multiplicity

Test Data

Training Data

Fairness,

Predictive

Multiplicity

E.1 Data

The HSLS dataset [27, 28] is an education dataset collected from 23,000+ students across high schools in the USA. Features of the dataset contain extensive information on students' demographic information, their parents' income and education level, schools' information, and students' academic performances across years. We apply the pre-processing techniques adopted by Alghamdi et al. [2], with the number of samples reduced to 14,509. For the binary classification task with fairness constraints, the group attribute chosen is RACE ∈ { WHITE , NON-WHITE } and the prediction label is students 9th-Grade GRADEBIN ∈ { 0 , 1 } , binarized according to whether a student's grade is higher or lower than the median.

The ENEM dataset [16] is a Brazilian high school national exam dataset introduced by Alghamdi et al. [2]. It has 138 features containing students' demographic information, socio-economic questionnaire answers (e.g., parents' education level and if they own a computer), and students' exam scores. Adopting the preprocessing technique in Alghamdi et al. [2], we sample 50K samples without replacement from the processed ENEM Year 2020 data. Identical to HSLS, the group attribute chosen is RACE ∈ { WHITE , NON-WHITE } and the prediction label is students Grade binarized into GRADEBIN ∈ { 0 , 1 } according to whether a student's grade is higher or lower than the median.

For the widely known Adult dataset [33], also known as "Census Income" dataset, we choose the group attribute as SEX ∈ { MALE , FEMALE } and predicted label to be INCOME ∈ { 0 , 1 } , where income bin denotes whether a person's income is higher or lower than 50K/yr.

E.2 Competing Baseline Models

We use the Scikit-learn implementation of logistic regression, gradient boosting, and random forest as baseline models. For logistic regression and gradient boosting, the default hyperparameter is used; for random forest, we set the number of trees and minimum number of samples per leaf to 10 to prevent over-fitting. To get 10 competing models for each hypothesis class, we use 10 random seeds (specifically 33-42).

In practice, the competing models, i.e., h ∈ ̂ R m can be obtained using different methodologies, such as sampling and adversarial weight perturbation [26, 40]. We suggest one method for sampling. First, split the data into training, validation, and test dataset. We train a set of models by changing the randomized procedures in the training process, e.g., using different initializations, different cuts for cross-validation, data shuffling, etc. In this paper, we change the random seed feed into the baseline models to obtain competing models. We use the validation set to measure ϵ corresponding to this empirical Rashomon Set.

E.3 Competing Fair Models

For EQODDS, REJECTION, and REDUCTION, we use the functions EQODDSPOSTPROCESSING, REJECTOPTIONCLASSIFICATION, and EXPONENTIATEDGRADIENTREDUCTION from AIF360 toolkits [4]. For LEVERAGING and FAIR PROJECTION, we use the codes provided in the corresponding Github repositories of Chzhen et al. [12] and Alghamdi et al. [2].

Remark 3. We observe in practice that fairness classifiers are more confident and have scores that are thresholded-like (Figure 7 Left ). From the similarity in the shape of the thresholded baseline curve and the fair models' curves (Figure 7 Right ), thresholding-like behavior of some interventions may explain some-but certainly not all (see Figure 1 Right )-increase in score std dev and the ensuing arbitrariness. Recall from the violin plot in Figure 1 that the largest group (blue area) are

BaselineScores

1.0

Bins

0.4

Post-InterventionScores

0.8

Samples inE

0.3

0.6

Reduction High Fair(Acc:0.654,Mean EO:0.04)

DensityofS

0.2

0.4

ReductionLowFair(Acc:0.661,MeanEO:0.15)

Rejection HighFair(Acc:0.652,Mean EO:0.03)

0.1

0.2

Rejection LowFair(Acc:0.663,Mean EO:0.26)

Baseline(Acc:0.663,MeanEO:0.28)

Thresholded Baseline(Acc:0.663,MeanEO:0.28)

0.0

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.1

0.2

0.3

0.4

0.5

ScoreStandard Deviation

those individuals for which std. increases from 0 to a large positive value (median around > 0.15). Hence, the blue area shows that significant arbitrariness is introduced by the fairness intervention, in addition to and separate from the effects of thresholding the baseline.

E.4 Complete Experimental Plots

In order to evaluate the predictive multiplicity of models with similar accuracy and fairness levels, we divide the accuracy-fairness frontier plots into 8x8 grids and put models in the corresponding bins. To compare the arbitrariness of models satisfying high/low fairness constraints, we select bins in two different MEO ranges and bins with baseline models. Then, we compute the standard deviation of scores of models corrected by REDUCTION in the three bins (high fairness/low fairness/baseline) and plot the quantile curves. We use PANDAS package's quantile function with its default linear interpolation method.

Across three baseline model classes (random forest, gradient boosting, and logistic regression), fair models exhibit higher score arbitrariness. Especially at top quantiles, all fair models have standard deviations in scores going up to 0.5. This means that for the individuals at the top percentile, the model prediction can flip if another random seed is used in model training.

Furthermore, we evaluate the predictive multiplicity of models corrected by different fairness intervention methods. Across datasets, all fairness intervention methods exhibit maximal standard deviation of scores of 0.5 at top quantiles for random forest baseline methods. LEVERAGING [12] exhibit score arbitrariness comparable to that of baseline for GBM and Logistic Regression methods. REJECTION and LEVERAGING output thresholded scores directly, while REDUCTION outputs probabilities (with most scores close to 0 or 1).

(Random Forest)

(GBM)

(Logistic Regression)

HSLS

(Random Forest)

HSLS

(GBM)

HSLS

(Logistic Regression)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the interaction of predictive multiplicity with fairness interventions, making the observation that models which receive common fairness interventions often result in increased predictive multiplicity within the Rashomon set. They back this up with some theoretical exploration and experiments, proposing ensembling as an approach for reducing multiplicity.\n\nSTRENGTHS:\n- the observation that models under fairness interventions have higher predictive multiplicity is a novel and interesting one\n- the paper is well-presented and well-scoped, making a clear and useful contribution\n- Figure 3 is a nice intuitive explanation for why the observed phenomenon occurs\n- ensembles are a reasonable mitigation for this problem\n\nWEAKNESSES:\n- I'm a little confused by the discussion around \"confidence\" in 4.2. It would be useful to give more explanation of how confident (or unconfident) classifiers improve the multiplicity problem, and if this matters before or after ensembling. I also cannot seem to find the referred discussion around fairness and confidence in the Appendix - it sounds like an interesting observation and would be interesting to highlight it a bit more.\n- some questions around the formalisms in Sec 2: 1. I think T(D) should be a distribution rather than a set, 2. the writing of equation 1 is a little confusing - it's not quite clear how both m and epsilon can be parameters (what if there aren't m models of loss less than epsilon? is it the m lowest loss or any m?) 3. is the domain of \\ell above Eq 1 [0, 1] x {0, 1}?\n- in Prop 3.1, is there some assumption made around what loss function is used?\n\nQUESTIONS:\n- is reducing predictive multiplicity ultimately a good thing? the upside is that the output is more consistent, but the downside is that the more consistent output is not necessarily chosen in a principled manner. I'd be interested to hear more discussion of this tradeoff\n- I'm curious about why the Leveraging approach improves multiplicity so much - this would be good to explore a bit if you have space or at least discuss\n- do we know that ensembles will satisfy fairness metrics if their component models do?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors discuss arbitrariness in automated decision-making, i.e. the fact that similarly trained classifiers may end up disagreeing on the most suitable label for a data point. In particular, the authors observe that arbitrariness can be worsened by fairness interventions, suggesting that increased group fairness may mean that some individuals arbitrarily receive worse decisions. A solution is proposed in the form of an ensemble algorithm that takes many (arbitrarily) fair classifiers and takes a majority vote.\n\nSTRENGTHS:\nThe authors visit the active yet understudied topic of arbitrariness in a seemingly novel way. Not only may fairness metrics be oblivious to arbitrariness, their optimization may even worsen it. This nuance meaningfully adds to the debate.\n\nThe arguments are presented in a simple manner, and the theoretical results are only as complicated as they need to be.\n\nThe paper is clearly thought-provoking and it invites discussion at a level fit for NeurIPS.\n\nWEAKNESSES:\n1. My biggest concern for the paper is a philosophical one: why is arbitrariness (under your definition) harmful? \n    * In Example 1, this is not very clear: in the 'fair' column, each group receives mistakes at the same rates. Why is arbitrariness problematic here? \n    * I have a similar concern with Example 2: it is elegantly shown that two classifiers are each equally EO-fair yet use a different threshold (leading to arbitrariness). Why should individuals care which classifier is chosen?\n    * Perhaps the argument could be made that arbitrariness is a form of unfairness, and that certain fairness notions are oblivious to it. Yet also in this case I wonder whether there is not always a fairness notion that *does* appropriately consider arbitrariness. For example in Figure 2, the arbitrariness could be detected through the Equalised Opportunity fairness notion (i.e. equal false negative rates). In that case, model 2 would be considered fair and model 1 would not be.\n    * Please note I have not read up on the socio-technical arguments for studying arbitrariness. Nevertheless, I believe the examples in this paper should clarify why arbitrariness is important.\n\n2. Figure 4 presents a worrying result. Should we not be able to see the orange curve (low fairness) trend towards the green curve (baseline)? To their credit, the authors remark this as a remarkable result, but I wonder whether the authors can discuss why we don't see a more gradual shape evolution from green to orange as the fairness intervention is done more and more intensively. If not, then it raises the question if the baseline curve is comparable to the others.\n\n3. Many parts of the presentation are unpolished. Here are some examples.\n    * Equations 1, 2 and 3 all use different symbols for the score / classifier function. Equation 2 in particular is inconsistent, as it indexes the data samples with $i$, and it selects models as elements from a set whereas other equations use the symbol $i$.\n    * It seems $\\mu$ in Eq. 3 should also have a subscript $x$.\n    * Definition 3 presents the random variable $\\hat{Y}$ whereas $h(x)$ was used before.\n    * Section 4 rather suddenly makes the jump from binary classification to multiclass classification.\n    * Figure 3 unnecessarily has a different font and layout. Its x-axis is unlabeled on the right, even though it seems different from the x-axis on the left. The colors in Figure 5 should also be more consistent (as the same methods are used on both sides).\n\n4. Something appears to be wrong with the LaTeX settings. There are no line numbers, and the \"Anonymous Authors\" block is hidden. If this was done to save space, then this gives an unfair advantage compared to other authors that do follow the template correctly.\n\nQUESTIONS:\nPlease find my questions and suggestions in the Weaknesses box.\n\nIf the authors have limited time during the rebuttal, then please consider only answering my questions related to the first weakness.\n\nLIMITATIONS:\nThe (negative) societal impact of the work was not discussed. The authors may want to consider whether ensemble classifiers are a reasonable solution to arbitrariness in practice. Fairness in general has many societal risks if implemented incorrectly.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
The authors theoretically and empirically study predictive multiplicity as it relates to fairness and accuracy. They show in both ways that multiplicity increases as a result of bias mitigation.

STRENGTHS:
This is a problem that has not been studied and was worth studying because practitioners don't think about it, and should start doing so.

Both the theory and the experiments are compelling.

The paper is well written.

The math looks correct.

Bravo.

WEAKNESSES:
The authors should justify why arbitrariness is so bad under limited resources that can be allocated. If two people are exactly the same and right at the decision boundary, what other than an arbitrary decision is just if there is only one spot? Similarly, the authors should mention that several bias mitigation post-processing algorithms including [23] have an arbitrariness nature to them as part of the way they work. 

It would be nice if the authors could connect their concentration results more to the original work in (non-fairness) ensemble classifiers, including Breiman's original paper on Random Forests (https://link.springer.com/article/10.1023/a:1010933404324) and the parameters used therein: strength and correlation. There may be simple ways of doing so by extending the operating points of  https://doi.org/10.1109/TKDE.2012.219 to also consider fairness metrics.

QUESTIONS:
The paper is pretty clear.

LIMITATIONS:
Nothing specific noted.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the effect of group fairness regularization constraints on a notion termed arbitrariness. Here, arbitrariness is defined to be the variability of the predictions, for a set of similarly accurate models, on individual samples. In this paper, they show that when a model is regularized to satisfy group fairness constraints, it is still possible that the predictions of similarly accurate models show variability in the individual predictions. The set of similarly accurate models on a given subpopulation of data is termed the rashomon set. Here, the authors measure arbitrariness with the quantile of the standard deviation of the output of the models in a rashomon set. They also measure ambiguity by computing the proportion of points in the dataset that can be assigned multiple conflicting predictions. They prove that it is possible for a model to satisfy accuracy equality (i.e. all protected subgroups have similar accuracy), but for the arbitrariness of the rashomon set to be as high as up to 100 percent. To counter this challenge, they propose a simple scheme that ensembles the models in the rashomon set (convex combination), and show theoretically and empirically (on three datasets for random forest models) that this ensembling scheme reduces arbitrariness.\n\nSTRENGTHS:\n**Originality**\\\nThis paper follows in the line of work on predictive multiplicity and underspecification. Overall, this paper explores the interplay between the predictive multiplicity of a rashomon set and and group fairness properties of that set. A key insight in this work that I hadn't seen before is that it is possible for group fairness constraints to increase the predictive multiplicity of the rashomon set. This finding seems counter-intuitive. The ensembling approach discussed in this work has been mentioned in past work, like the underspecification paper by D'amour et. al., however, it was not explicitly used to address predictive multiplicity as it is used here. Overall, this paper furthers the discussion in a particularly interesting way.\n\n**Quality/Clarity**\\\nThe paper is well-written and clear. Figure 2 was quite helpful for trying to understand the main message of the paper. Overall, this is a high quality paper.\n\n**Significance**\\\nIf it remains the case that arbitrariness is orthogonal to group fairness constraints, then that is an important finding and has far reaching impact on how to obtain models with reliable predictions. The paper has opened up a line of work that can be explored in several ways.\n\nWEAKNESSES:\nNone of the weaknesses I discuss here are disqualifying, but I note below places where the paper can be improved.\n\n**Unify the terms Predictive multiplicity, Arbitrariness, and Ambiguity**: Right now, I think these three terms are all referring to the same general phenomenon, but it is unclear whether there is an instantiation of each that is different. Perhaps the authors could unify this.\n\n**Proposition 3.1 is only for accuracy parity/equality**: Unclear if this proposition generalizes to other metrics (see questions section).\n\n**Restricted model class and dataset**: This paper only explores random forest models on tabular datasets.\n\nQUESTIONS:\n- Is equality of accuracy interchangeable with any other group fairness constraint in proposition 3.1? I know Example 2 discusses equality of opportunity, but the main argument for why group fairness constraints and predictive multiplicity are perhaps orthogonal is this proposition, but in reading the proof, it is not clear to me that you can extend this to other fairness metrics. Accuracy parity makes sense here, but I don't think something like group calibration fits the theme. Can the authors comment on this? \n\n\n- How does one pick the number of models in a rashomon set? In practice, in the experiments, it seems like you need to set a number for the size of the rashomon set, and this parameter might have an oversized influence on the arbitrariness and other properties of the set. How was this number determined?\n\nLIMITATIONS:\nDiscussed in the final section of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-19"}