SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Individual Arbitriness and Group Fairness
Carol Xuan Long $^{\ddagger}$ , Hsiang Hsu $^{*\dagger}$ , Wael Alghamdi $^{* \ddagger}$ , Flavio P. Calmon $^{\ddagger}$
Machine learning tasks may admit multiple competing models that achieve similar performance yet produce arbitrary outputs for individual samples—a phenomenon known as predictive multiplicity. We demonstrate that fairness interventions in machine learning optimized solely for group fairness and accuracy can exacerbate predictive multiplicity. Consequently, state-of-the-art fairness interventions can mask high predictive multiplicity behind favorable group fairness and accuracy metrics. We argue that a third axis of "arbitrariness" should be considered when deploying models to aid decision-making in applications of individual-level impact. To address this challenge, we propose an ensemble algorithm applicable to any fairness intervention that provably ensures more consistent predictions.
Non-arbitrariness is an important facet of non-discriminatory decision-making. Substantial arbitrariness exists in the training and selection of machine learning (ML) models. By simply varying hyperparameters of the training process (e.g., random seeds in model training), we can produce models with arbitrary outputs on individual input samples [9, 13, 26, 35]. The phenomenon where distinct models exhibit similar accuracy but arbitrary individual predictions is called predictive multiplicity<sup>1</sup> [35]. The arbitrary variation of outputs due to unjustified choices made during training can disparately impact individual samples, i.e., predictive multiplicity is not equally distributed across inputs of a model. When deployed in high-stakes domains (e.g., medicine, education, resume screening), the arbitrariness in the ML pipeline may target and cause systemic harm to specific individuals by excluding them from favorable outcomes [7, 15, 40].
Popular fairness metrics in the ML literature do not explicitly capture non-arbitrariness. A widely recognized notion of non-discrimination is group fairness. Group fairness is quantified in terms of, for example, statistical parity [18], equal opportunity, equalized odds [23], and variations such as multi-accuracy [30] and multi-calibration [24]. Broadly speaking, methods that control for group fairness aim to guarantee comparable performance of a model across population groups in the data. The pursuit of group fairness has led to hundreds of fairness interventions that seek to control for performance disparities while preserving accuracy [25].
The central question we tackle in this paper is: Do models corrected for group fairness exhibit less arbitrariness in their outputs? We answer this question in the negative. We demonstrate that state-of-the-art fairness interventions may improve group fairness metrics at the expense of exacerbating arbitrariness. The harm is silent: the increase in arbitrariness is masked by favorable group fairness and accuracy metrics. Our results show that arbitrariness lies beyond the fairness-accuracy frontier:
* equal contributions.
Prepared prior to employment at JPMorgan Chase & Co.. Email: hsiang.hsu@jpmchase.com.
$^{\ddagger}$ John A. Paulson School of Engineering and Applied Sciences, Harvard University, Boston, MA 02134. Emails: carol_long@g.harvard.edu, alghamdi@g.harvard.edu, flavio@seas.harvard.edu.
<sup>1</sup>In the rest of the paper, we informally use the terms "arbitrariness" and "predictive multiplicity" exchangeably to refer to the phenomenon of inconsistent predictions caused by the randomness in model training.
Figure 1: Accuracy-fairness frontier does not reveal arbitrariness in competing models. Left: Fairness-Accuracy frontier of baseline and fair models corrected by 5 fairness interventions; point clouds generated by different random seed choices. Middle: The cumulative distribution functions (CDF) of per-sample score std. across classifiers at different intervention levels (see Definition 3). For each sample, std. is measured across competing scores produced by classifiers initialized with different random seeds. A wider CDF indicates more disparity of the impact of arbitrariness on different individuals. Right: The distribution of score std. relative to the thresholded baseline model. Removing samples that receive very low score std. both from thresholded baseline and fair classifiers, the largest group (blue area) in this violin plot are those individuals for which std. increases from 0 to a large positive value (median around 0.15). Hence, significant arbitrariness is introduced by the fairness intervention, in addition to and separate from the effects of thresholding the baseline.
predictive multiplicity should be accounted for in addition to usual group-fairness and accuracy metrics during model development.
Figure 1 illustrates how fairness interventions can increase predictive multiplicity. Here, state-of-the-art fairness interventions are applied<sup>4</sup> to a baseline random forest classifier to ensure group fairness (mean equalized odds [23], see Definition 6) in a student performance binary prediction task. We produce multiple baseline classifiers by varying the random seed used to initialize the training algorithm. Each baseline classifier achieves comparable accuracy and fairness violation. They also mostly agree in their predictions: for each input sample, the standard deviation of output scores across classifiers is small (see Definition 3). After applying a fairness intervention to each randomly initialized baseline classifier, we consistently reduce group fairness violations at a small accuracy cost, as expected. However, predictive multiplicity changes significantly post-intervention: for roughly half of the students, predictions are consistent across seeds, whereas for  $20\%$  of the students, predictions are comparable to a coin flip. For the latter group, the classifier output depends on the choice of a random seed instead of any specific input feature. The increase in predictive multiplicity is masked by the fairness-accuracy curve, does not impact all samples equally, and is consistent across datasets and learning tasks.
At first, the increase in predictive multiplicity may seem counter-intuitive: adding fairness constraints to a learning task should reduce the solution space, leading to less disagreement across similarly-performing classifiers relative to an unconstrained baseline. We demonstrate that, in general, this is not the case. For a given hypothesis class, the non-convex nature of group fairness constraints can in fact increase the number of feasible classifiers at a given fairness and accuracy level. We show that this phenomenon occurs even in the simple case where the hypothesis space is comprised of threshold classifiers over one-dimensional input features, and the optimal baseline classifier is unique. To address this challenge, we demonstrate – both theoretically and through experiments – that ensembling classifiers is an effective strategy to counteract this multiplicity increase.
1. We demonstrate that the usual "fairness-accuracy" curves can systematically mask an increase of predictive multiplicity. Notably, applying state-of-the-art fairness interventions can incur higher arbitrariness in the ML pipeline.
2. We show that multiplicity can be arbitrarily high even if group fairness and accuracy are controlled, when we do not have perfect classifiers. Hence, fairness interventions optimized solely for fairness and accuracy cannot, in general, control predictive multiplicity. We also provide examples of why fairness constraints may exacerbate arbitrariness.
See Section 5 for a detailed description of the experiment and dataset.
$^{5}$ Proofs and additional experiments are included in the Appendices.
3. We propose an ensemble algorithm that reduces multiplicity while maintaining fairness and accuracy. We derive convergence rate results to show that the probability of models disagreeing drops exponentially as more models are added to the ensemble.
4. We demonstrate the multiplicity phenomena and benchmark our ensemble method through comprehensive experiments using state-of-the-art fairness interventions across real-world datasets.
Multiplicity, its implications, and promises. Recent works have investigated various factors that give rise to multiplicity. D'Amour et al. [17] studied how under-specification presents challenges to the credibility of modern machine learning algorithms. More precisely, under-specified optimization problems in machine learning admit a plethora of models that all attain similar performance, and which model to deploy in practice may ultimately depend on arbitrary choices of the randomization made during training procedure [3]. The arbitrariness of the model could potentially harm the reproducibility of model predictions [5], and hence the credibility of the conclusion made thereof.
Creel and Hellman [15] thoroughly explored the notion of arbitrariness in machine learning and discuss how high-multiplicity predictions can lead to systematized discrimination in society through "algorithmic leviathans." Multiplicity in prediction and classification can also have beneficial effects. Black et al. [7], Semenova et al. [39], and Fisher et al. [20] view multiplicity of equally-performing models as an opportunity to optimize for additional criteria such as generalizability, interpretability, and fairness. Coston et al. [14] develop a framework to search over the models in the Rashomon set for a better operation point on the accuracy-fairness frontier. However, they do not discuss the potential predictive multiplicity cost of existing fairness interventions nor propose algorithms to reduce this cost.
The work most similar to ours is [13]. Cooper et al. [13] consider the problem of predictive multiplicity as a result of using different splits of the training data. Therein, they quantify predictive multiplicity by prediction variance, and they propose a bagging strategy [8] to combine models. Our work considers a different problem where predictive multiplicity is exacerbated by group-fairness interventions. Our work is also different from Cooper et al. [13] as we fix the dataset when training models and consider multiplicity due to randomness used during training. In this sense, our ensemble algorithm is actually a voting ensemble [41] (see Section 4); see also ensembling and reconciliation strategies proposed by Black et al. [6] and Roth et al. [38] that aim to create more consistent predictions among competing models. To the best of the authors' knowledge, we are the first to measure and report the arbitrariness cost of fairness interventions.
Hidden costs of randomized algorithms. Recent works [21, 31, 32] examine the potential detrimental consequences of randomization in the ML pipeline. In their empirical study, Ganesh et al.[21] observe that group fairness metrics exhibit high variance across models at different training epochs of Stochastic Gradient Descent (SGD). The authors point out that random data reshuffling in SGD makes empirical evaluation of fairness (on a test set) unreliable, and they attribute this phenomenon to the volatility of predictions in minority groups. Importantly, they do not incorporate fairness interventions in their experiments. In contrast, we apply fairness interventions to baseline models. Specifically, we examine the variance in predictions among models with similar fairness and accuracy performances. In addition to the observations made by Ganesh et al., our theoretically-grounded study reveals the different paths that lead to group-fairness, i.e., that arbitrariness can be an unwanted byproduct of imposing fairness constraints. Krco et al. [31] empirically study if fairness interventions reduce bias equally across groups, and examine whether affected groups overlap across different fairness interventions. In contrast, our work examines the multiplicity cost of group fairness and its tension with individual-level prediction consistency, rather than the fairness cost of randomness in the ML pipeline. Another work on the hidden cost of randomized algorithms is given by Kulynych et al. [32], who report that well-known differentially-private training mechanisms can exacerbate predictive multiplicity.
In an early work [34], Lipton et al. indirectly points to the potential arbitrary decision on individuals as a result of imposing group fairness constraints. They give an illustrative example using synthetic hiring data to show that a fair model resorts to using irrelevant attribute (hair length) to make hiring decision in order to achieve near-equal hiring rate for men and women.
2 Problem Formulation and Related Works
We explain the setup and relevant definitions in this section.
Prediction tasks. We consider a binary classification setting with training examples being triplets  $(X, S, Y)$  with joint distribution  $P_{\mathbf{X}, S, Y}$ . Here,  $\mathbf{X}$  is an  $\mathbb{R}^d$ -valued feature vector,  $S$  is a discrete random variable supported on  $[K] \triangleq \{1, \dots, K\}$  representing  $K$  (potentially overlapping) group memberships, and  $Y$  is a binary (i.e.,  $\{0, 1\}$ -valued) random variable denoting class membership. We consider probabilistic classifiers in a hypothesis space  $\mathcal{H}$ , where each  $h \in \mathcal{H}$  is a mapping  $h: \mathbb{R}^d \to [0, 1]$ . Each value of a classifier  $h(\mathbf{x})$  aims to approximate  $P_{Y|\mathbf{X} = \mathbf{x}}(1)$ . The predicted labels  $\widehat{y}$  can be obtained by thresholding the scores, e.g.,  $\widehat{y} = \mathbb{1}\{h(\mathbf{x}) \geq 0.5\}$ , where  $\mathbb{1}\{\cdot\}$  is the indicator function. Finally, we denote by  $\Delta_c$  the probability simplex over  $c$  dimensions.
Randomized training procedures and the Rashomon set. We assume access to the following:
1. a training dataset of  $n$  i.i.d samples  $\mathcal{D} \triangleq \{(x_i, s_i, y_i)\}_{i=1}^n$  drawn from  $P_{X,S,Y}$ ;
2. a randomized training procedure  $\mathcal{T}$ ; and
3. an induced distribution  $\mathcal{T}(\mathcal{D})$  on the hypothesis class of predictors  $\mathcal{H}$
We denote a sampled classifier by  $h \sim \mathcal{T}(\mathcal{D})$ , which can be sampled, for example, using different random seeds at the beginning of the execution of procedure  $\mathcal{T}$  on  $\mathcal{D}$ . For concreteness, the above data may for example correspond to the following practical setting.
Example 1. The dataset  $\mathcal{D}$  can be comprised of resumes of individuals applying for a job, and the training procedure  $\mathcal{T}$  is an algorithm to predict whether to extend an interview opportunity for an applicant. For example,  $\mathcal{T}$  can be a neural network with unspecified hyperparameters (e.g., random seed that needs to be chosen at the outset); alternatively,  $\mathcal{T}$  can be the same pre-trained neural network composed with a fairness intervention method. The classifiers considered will be the last layer of the neural network (or the classifier after fairness enhancement), which will belong to a hypothesis class  $\mathcal{H}$  determined by the chosen neural network architecture. By varying the random seed, say,  $m$  times, we would obtain independent classifiers, denoted by  $h_1,\dots ,h_m\stackrel {i.i.d.}{\sim}\mathcal{T}(\mathcal{D})$
We are interested in detecting whether competing classifiers (i.e., deployed for, and performing similarly in the same prediction task) have conflicting predictions non-uniformly across individuals. Next, we define the set of competing models obtained from the randomized training procedure  $\mathcal{T}$ .
For a loss function  $\ell : [0,1] \times \{0,1\} \to \mathbb{R}^+$ , finite dataset  $\mathcal{D} \subset \mathbb{R}^d \times [K] \times \{0,1\}$ , and classifier  $h: \mathbb{R}^d \to [0,1]$ , we let the empirical loss incurred by  $h$  on  $\mathcal{D}$  be denoted by  $\ell(h; \mathcal{D}) \triangleq |\mathcal{D}|^{-1} \sum_{(\boldsymbol{x}, s, y) \in \mathcal{D}} \ell(h(\boldsymbol{x}), y)$ . The (empirical) Rashomon set ( $\epsilon$ -level set) of competing models is defined as the set of models with loss lower than  $\epsilon$  [26], i.e.,  $\mathcal{R}(\mathcal{H}, \mathcal{D}, \epsilon) \triangleq \{h \in \mathcal{H} : \ell(h; \mathcal{D}) \leq \epsilon\}$ . We extend the definition of the Rashomon set to take into consideration the effect of the randomized algorithm  $\mathcal{T}$ , as follows.
Definition 1 (Empirical Rashomon Set of Randomized Training Procedure). Fix a finite dataset  $\mathcal{D}$ , a hypothesis class  $\mathcal{H}$ , and a randomized training procedure  $\mathcal{T}$  inducing the distribution  $\mathcal{T}(\mathcal{D})$  on  $\mathcal{H}$ . Given a loss function  $\ell :[0,1]\times \{0,1\} \to \mathbb{R}^+$  and a parameter  $\epsilon >0$ , we define the empirical Rashomon set with  $m$  models induced by  $\mathcal{T}$  as the collection of  $m$  classifiers independently sampled from  $\mathcal{T}(\mathcal{D})$  and having empirical loss less than  $\epsilon$ :
$$
\widehat {\mathcal {R}} _ {m} (\mathcal {T}, \mathcal {D}, \epsilon) \triangleq \left\{h _ {1}, \dots , h _ {m} \in \mathcal {H}: h _ {1}, \dots , h _ {m} \stackrel {i. i. d.} {\sim} \mathcal {T} (\mathcal {D}) \text {a n d} \ell \left(h _ {j}; \mathcal {D}\right) \leq \epsilon \forall j \in [ m ] \right\}. \tag {1}
$$
Here,  $\epsilon$  is an approximation parameter that determines the size of the set. The set  $\widehat{\mathcal{R}}_m(\mathcal{T},\mathcal{D},\epsilon)$  can be viewed as an approximation of the Rashomon set of "good" models [9, 35, 40], and indeed we have the inclusion  $\widehat{\mathcal{R}}_m(\mathcal{T},\mathcal{D},\epsilon)\subset \mathcal{R}(\mathcal{H},\mathcal{D},\epsilon)$  where  $\mathcal{H} = \mathrm{supp}(\mathcal{T}(\mathcal{D}))$ . Note that the set  $\widehat{\mathcal{R}}_m(\mathcal{T},\mathcal{D},\epsilon)$  is itself random even for a fixed dataset  $\mathcal{D}$ , where the source of randomness is coming from the distribution  $\mathcal{T}(\mathcal{D})$ . In the sequel, we omit the arguments of  $\widehat{\mathcal{R}}_m(\mathcal{T},\mathcal{D},\epsilon)$  when they are clearly implied from context.
<sup>6</sup>We note that our setup can be readily extended to multi-class prediction.
There are various metrics to quantify predictive multiplicity across models in  $\widehat{\mathcal{R}}_m$  by either considering their output scores [40] or thresholded predictions [35]. e focus on two metrics: 1) ambiguity for evaluating the predictive multiplicity of thresholded predictions, and 2) cumulative distribution function (CDF) of standard deviation (std.) of output scores when model outputs are in the interval [0, 1] (interpreted as the probability of the positive class). Those two metrics are defined as follows.
Definition 2 (Ambiguity [35]). Fix a dataset  $\mathcal{D} = \{(\pmb{x}_i, s_i, y_i)\}_{i \in [n]} \subset \mathbb{R}^d \times [K] \times \{0,1\}$  and a finite set of models  $\mathcal{R} \subset \mathcal{H}$ . Let  $f(r) \triangleq \mathbb{1}\{r \geq 0.5\}$  be the thresholding function. The ambiguity of a dataset over the set of models  $\mathcal{R}$  is the proportion of points in the dataset that can be assigned a conflicting prediction by a competing classifier within  $\mathcal{R}$ :
$$
\alpha (\mathcal {D}, \mathcal {R}) \triangleq \frac {1}{| \mathcal {D} |} \sum_ {i \in [ n ]} \max  _ {h, h ^ {\prime} \in \mathcal {R}} \mathbb {1} \left\{f \left(h \left(\boldsymbol {x} _ {i}\right)\right) \neq f \left(h ^ {\prime} \left(\boldsymbol {x} _ {i}\right)\right) \right\}. \tag {2}
$$
To define the CDF of std. of scores, we first delineate what we mean by empirical std. of scores.
Definition 3 (Std. of Scores). Fix a finite set of models  $\mathcal{R} = \{h_j\}_{j\in [m]}\subset \mathcal{H}$ . The empirical standard deviation (std.) of scores for a sample  $\pmb {x}\in \mathbb{R}^d$  relative to  $\mathcal{R}$  is defined by
$$
s (\boldsymbol {x}, \mathcal {R}) \triangleq \sqrt {\frac {1}{m - 1} \sum_ {j \in [ m ]} \left(h _ {j} (\boldsymbol {x}) - \bar {\mu} _ {\boldsymbol {x}}\right) ^ {2}}, \tag {3}
$$
where  $\bar{\mu}_{\pmb{x}} \triangleq \frac{1}{m} \sum_{j \in [m]} h_j(\pmb{x})$  denotes the empirical mean (over  $\mathcal{R}$ ) of the scores.
Further, to understand the std. of scores of the population, we consider the empirical cumulative distribution function of the std. of the scores, defined as follows.
Definition 4 (Quantiles of std. of Scores). Fix a dataset  $\mathcal{D} = \{(\pmb{x}_i, s_i, y_i)\}_{i \in [n]} \subset \mathbb{R}^d \times [K] \times \{0, 1\}$  and a finite set of models  $\mathcal{R} \subset \mathcal{H}$ . We define the empirical cumulative distribution function of the std. of the scores by (where  $s(\pmb{x}, \mathcal{R})$  is the empirical std. as in Definition 3)
$$
\widehat {F} _ {\mathcal {D}, \mathcal {R}} (t) \triangleq \frac {1}{| \mathcal {D} |} \sum_ {i \in [ n ]} \mathbb {1} \left\{s \left(\boldsymbol {x} _ {i}, \mathcal {R}\right) \leq t \right\}. \tag {4}
$$
Example 2. Consider a resume screening task where the algorithm decides whether to extend an interview opportunity. If  $\widehat{F}_{\mathcal{D},\mathcal{R}}(0.5) = 90\%$ , then for  $10\%$  of the individuals in the dataset, the predictions produced by the competing models are arbitrary and conflicting: regardless of the mean scores, with an std. of at least 0.5, there would exist models with scores falling above and below the one-half threshold, so the thresholded output can be both 0 (no interview) and 1 (offer interview).
A note on related metrics. An alternative measurement of score variation is Viable Prediction Range as defined in [40], which measures the difference in max and min scores among competing models on each individual. For thresholded scores, the original definition of Ambiguity [35] considers the proportion of a flip in prediction with respect to a baseline model (from empirical risk minimization with fixed hyperparameters and randomness). Since we consider randomized training procedures with no clear baseline model, the definition for ambiguity above is a variation of the original.
Group fairness. We consider three group fairness definitions for classification tasks—statistical parity (SP), equalized odds (EO), and overall accuracy equality (OAE) [11, 18, 23, 37]. OAE and Mean Equalized Odds (MEO) are defined below as they are used in the next sections, and we refer the reader to Appendix D for the remaining definitions.
Definition 5 (Overall Accuracy Equality, OAE). Let  $\widehat{Y}$  be the predicted label obtained, e.g., from thresholding the scores of a classifier  $h: \mathbb{R}^d \to [0,1]$ . The predictor  $\widehat{Y}$  satisfies overall accuracy equality (OAE) if its accuracy is independent of the group attribute: for all groups  $s, s' \in [K]$ ,
$$
\Pr (\widehat {Y} = Y \mid S = s) = \Pr (\widehat {Y} = Y \mid S = s ^ {\prime}). \tag {5}
$$
For binary classification, SP boils down to requiring the average predictions to be equal across groups, while EO requires true positive rates (TPR) and false positive rates (FPR) to be calibrated. In this paper, we consider mean equalized odds (MEO): the average of absolute difference in FPR and TPR for binary groups  $S \in \{0,1\}$ . We consider binary group since this is the setup for most fairness intervention methods.
Definition 6 (Mean Equalized Odds, MEO [4, 23]). Let  $\widehat{Y}$  be the predicted label,  $S \in \{0,1\}$  denotes binary group membership. Mean Equalized Odds is the average odds difference for binary groups:
$$
\mathrm {M E O} \triangleq \frac {1}{2} \left(\left| \mathrm {T P R} _ {S = 0} - \mathrm {T P R} _ {S = 1} \right| + \left| \mathrm {F P R} _ {S = 0} - \mathrm {F P R} _ {S = 1} \right|\right), \tag {6}
$$
where  $\mathrm{TPR}_{S = s} \triangleq \operatorname{Pr}(\widehat{Y} = 1 \mid Y = 1, S = s)$  and  $\mathrm{FPR}_{S = s} \triangleq \operatorname{Pr}(\widehat{Y} = 1 \mid Y = 0, S = s)$ .
To examine whether current fairness intervention methods lead to an exacerbation of multiplicity, we survey state-of-the-art intervention methods, including Reductions [1], Fair Projection [2], Reject Options [29], and EqOdds [23]. We offer a brief discussion of their mechanism in Appendix D.
3 Orthogonality of Fairness and Arbitrariness