SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We discuss next why arbitrariness is a third axis not captured by fairness and accuracy. Models with similar fairness and accuracy metrics can differ significantly in predictions. Moreover, a set of fair and approximately accurate models can attain maximal predictive multiplicity. We also explore through an example one fundamental reason why adding fairness constraints can lead to more arbitrariness.
Example 3 (Ambiguity  $\neq$  OAE). Overall Accuracy Equality (OAE, Definition 5) does not capture the ambiguity of model outputs (Definition 2). Consider two hypothetical models that are fair/unfair and exhibit high/low predictive multiplicity in Figure 2. Here, in each panel, the rectangle represents the input feature space and the shaded regions represent the error region of each model.
In the top left panel, both Model 1 and 2 have equal accuracy for both groups, since the proportion of the error regions (red stripes and pink shade) for both groups are the same. Hence, both models are considered group-fair in terms of OAE. However, the error regions of the two models are disjoint. Since ambiguity is measured by the percentage of the samples that receive conflicting predictions from either models, samples from the union of the two error regions contribute to ambiguity. Hence, Model 1 and 2 bear high predictive multiplicity despite being group-fair.
In the lower right panel, Model 1 and 2 attain low fairness and low predictive multiplicity. Both models have higher accuracy for Group 2 than Group 1, so they are both unfair. The error regions completely overlap, which means that the two models are making the same error—ambiguity is 0.
The schematic diagram in Figure 2 shows that predictive multiplicity is not captured by OAE. Indeed, ambiguity of a collection of models is a global property (i.e., verified at the collection level), whereas OAE is a local property (i.e., verified at the classifier level). Hence, one should not a priori expect that a set of competing models each satisfying OAE would necessarily comprise a Rashomon set with favorable ambiguity.
We prove the orthogonality of OAE and Statistical Parity (SP) from ambiguity formally in the proposition below, where we show that it is possible to construct classifiers with very stringent accuracy and perfect fairness constraint, albeit with maximal ambiguity. We determine the Rashomon set using the 0-1 loss:
$$
\ell_ {0 - 1} (h; \mathcal {D}) = \frac {1}{| \mathcal {D} |} \sum_ {\left(\boldsymbol {x} _ {i}, s _ {i}, y _ {i}\right) \in \mathcal {D}} \mathbb {1} \left\{\widehat {y} _ {i} \neq y _ {i} \right\},
$$
Figure 2: Illustration on two models being fair/unfair and exhibit high/low predictive multiplicity through the models' error regions in each of the 4 cases. The metrics for fairness and predictive multiplicity are Overall Accuracy Equality (Definition 5) and ambiguity (Definition 2), respectively.
where  $\widehat{y}_i\in \{0,1\}$  is the class membership of  $x_{i}$  predicted by  $h$  . We prove the following orthogonality in Appendix A.
Proposition 3.1 (Orthogonality of OAE/SP and Ambiguity). Fix any empirical loss value  $0 < \epsilon \leq \frac{1}{2}$  and any number of models  $m > \frac{1}{\epsilon}$ . Then, for some finite dataset  $\mathcal{D} \subset \mathbb{R}^d \times [K] \times \{0,1\}$ , there is a realization of the empirical Rashomon set  $\widehat{\mathcal{R}}_m = \{h_j\}_{j \in [m]}$  satisfying the following simultaneously:
Figure 3: Data distribution of a population with two groups used in Example 2 (Left). In Right, without the Mean EO constraint (6) (green line), there is a unique optimal classifier (with threshold 0) that attains the smallest probability of error (blue line). Adding the Mean EO constraint enlarges the set of optimal threshold classifiers to two classifiers (red and blue dots) with indistinguishable accuracy and fairness levels (Right) but different decision regions. We illustrate the decision regions of each classifier as red and blue arrows on the Left.
1. Each  $h_j$  has 0-1 loss upper bounded by  $\ell_{0-1}(h_j; \mathcal{D}) \leq \epsilon$ ;
2. Each  $h_j$  satisfies OAE perfectly, or each  $h_j$  satisfies SP perfectly;
3. The collection  $\widehat{\mathcal{R}}_m$  has the worst ambiguity, i.e.,  $\alpha (\mathcal{D},\widehat{\mathcal{R}}_m) = 100\%$
Remark 1. For OAE, such Rashomon set  $\widehat{\mathcal{R}}_m$  exists for any dataset  $\mathcal{D}$  satisfying the two conditions:
1. with  $n_k$  denoted the number of samples in  $\mathcal{D}$  belonging to group  $k \in [K]$ , the greatest common divisor of the  $n_k$  is at least  $(m - 1) / (m\epsilon - 1)$ .
2. if  $(\pmb {x},s,y),(\pmb {x},s^{\prime},y^{\prime})\in \mathcal{D}$  share the same feature vector  $\pmb{x}$  , then  $y = y^{\prime}$  too.
The requirement that the  $n_k$  share a large enough common divisor is used in the proof to guarantee perfect OAE. One could relax this requirement at the cost of nonzero OAE violation.
Proposition 3.1 implies that there exists a dataset and competing models for which all samples receive conflicting predictions. Specifically, we can construct a large enough set of competing models  $(m > \frac{1}{\epsilon})$  such that  $100\%$  of the samples in the dataset can receive conflicting predictions from this set of perfectly fair models with respect to OAE.
In the next example, we demonstrate that, counter-intuitively, adding a fairness constraint can enlarge the set of optimal models, thereby increasing predictive multiplicity. This points to a fundamental reason why adding fairness constraints can lead to more arbitrariness in model decisions.
Example 4 (Arbitrariness of Threshold Classifiers with Fairness Constraint). Given a data distribution of a population with two groups (Figure 3 Left), consider the task of selecting a threshold classifier that predicts the true label. Without fairness considerations, the optimal threshold is 0 - i.e., assigning positive predictions to samples with  $x > 0$  and negative predictions to  $x \leq 0$  minimizes the probability of error (Figure 3 Right). This optimal model is unique. Adding a fairness constraint that requires Mean EO  $\leq 0.1$ , the previously optimal classifier at 0 (with Mean EO = 0.15, Right) does not meet the fairness criteria. Searching over the threshold classifiers that minimize the probability of error while satisfying Mean EO constraint yields two equally optimal models (red and blue dots Right) with distinct decision regions (red and blue arrows Left). Even in this simple hypothesis class, the addition of fairness constraints yields multiple models with indistinguishable fairness and accuracy but with distinct decision regions. The arbitrary selection between these points can lead to arbitrary outputs to points near the boundary.
4 Ensemble Algorithm for Arbitrariness Reduction
To tackle the potential arbitrariness cost of fairness intervention algorithms, we present a disparity-reduction mechanism through ensembling. We provide theoretical guarantees and numerical benchmarks to demonstrate that this method significantly reduces the predictive multiplicity of fair and accurate models.
In a nutshell, given competing models  $h_1,\dots ,h_m$ , we argue that the disparity in their score assignment can be reduced by considering a convex combination of them, defined as follows.
Definition 7 (Ensemble Classifier). Given  $m$  classifiers  $\{h_1,\dots ,h_m:\mathbb{R}^d\to [0,1]\}$  and a vector  $\lambda \in \Delta_{m}$ , we define the  $\lambda$ -ensembling of the  $h_j$  to be the convex combination  $h^{\mathrm{ens},\lambda}\triangleq \sum_{j\in [m]}\lambda_jh_j$ .
4.1 Concentration of Ensembled Scores
We prove in the following result that any two different ensembling methods agree for fixed individuals with high probability. Recall that we fix a dataset  $\mathcal{D}$  and a set of competing models  $\mathcal{T}(\mathcal{D})$  coming from a stochastic training algorithm  $\mathcal{T}$  (see Section 2). All proofs are provided in Appendix B.
Theorem 4.1 (Concentration of Ensembles' Scores). Let  $h_1, \ldots, h_m; \widetilde{h}_1, \ldots, \widetilde{h}_m \stackrel{iid}{\sim} \mathcal{T}(\mathcal{D})$  be  $2m$  models drawn from  $\mathcal{T}(\mathcal{D})$ , and  $\pmb{h}^{\mathrm{ens},\lambda}, \widetilde{\pmb{h}}^{\mathrm{ens},\gamma}$  be the ensembled models (constructed with  $\{h_1, \ldots, h_m\}$  and  $\{\widetilde{h}_1, \ldots, \widetilde{h}_m\}$  respectively) for  $\lambda, \gamma \in \Delta_m$  (see Definition 7) satisfying  $\| \pmb{\lambda} \|_2^2, \| \pmb{\gamma} \|_2^2 \leq c / m$  for an absolute constant  $c$ . For every  $\pmb{x} \in \mathbb{R}^d$  and  $\nu \geq 0$ , we have the exponentially-decaying (in  $m$ ) bound
$$
\mathbb {P} \left(\left| \boldsymbol {h} ^ {\text {e n s}, \boldsymbol {\lambda}} (\boldsymbol {x}) - \widetilde {\boldsymbol {h}} ^ {\text {e n s}, \gamma} (\boldsymbol {x}) \right| \geq \nu\right) \leq 4 e ^ {- \nu^ {2} m / (2 c)}. \tag {7}
$$
In particular, for any validation set  $\mathcal{D}_{\mathrm{valid.}} \subset \mathbb{R}^d$  of size  $|\mathcal{D}_{\mathrm{valid.}}| = n$ , we have the uniform bound
$$
\mathbb {P} \left(\left| \boldsymbol {h} ^ {\text {e n s}, \boldsymbol {\lambda}} (\boldsymbol {x}) - \widetilde {\boldsymbol {h}} ^ {\text {e n s}, \gamma} (\boldsymbol {x}) \right| <   \nu \text {f o r a l l} \boldsymbol {x} \in \mathcal {D} _ {\text {v a l i d .}}\right) > 1 - 4 n e ^ {- \nu^ {2} m / (2 c)}. \tag {8}
$$
4.2 Concentration of Predictions Under Ensembling
The above theorem implies that we can have a dataset of size that is exponential in the number of accessible competing models and still obtain similar scoring for any two ensembled models (uniformly across the dataset).
In practice, one cares more about the agreement of the final prediction of the classifiers. The following result extends Theorem 4.1 to the concentration of thresholded classifiers. For this, we need to define the notion of confident classifiers.
Definition 8 (Confident Classifier). Fix a probability measure  $P_{\mathbf{X}}$  over  $\mathbb{R}^d$  and constants  $\delta, \theta \in [0,1]$ . We say that a classifier  $h: \mathbb{R}^d \to [0,1]$  is  $(P_{\mathbf{X}}, \delta, \theta)$ -confident if  $\mathbb{P}\left(|h(\mathbf{X}) - \frac{1}{2}| < \delta\right) < \theta$ .
In other words,  $h$  is a confident classifier if it is "more sure" of its predictions. We observe in experiments that models corrected by fairness interventions have scores concentrated around 0 and 1.
Using confident classifiers, we are able to extend Theorem 4.1 to thresholded ensembles, as follows.
Theorem 4.2. Let  $h^{\mathrm{ens},\lambda}$ ,  $\widetilde{h}^{\mathrm{ens},\gamma}$  be as in Theorem 4.1, and assume that both ensembled classifiers are  $(P_X, \delta, \theta)$ -confident in the sense of Definition 8. Let  $f(t) \triangleq \mathbb{1}\{t \geq 0.5\}$  be the thresholding function. For any set  $\mathcal{D}_{\mathrm{valid.}} \subset \mathbb{R}^d$  of size  $|\mathcal{D}_{\mathrm{valid.}}| = n$ , we may guarantee the probability of agreement in the predictions for all samples under the two ensembles to be at least
$$
\mathbb {P} \left(f (\boldsymbol {h} ^ {\text {e n s}, \boldsymbol {\lambda}} (\boldsymbol {x})) = f (\widetilde {\boldsymbol {h}} ^ {\text {e n s}, \gamma} (\boldsymbol {x})) \text {f o r e v e r y} \boldsymbol {x} \in \mathcal {D} _ {\text {v a l i d .}}\right) \geq 1 - \left(4 e ^ {- 2 \delta^ {2} m / c} + 2 \theta\right) n. \tag {9}
$$
We note that in the fairness-intervention setting, the set  $\mathcal{D}_{\mathrm{valid}}$  in the above theorem would be chosen as the subset of samples having the same group attribute. Thus, the size  $n_0$  of  $\mathcal{D}_{\mathrm{valid}}$  would be significantly smaller than the total size of the dataset, and the parameter  $\theta$  then can be required to be moderately small.
Remark 2. In Appendix C, we discuss how to optimize the ensembling parameters  $\lambda$ . In the next section, we will stick to the uniform ensembling:  $h^{\mathrm{ens},\lambda} = \frac{1}{m}\sum_{j\in [m]}h_j$ , i.e.,  $\lambda = \frac{1}{m}\mathbf{1}$ . This simple uniform ensemble suffices to illustrate the main goal of this paper: that arbitrariness can be a by-product of fairness intervention methods, and ensembling can mitigate this unwanted effect.
We present empirical results to show that arbitrariness is masked by favorable group-fairness and accuracy metrics for multiple fairness intervention methods, baseline models, and datasets. We also demonstrate the effectiveness of the ensemble in reducing the predictive multiplicity of fair models.
$^{7}$ Code can be found at https://github.com/Carol-Long/Fairness_and_Arbitrariness
Figure 4: Quantile plot on high-fairness bin for various fairness interventions v.s. baseline on ENEM. Left: Fairness-Accuracy frontier. Right: Fair models produce larger score std. at top percentiles compared to the baseline model (horizontal axis computed via (6)). (REJECTION and LEVERAGING output thresholded scores directly.)
Setup and Metrics. We consider three baseline classifiers (BASE): random forest (RF), gradient boosting (GBM), and logistic regression (LR), implemented by Scikit-learn [36]. By varying the random seed, we obtain 10 baseline models with comparable performance. Then, we apply various state-of-the-art fairness methods (details in Appendix D) on the baseline models to get competing fair models.
On the test set, we compute mean accuracy, Mean EO (Definition 6), and predictive multiplicity levels on competing models before and after fairness interventions. We use ambiguity (Definition 2) and score standard deviations (Definition 3) as metrics for predictive multiplicity.
Datasets. We report predictive multiplicity and benchmark the ensemble method on three datasets - two datasets in the education domain: the high-school longitudinal study (HSLS) dataset [27, 28] and the ENEM dataset [16] (see Alghamdi et al. [2] Appendix B.1), and the UCI Adult dataset[33] which is based on the US census income data. The ENEM dataset contains Brazilian college entrance exam scores along with student demographic information and socio-economic questionnaire answers (e.g. if they own a computer). After pre-processing, the dataset contains 1.4 million samples with 139 features. Race is used as the group attribute  $S$ , and Humanities exam score is used as the label  $Y$ . Scores are quantized into two classes for binary classification. The race feature  $S$  is binarized into White and Asian ( $S = 1$ ) and others ( $S = 0$ ). The experiments are run with a smaller version of the dataset with 50k samples. Complete experimental results can be found in Appendix E.
Results that Reveal Arbitrariness. We juxtapose the fairness-accuracy frontier and metrics for predictive ambiguity to reveal arbitrariness masked by favorable group-fairness and accuracy metrics in Figure 1 and 4. Starting with 10 baseline classifiers by varying the random seed used to initialize the training algorithm, we apply the fair interventions REDUCTION [1], REJECTION [29], LEVERAGING [12] to obtain point clouds of models with comparable fairness and accuracy metrics. In Figure 4, we take models that achieve very favorable accuracy and MEO metrics (in blue rectangle in Left) and plot the std. of scores to illustrate predictive multiplicity Right. Group fairness violations are greatly reduced (from 0.28 in baseline to 0.04 in fair models) at a small accuracy cost (from  $67\%$  in baseline to  $66\%$  in fair models). However, there is higher arbitrariness.
Compared to baseline (red curve), fair models corrected by REDUCTION and ROC produce lower score arbitrariness for the bottom  $50\%$  but much higher arbitrariness for the top  $50\%$  of samples; importantly, the induced arbitrariness becomes highly nonuniform across different individuals after applying the two fairness intervention. We observe that LEVERAGING produce models that agree on  $90\%$  of the samples, thereby not inducing concerns of arbitrariness.
Remarkably, arbitrariness does not vary significantly among models with different fairness levels. We consider two sets of models trained with high and low fairness constraints using REDUCTION in Figure 1.
Results on the Effectiveness of Ensembling. We pair our proofs in Section 4 with experiments that demonstrate the concentration of scores of ensembled models. In Figure 5 Left, taking the competing
Figure 5: Standard deviation of ensembled models trained on ENEM and HSLS with baseline random forest classifiers. We fix the high-fairness bin and vary the number of models  $m$  in each ensemble. As we increase the number of ensembles, score std. (on 10 ensembles) drops and meets the score std. of 10 baseline RFC when  $m = 30$  on ENEM and  $m = 17$  on HSLS. (Mean EO is computed using (6).
models in the high-fairness bins corrected with REDUCTION that achieve an Mean EO violation of 0.04 but very high score std. for half of the samples (blue rectangle in Figure 4), we ensemble the models with increasing number of models per ensemble  $(m)$  ranging from 1 to 30. For each  $m$ , we measure std. of scores in 10 such ensembles. The top percentile std. of the ensembled fair models drops to baseline with 30 models. Similar convergence occurs on the HSLS dataset. Importantly, the ensembled models are still fair, the Mean EO violations of the ensembled models remain low.
6 Final Remarks
We demonstrate in this paper that arbitrariness is a facet of responsible machine learning that is orthogonal to existing fairness-accuracy analyses. Specifically, fairness-vs-accuracy frontiers are insufficient for detecting arbitrariness in the predictions of group-fair models: two models can have the same fairness-accuracy curve while at the same time giving widely different predictions for subsets of individuals. We demonstrate this undesirable phenomenon both theoretically and experimentally on state-of-the-art fairness intervention methods. Furthermore, towards mitigating this arbitrariness issue, we propose an ensemble algorithm, where a convex combination of several competing models is used for decision-making instead of any of the constituent models. We prove that the scores of the ensemble classifier concentrate, and that the ensuing predictions can be made to concentrate under mild assumptions. Importantly, we exhibit via real-world experiments that our proposed ensemble algorithm can reduce arbitrariness while maintaining fairness and accuracy.
Limitations. The proposed framework for estimating the predictive multiplicity of fairness interventions requires re-training multiple times, limiting its applicability to large models. We consider model variation due to randomness used during training. In practice, competing models may exist due to inherent uncertainty (i.e., a non-zero confidence interval) when evaluating model performance on a finite test set. In this regard, models with comparable average performance can be produced by searching over this Rashomon set even if training is deterministic (e.g., equivalent to solving a convex optimization problem).
Future directions. An interesting future direction is to explore the multiplicity cost of fairness interventions in such deterministic settings. Furthermore, our ensembling strategy may not guarantee that the ensemble classifier ensures fairness constraints due to the non-convex nature of such constraints. Though we empirically observe that fairness constraints are indeed satisfied by the ensemble model, proving such guarantees theoretically would be valuable.
Societal impacts. While fairness intervention algorithms can effectively reduce the disparate impact among population groups, they can induce predictive multiplicity in individual samples. The increase in predictive multiplicity does not impact all individuals equally. If predictive multiplicity caused by fairness interventions is not accounted for, some individuals will bear the brunt of arbitrary decision-making—their predictions could be arbitrary upon re-training the classifier using different random initializations, leading to another level of disparate treatment to certain population groups.
Acknowledgements. The authors would like to thank Jamelle Watson-Daniels, Arpita Biswas and Bogdan Kulynych for hepful discussions on initial ideas. This material is based upon work supported by the National Science Foundation under grants CAREER 1845852, CIF 1900750, CIF 2312667 and FAI 2040880, and by Meta Ph.D. fellowship.
Disclaimer. This paper was prepared by Hsiang Hsu prior to his employment at JPMorgan Chase & Co.. Therefore, this paper is not a product of the Research Department of JPMorgan Chase & Co. or its affiliates. Neither JPMorgan Chase & Co. nor any of its affiliates makes any explicit or implied representation or warranty and none of them accept any liability in connection with this paper, including, without limitation, with respect to the completeness, accuracy, or reliability of the information contained herein and the potential legal, compliance, tax, or accounting effects thereof. This document is not intended as investment research or investment advice, or as a recommendation, offer, or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction.
[1] Agarwal, A., Beygelzimer, A., Dudík, M., Langford, J., and Wallach, H. (2018). A reduction approach to fair classification. In International Conference on Machine Learning, pages 60–69. PMLR.
[2] Alghamdi, W., Hsu, H., Jeong, H., Wang, H., Michalak, P., Asoodeh, S., and Calmon, F. (2022). Beyond adult and compass: Fair multi-class prediction via information projection. Advances in Neural Information Processing Systems, 35:38747-38760.
[3] Bahri, D. and Jiang, H. (2021). Locally adaptive label smoothing for predictive churn. arXiv preprint arXiv:2102.05140.
[4] Bellamy, R. K., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2019). Ai fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias. IBM Journal of Research and Development, 63(4/5):4-1.
[5] Bhojanapalli, S., Wilber, K., Veit, A., Rawat, A. S., Kim, S., Menon, A., and Kumar, S. (2021). On the reproducibility of neural network predictions. arXiv preprint arXiv:2102.03349.
[6] Black, E., Leino, K., and Fredrikson, M. (2021). Selective ensembles for consistent predictions. arXiv preprint arXiv:2111.08230.
[7] Black, E., Raghavan, M., and Barocas, S. (2022). Model multiplicity: Opportunities, concerns, and solutions. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 850-863.
[8] Breiman, L. (1996). Bagging predictors. Machine learning, 24:123-140.
[9] Breiman, L. (2001). Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical science, 16(3):199-231.
[10] Caton, S. and Haas, C. (2020). Fairness in machine learning: A survey. arXiv preprint arXiv:2010.04053.
[11] Chouldechova, A. (2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. *Big data*, 5(2):153–163.
[12] Chzhen, E., Denis, C., Hebiri, M., Oneto, L., and Pontil, M. (2019). Leveraging labeled and unlabeled data for consistent fair binary classification. Advances in Neural Information Processing Systems, 32.
[13] Cooper, A. F., Barocas, S., De Sa, C., and Sen, S. (2023). Variance, self-consistency, and arbitrariness in fair classification. arXiv preprint arXiv:2301.11562.
[14] Coston, A., Rambachan, A., and Chouldechova, A. (2021). Characterizing fairness over the set of good models under selective labels. In International Conference on Machine Learning, pages 2144-2155. PMLR.