SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[32] Kulynych, B., Hsu, H., Troncoso, C., and Calmon, F. P. (2023). Arbitrary decisions are a hidden cost of differentially-private training. In Conference on Fairness, Accountability, and Transparency (FAccT). ACM.
[33] Lichman, M. et al. (2013). Uci machine learning repository, 2013.
[34] Lipton, Z., McAuley, J., and Chouldechova, A. (2018). Does mitigating ml's impact disparity require treatment disparity? Advances in neural information processing systems, 31.
[35] Marx, C., Calmon, F., and Ustun, B. (2020). Predictive multiplicity in classification. In International Conference on Machine Learning, pages 6765-6774. PMLR.
[36] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. (2011). Scikit-learn: Machine learning in python. the Journal of machine learning research, 12:2825-2830.
[37] Pleiss, G., Raghavan, M., Wu, F., Kleinberg, J., and Weinberger, K. Q. (2017). On fairness and calibration. Advances in neural information processing systems, 30.
[38] Roth, A., Tolbert, A., and Weinstein, S. (2022). Reconciling individual probability forecasts. arXiv preprint arXiv:2209.01687.
[39] Semenova, L., Rudin, C., and Parr, R. (2019). A study in rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning. arXiv preprint arXiv:1908.01755.
[40] Watson-Daniels, J., Parkes, D. C., and Ustun, B. (2022). Predictive multiplicity in probabilistic classification. arXiv preprint arXiv:2206.01131.
[41] Witten, I. H. and Frank, E. (2002). Data mining: practical machine learning tools and techniques with java implementations. Acm Sigmoid Record, 31(1):76-77.
The appendix is divided into the following five parts. Appendix A: proof of the orthogonality of OAE and SP from ambiguity (Proposition 3.1); Appendix B: proofs from Section 4; Appendix C: discussion on optimizing ensemble parameters; Appendix D: additional discussions on group fairness and fairness interventions; and Appendix E: additional experiments and details on the experimental setup.
A Proof of Proposition 3.1: Orthogonality of OAE and SP from Ambiguity
We divide the proof into two cases according to the group-fairness metric considered (OAE or SP).
A.1 Proof for the OAE Metric
Let  $\mathcal{D} \subset \mathbb{R}^d \times [K] \times \{0,1\}$  be a dataset satisfying the two conditions listed in Remark 1, namely,
1. with  $n_k$  denoting the number of samples in  $\mathcal{D}$  belonging to group  $k \in [K]$ , the greatest common divisor of the  $n_k$  is at least  $(m - 1) / (m\epsilon - 1)$ .
2. if  $(\pmb {x},s,y),(\pmb {x},s^{\prime},y^{\prime})\in \mathcal{D}$  share the same feature vector  $\pmb{x}$  , then  $y = y^{\prime}$  too.
Consider the partition along group membership  $\mathcal{D} = \bigcup_{k\in [K]}\mathcal{D}_k$ , so we may write  $\mathcal{D}_k = \{(x_{k,i},k,y_{k,i})\}_{i\in [n_k]}$  for each  $k\in [K]$ , where  $n_k\triangleq |\mathcal{D}_k|$ . By assumption on the  $n_k$ , we have
$$
g \triangleq \operatorname {g c d} \left(n _ {1}, \dots , n _ {K}\right) \geq \frac {m - 1}{m \epsilon - 1}. \tag {10}
$$
Now, let  $\theta \triangleq \lfloor \epsilon g \rfloor / g$ , and note that we obtain the integers  $\mu_k \triangleq \theta \cdot n_k$  for  $k \in [K]$ , because  $g$  divides each  $n_k$  by definition of the gcd. Let  $\lambda \triangleq \left\lceil \frac{1 - \epsilon}{\theta} \right\rceil$ . Using  $\lfloor t \rfloor > t - 1$ , we note that
$$
\frac {1 - \epsilon}{\theta} + 1 <   \frac {1 - \epsilon}{\epsilon - \frac {1}{g}} + 1 \leq m, \tag {11}
$$
where the last inequality follows by assumption on  $g$  being large enough (10). Thus, taking the ceiling of both sides above, we obtain  $\lambda + 1 \leq m$ . Hence, it suffices to prove that there are  $\lambda + 1$  classifiers  $\mathcal{R} = \{h_j\}_{j \in [\lambda + 1]}$  satisfying the claim in the proposition (i.e., with  $\lambda + 1$  replacing  $m$ ).
It is straightforward to check that  $\lambda \mu_k \leq n_k$  for each  $k$ . We divide each  $\mathcal{D}_k$  into  $\lambda$  sets of size  $\mu_k$  each, and collect the remainder into a separate subset. Thus, for each  $k \in [K]$ , consider any partition  $\mathcal{D}_k = \bigcup_{j \in [\lambda + 1]} \mathcal{D}_k^j$  where  $|\mathcal{D}_k^j| = \mu_k$  for each  $j \in [\lambda]$  and  $|\mathcal{D}_k^{\lambda + 1}| = n_k - \lambda \mu_k$ .
Next, we define the classifiers based on the partitions of the  $\mathcal{D}_k$ . For each  $j \in [\lambda + 1]$ , define the classifier  $h_j$  over  $\mathcal{D}$  as follows. Fix  $(\pmb{x}, k, y) \in \mathcal{D}$ . We set the value  $h_j(\pmb{x})$  to be
$$
h _ {j} (\boldsymbol {x}) \triangleq \left\{ \begin{array}{l l} 1 - y & \text {i f} (\boldsymbol {x}, k, y) \in \mathcal {D} _ {k} ^ {j}, \\ y & \text {o t h e r w i s e .} \end{array} \right. \tag {12}
$$
Note that this makes  $h_j$  well-defined by the second property assumed on  $\mathcal{D}$  at the beginning of this proof. We show that the set  $\mathcal{R} = \{h_j\}_{j \in [\lambda + 1]}$  satisfies the desired properties in the proposition.
Accuracy. First, we show that each  $h_j$  incurs a 0-1 loss less than  $\epsilon$ . Indeed, we have that, for each  $j \in [\lambda]$ ,
$$
\ell \left(h _ {j}; \mathcal {D}\right) = \frac {1}{n} \sum_ {k \in [ K ]} \left| \mathcal {D} _ {k} ^ {j} \right| = \frac {1}{n} \sum_ {k \in [ K ]} \mu_ {k} = \theta , \tag {13}
$$
whereas the error for the last classifier is
$$
\ell \left(h _ {\lambda + 1}; \mathcal {D}\right) = \frac {1}{n} \sum_ {k \in [ K ]} \left| \mathcal {D} _ {k} ^ {\lambda + 1} \right| = \frac {1}{n} \sum_ {k \in [ K ]} n _ {k} - \lambda \mu_ {k} = 1 - \lambda \theta . \tag {14}
$$
Thus, it suffices to check that  $\theta, 1 - \lambda \theta \leq \epsilon$ . Recall that we set  $\theta = \lfloor \epsilon g \rfloor / g$ , hence  $\theta \leq \epsilon$  is immediate. Further, as we take  $\lambda = \left\lceil \frac{1 - \epsilon}{\theta} \right\rceil \geq \frac{1 - \epsilon}{\theta}$ , the inequality  $1 - \lambda \theta \leq \epsilon$  follows immediately too. Hence, we have that  $\ell(h_j; \mathcal{D}) \leq \epsilon$  for all  $j \in [\lambda + 1]$ .
Fairness. Next, we check that each  $h_j$  satisfies Overall Accuracy Equality (OAE) perfectly. Let  $\widehat{Y}_j$  be the prediction of the classifier  $h_j$ , so  $P_{\widehat{Y}_j|\mathbf{X} = \mathbf{x}}(1) = h_j(\mathbf{x})$ . Then, for  $j \in [\lambda]$  the predictions of the  $j$ -th classifier satisfy
$$
\Pr \left(\widehat {Y} _ {j} \neq Y \mid S = k\right) = \frac {\left| \mathcal {D} _ {k} ^ {j} \right|}{n _ {k}} = \frac {\mu_ {k}}{n _ {k}} = \theta \quad \text {f o r e v e r y g r o u p} k \in [ K ], \tag {15}
$$
and similarly the last classifier satisfies
$$
\Pr \left(\widehat {Y} _ {\lambda + 1} \neq Y \mid S = k\right) = \frac {\left| \mathcal {D} _ {k} ^ {\lambda + 1} \right|}{n _ {k}} = \frac {n _ {k} - \lambda \mu_ {k}}{n _ {k}} = 1 - \lambda \theta \quad \text {f o r e v e r y g r o u p} k \in [ K ]. \tag {16}
$$
Hence, the  $h_j$ , for  $j \in [\lambda + 1]$ , all satisfy OAE perfectly.
Ambiguity. Finally, we show that the set  $\mathcal{R} = \{h_j\}_{j \in [\lambda + 1]}$  exhibits full ambiguity. Note that we have a partition  $\mathcal{D} = \bigcup_{(k,j) \in [K] \times [\lambda + 1]} \mathcal{D}_k^j$ . Fix a sample  $(\boldsymbol{x}, s, y) \in \mathcal{D}$  and consider the index  $(k,j) \in [K] \times [\lambda + 1]$  of the unique part including it in the partition, i.e.,  $(\boldsymbol{x}, s, y) \in \mathcal{D}_k^j$ . By construction of  $\mathcal{R}$ , we have that  $h_j(\boldsymbol{x}) = 1 - y$  (so  $\widehat{Y}_j = 1 - y$ ) but  $h_{j'}(\boldsymbol{x}) = y$  (so  $\widehat{Y}_{j'} = y$ ) for any  $j' \neq j$ . In other words, for every fixed sample in  $\mathcal{D}$ , there is a pair of classifiers in  $\mathcal{R}$  assigning it conflicting predictions. Thus, the ambiguity is  $\alpha(\mathcal{D}, \mathcal{R}) = 100\%$ .
A.2 Proof for the SP Metric
Consider now the SP group-fairness constraint instead. In this case, we define we construct the following alternative dataset  $\mathcal{D} \subset \mathbb{R}^d \times [K] \times \{0,1\}$ . Let  $t \triangleq \lceil 1 / \epsilon \rceil \geq 2$ . For each  $(k,y) \in [K] \times \{0,1\}$ , fix any dataset  $\mathcal{D}_{k,y} = \{(x_{k,y,i}, k,y)\}_{i \in [t]} \subset \mathbb{R}^d \times \{k\} \times \{y\}$ , let  $\mathcal{D} \triangleq \bigcup_{(k,y) \in [K] \times \{0,1\}} \mathcal{D}_{k,y} \subset \mathbb{R}^d \times [K] \times \{0,1\}$ , and assume that if  $x_{k,y,i} = x_{k',y',i'}$  then  $y = y'$ . Denote  $n \triangleq |\mathcal{D}| = 2Kt$ .
For each  $(k,y)\in [K]\times \{0,1\}$ , let  $\sigma_{k,y}$  be a permutation on  $[t]$ , i.e.,  $\{\sigma_{k,y}(j)\}_{j\in [t]} = [t]$ . Define the set of classifiers  $\mathcal{R} = \{h_j\}_{j\in [t]}$  on  $\mathcal{D}$  as follows. For each  $j\in [t]$  and each  $(k,y,i)\in [K]\times \{0,1\} \times [t]$ , we set
$$
h _ {j} \left(\boldsymbol {x} _ {k, y, i}\right) \triangleq \left\{ \begin{array}{l l} 1 - y & \text {i f} i = \sigma_ {k, y} (j), \\ y & \text {o t h e r w i s e .} \end{array} \right. \tag {17}
$$
We show that the set  $\mathcal{R} = \{h_j\}_{j\in [t]}$  satisfies the desired properties in the proposition.
Accuracy. Consider first the 0-1 loss incurred by the  $h_j$ . We have that, for each  $j \in [t]$ ,
$$
\ell \left(h _ {j}; \mathcal {D}\right) = \frac {1}{n} \sum_ {(k, y, i) \in [ K ] \times \{0, 1 \} \times [ t ]} \mathbb {1} \{i = \sigma_ {k, y} (j) \} = \frac {1}{n} \sum_ {(k, y) \in [ K ] \times \{0, 1 \}} 1 = \frac {2 K}{2 K t} = \frac {1}{t}. \tag {18}
$$
Therefore,  $\ell(h_j; \mathcal{D}) = 1 / t = 1 / \lceil 1 / \epsilon \rceil \leq \epsilon$ . In other words,  $\mathcal{R}$  is a realization of the  $\epsilon$ -level empirical Rashomon set of size  $t$ .
Fairness. Next, we check that each  $h_j$  satisfies SP perfectly. In other words, we check that  $h_j$  assigns to the class  $y = 1$  the same percentage of samples across the groups  $k \in [K]$ . Indeed, this is true as we are switching the class memberships of exactly one sample from each of the  $\mathcal{D}_{k,y}$ . In particular,  $h_j$  assigns the class membership 1 to exactly  $1 + (|\mathcal{D}_{k,1}| - 1) = |\mathcal{D}_{k,1}| = t$  samples out of the total  $|\mathcal{D}_{k,0} \cup \mathcal{D}_{k,1}| = 2t$  samples belonging group  $k$ . As the ratio  $t / (2t) = 1 / 2$  is independent of the group  $k$ , we obtain the desired result that  $h_j$  satisfies SP.
Ambiguity. Finally, we check that the set  $\mathcal{R} = \{h_j\}_{j\in [t]}$  suffers from  $100\%$  ambiguity. Fix a sample  $(\pmb{x}_{k,y,i},k,y)\in \mathcal{D}$ , and we will show that there are two classifiers in  $\mathcal{R}$  assigning conflicting predictions to it. Indeed, let  $j = \sigma_{k,y}^{-1}(i)$  and let  $j^{\prime}\in [t]\setminus \{j\}$  be any other index. Then,  $h_j(\pmb{x}_{k,y,i}) = 1 - y$ , whereas  $h_{j^{\prime}}(\pmb{x}_{k,y,i}) = y$ . Therefore, the sample  $(\pmb{x}_{k,y,i},k,y)$  contributes to the overall ambiguity  $\alpha (\mathcal{D},\mathcal{R})$ . As this is true for all samples in  $\mathcal{D}$ , we conclude that  $\alpha (\mathcal{D},\mathcal{R}) = 100\%$ , as desired. This completes the proof of the proposition.
We assume that  $\| \pmb{\lambda}\| _2^2,\| \pmb {\gamma}\| _2^2\leq c / m$  for an absolute constant  $c$ , e.g., we have  $c = 1$  for the uniform assembling  $\pmb {\lambda} = (1 / m,\dots ,1 / m)$  as then  $\| \pmb {\lambda}\| _2^2 = 1 / m$ . Fix  $\pmb{x}$ , and denote the mean of the classifiers  $\mu_{\pmb{x}} = \mathbb{E}_{h\sim \mathcal{T}(\mathcal{D})}[h(\pmb {x})]$ . The mapping  $(h_1,\dots ,h_m)\mapsto h^{\mathrm{ens},\pmb{\lambda}}$  satisfies the bounded-difference condition in the McDiarmid inequality. Indeed, changing  $h_i$  can change  $h^{\mathrm{ens},\pmb{\lambda}}$  by at most  $\lambda_{i}$ . Furthermore,  $h^{\mathrm{ens},\pmb{\lambda}}(\pmb {x})$  has the mean
$$
\mathbb {E} \left[ \boldsymbol {h} ^ {\text {e n s}, \boldsymbol {\lambda}} (\boldsymbol {x}) \right] = \sum_ {i \in [ m ]} \lambda_ {i} \mathbb {E} \left[ h _ {i} (\boldsymbol {x}) \right] = \mu_ {\boldsymbol {x}} \sum_ {i \in [ m ]} \lambda_ {i} = \mu_ {\boldsymbol {x}}. \tag {19}
$$
Hence, by Mcdiarmid's inequality, we have the bound
$$
\mathbb {P} \left(\left| \boldsymbol {h} ^ {\text {e n s}, \boldsymbol {\lambda}} (\boldsymbol {x}) - \mu_ {\boldsymbol {x}} \right| \geq \nu\right) \leq 2 \exp \left(\frac {- 2 \nu^ {2}}{\sum_ {i \in [ m ]} \lambda_ {i} ^ {2}}\right) \leq 2 \exp \left(\frac {- 2 \nu^ {2} m}{c}\right). \tag {20}
$$
The same inequality holds for  $\gamma$  in place of  $\lambda$ :
$$
\mathbb {P} \left(\left| \widetilde {\boldsymbol {h}} ^ {\text {e n s}, \gamma} (\boldsymbol {x}) - \mu_ {\boldsymbol {x}} \right| \geq \nu\right) \leq 2 \exp \left(\frac {- 2 \nu^ {2} m}{c}\right). \tag {21}
$$
Therefore, we obtain the bound
$$
\begin{array}{l} 1 - \mathbb {P} \left(\left| \boldsymbol {h} ^ {\text {e n s}, \boldsymbol {\lambda}} (\boldsymbol {x}) - \widetilde {\boldsymbol {h}} ^ {\text {e n s}, \gamma} (\boldsymbol {x}) \right| \geq \nu\right) = \mathbb {P} \left(\left| \boldsymbol {h} ^ {\text {e n s}, \boldsymbol {\lambda}} (\boldsymbol {x}) - \mu_ {\boldsymbol {x}} + \mu_ {\boldsymbol {x}} - \widetilde {\boldsymbol {h}} ^ {\text {e n s}, \gamma} (\boldsymbol {x}) \right| <   \nu\right) (22) \\ \geq \mathbb {P} \left(\left| \boldsymbol {h} ^ {\text {e n s}, \boldsymbol {\lambda}} (\boldsymbol {x}) - \mu_ {\boldsymbol {x}} \right| + \left| \widetilde {\boldsymbol {h}} ^ {\text {e n s}, \gamma} (\boldsymbol {x}) - \mu_ {\boldsymbol {x}} \right| <   \nu\right) (23) \\ \geq \mathbb {P} \left(\left| \boldsymbol {h} ^ {\text {e n s}, \boldsymbol {\lambda}} (\boldsymbol {x}) - \mu_ {\boldsymbol {x}} \right| <   \frac {\nu}{2} \cap \left| \widetilde {\boldsymbol {h}} ^ {\text {e n s}, \gamma} (\boldsymbol {x}) - \mu_ {\boldsymbol {x}} \right| <   \frac {\nu}{2}\right) (24) \\ = 1 - \mathbb {P} \left(\left| \boldsymbol {h} ^ {\text {e n s}, \boldsymbol {\lambda}} (\boldsymbol {x}) - \mu_ {\boldsymbol {x}} \right| \geq \frac {\nu}{2} \cup \left| \widetilde {\boldsymbol {h}} ^ {\text {e n s}, \gamma} (\boldsymbol {x}) - \mu_ {\boldsymbol {x}} \right| \geq \frac {\nu}{2}\right) (25) \\ \geq 1 - 4 \exp \left(\frac {- \nu^ {2} m}{2 c}\right), (26) \\ \end{array}
$$
where the first inequality comes from triangle inequality, the following from probability of subset of events  $(\mathbb{P}(A) \geq \mathbb{P}(B)$  if  $A \supseteq B$ ), the equality from taking complement, and the last line from applying McDiarmid's inequality along with the union bound.
Finally, applying the union bound on  $\mathcal{D}_{\mathrm{valid.}}$  with  $|\mathcal{D}_{\mathrm{valid.}}| = n$ , we obtain the bound
$$
\begin{array}{l} \mathbb {P} \left(\bigcap_ {\boldsymbol {x} \in \mathcal {D} _ {\text {v a l i d .}}} \left| \boldsymbol {h} ^ {\text {e n s}, \boldsymbol {\lambda}} (\boldsymbol {x}) - \widetilde {\boldsymbol {h}} ^ {\text {e n s}, \gamma} (\boldsymbol {x}) \right| <   \nu\right) = 1 - \mathbb {P} \left(\bigcup_ {\boldsymbol {x} \in \mathcal {D} _ {\text {v a l i d .}}} \left| \boldsymbol {h} ^ {\text {e n s}, \boldsymbol {\lambda}} (\boldsymbol {x}) - \widetilde {\boldsymbol {h}} ^ {\text {e n s}, \gamma} (\boldsymbol {x}) \right| \geq \nu\right) (27) \\ \geq 1 - 4 n \exp \left(\frac {- \nu^ {2} m}{2 c}\right), (28) \\ \end{array}
$$
The main idea is as follows: first observe that for the ensembled labels to disagree on a sample (given that the scores are bounded away from  $\frac{1}{2}$  with high probability), the two models need to produce scores in the range  $[0, \frac{1}{2} - \delta] \cup [\frac{1}{2} + \delta, 1]$ . This means that the scores need to deviate at least  $2\delta$  which has an exponentially low probability given Theorem 4.1.
$$
\mathbb {P} \left(\bigcup_ {\boldsymbol {x} \in \mathcal {D} _ {0}} f \left(\boldsymbol {h} ^ {\text {e n s}, \boldsymbol {\lambda}} (\boldsymbol {x})\right) \neq f \left(\widetilde {\boldsymbol {h}} ^ {\text {e n s}, \gamma} (\boldsymbol {x})\right)\right) \leq \left(4 e ^ {- 2 \delta^ {2} m / c} + 2 \theta\right) n _ {0}. \tag {29}
$$
Indeed, for each fixed  $\pmb{x} \in \mathcal{D}_0$ , we may reduce the failure probability to the case of separation of scores:
$$
\mathbb {P} \left(f \left(\boldsymbol {h} ^ {\text {e n s}, \boldsymbol {\lambda}} (\boldsymbol {x})\right) \neq f \left(\widetilde {\boldsymbol {h}} ^ {\text {e n s}, \gamma} (\boldsymbol {x})\right)\right) \leq \mathbb {P} \left(\left(\boldsymbol {h} ^ {\text {e n s}, \boldsymbol {\lambda}} (\boldsymbol {x}) \in \left[ 0, \frac {1}{2} - \delta \right] \cap \widetilde {\boldsymbol {h}} ^ {\text {e n s}, \gamma} (\boldsymbol {x}) \in \left[ \frac {1}{2} + \delta , 1 \right]\right) \right. \tag {30}
$$
$$
\cup \left(\widetilde {\boldsymbol {h}} ^ {\text {e n s}, \gamma} (\boldsymbol {x}) \in \left[ 0, \frac {1}{2} - \delta \right] \cap \boldsymbol {h} ^ {\text {e n s}, \lambda} (\boldsymbol {x}) \in \left[ \frac {1}{2} + \delta , 1 \right]\right) \tag {31}
$$
$$
\cup \boldsymbol {h} ^ {\text {e n s}, \boldsymbol {\lambda}} (\boldsymbol {x}) \in \left[ \frac {1}{2} - \delta , \frac {1}{2} + \delta \right] \tag {32}
$$
$$
\left. \cup \widetilde {\boldsymbol {h}} ^ {\text {e n s}, \gamma} (\boldsymbol {x}) \in \left[ \frac {1}{2} - \delta , \frac {1}{2} + \delta \right]\right) \tag {33}
$$
$$
\leq \mathbb {P} \left(\left| \boldsymbol {h} ^ {\text {e n s}, \boldsymbol {\lambda}} (\boldsymbol {x}) - \widetilde {\boldsymbol {h}} ^ {\text {e n s}, \gamma} (\boldsymbol {x}) \right| \geq 2 \delta\right) + 2 \theta \tag {34}
$$
$$
\leq 4 \exp \left(\frac {- 2 \delta^ {2} m}{c}\right) + 2 \theta . \tag {35}
$$
Finally, applying the union bound, we obtain that
$$
\mathbb {P} \left(\bigcap_ {\boldsymbol {x} \in \mathcal {D} _ {0}} f (\boldsymbol {h} ^ {\text {e n s}, \boldsymbol {\lambda}} (\boldsymbol {x})) = f (\widetilde {\boldsymbol {h}} ^ {\text {e n s}, \boldsymbol {\gamma}} (\boldsymbol {x}))\right) = 1 - \mathbb {P} \left(\bigcup_ {\boldsymbol {x} \in \mathcal {D} _ {0}} f (\boldsymbol {h} ^ {\text {e n s}, \boldsymbol {\lambda}} (\boldsymbol {x})) \neq f (\widetilde {\boldsymbol {h}} ^ {\text {e n s}, \boldsymbol {\gamma}} (\boldsymbol {x}))\right) \tag {36}
$$
$$
\geq 1 - \left(4 e ^ {- 2 \delta^ {2} m / c} + 2 \theta\right) n _ {0}, \tag {37}
$$
C Discussion on optimizing ensemble parameters
We have taken the weights  $\lambda \in \Delta_{m}$  which determines the ensembled model  $h^{\mathrm{ens},\lambda}$  to be fixed. We explain here how  $\lambda$  can be optimized according to a given cost. Specifically, given a loss function  $\ell :[0,1]\times \{0,1\} \to \mathbb{R}_{+}$ , we can search for the optimal  $\lambda \in \Delta_{m}$  that minimizes the total cost
$$
L _ {\mathrm {e n s}} (\boldsymbol {\lambda}) \triangleq \mathbb {E} \left[ \ell \left(\boldsymbol {h} ^ {\mathrm {e n s}, \boldsymbol {\lambda}} (X), Y\right) \right]. \tag {38}
$$
For the above optimization problem, we think of the constituent models  $h_1, \dots, h_m$  as being fixed and the randomness is from that of  $(X, Y)$ .
However, in practice, we have access to only samples  $(\pmb{x}_i,y_i)\sim P_{\pmb{X},Y}$ . Thus, we consider minimizing the regularized sample mean (for fixed  $\beta >0$ )
$$
\widehat {L} _ {\text {e n s}} (\boldsymbol {\lambda}) \triangleq \frac {1}{n} \sum_ {i \in [ n ]} \ell \left(\boldsymbol {h} ^ {\text {e n s}, \boldsymbol {\lambda}} \left(\boldsymbol {x} _ {i}\right), y _ {i}\right) + \frac {\beta}{\sqrt {n}} \| \boldsymbol {\lambda} \| _ {2} ^ {2}. \tag {39}
$$
The 2-norm regularization is added to facilitate proving convergence. This convergence result can be obtained via known results from statistical learning theory, e.g., using Theorem 13.2 in [22]. Specifically, consider the following two restrictions:
- Consider only  $\lambda \in \Delta_m$  satisfying  $\|\lambda\|_2 \leq \alpha$  for a prescribed  $\alpha$ . Note that we may take  $\alpha = 1$  to encapsulate the whole probability simplex. However, we may choose  $\lambda$  to be a slight modification of the uniform ensembling, in which case we would have  $\alpha$  of order  $1/\sqrt{m}$ .
- Assume that the function  $\lambda \mapsto \ell(h^{\mathrm{ens},\lambda}(\pmb{x}),y)$  is convex and  $A$ -Lipschitz for each fixed  $(\pmb{x},y)$ .
In this case, choosing  $\beta = A / \alpha$  and denoting the optimizers
$$
\boldsymbol {\lambda} ^ {(n)} \triangleq_ {\| \boldsymbol {\lambda} \| _ {2} \leq \alpha} \widehat {L} _ {\text {e n s}} (\boldsymbol {\lambda}), \tag {40}
$$
we can bound the utility of these minimizers by
$$
\mathbb {P} \left(L _ {\mathrm {e n s}} \left(\boldsymbol {\lambda} ^ {(n)}\right) \leq \inf  _ {\| \boldsymbol {\lambda} \| _ {2} \leq \alpha} L _ {\mathrm {e n s}} (\boldsymbol {\lambda}) + \frac {\beta \alpha^ {2}}{\sqrt {n}} \cdot \left(1 + \frac {1}{\delta} + \frac {8}{\delta \sqrt {n}}\right)\right) \geq 1 - \delta \tag {41}
$$
for any  $\delta \in (0,1)$ .
D Additional discussion on group fairness and fairness interventions
In addition to OAE (Definition 5), two other important fairness criteria are Statistical Parity [18] and Equalized Odds [23].
Definition 9 (SP).  $\operatorname*{Pr}(\widehat{Y} = 1|S = s) = \operatorname*{Pr}(\widehat{Y} = 1|S = s^{\prime})$  for all groups  $s,s^{\prime}\in [K]$
Definition 10 (EO).  $\operatorname*{Pr}(\widehat{Y} = 1|S = s,Y = b) = \operatorname*{Pr}(\widehat{Y} = 1|S = s',Y = y)$  for all groups  $s,s'\in [K]$ , and binary labels  $y\in \{0,1\}$ .
Essentially, SP requires the predicted label  $\widehat{Y} \triangleq \arg \max h(\mathbf{X})$  to be independent of the group membership  $S$  [19]. In comparison, EO conditions on both group and the true label [23]. EO improves upon SP in the sense that it does not rule out the perfect classifiers whenever the true label  $Y$  is correlated with the group membership  $S$  [1]. In practice, we quantify EO violation by measuring Mean EO as in Equation 6 (for two groups) and, more generally, in Equation 42 below (beyond two groups). Similarly, we can measure SP violation as in Equation 43.
$$
\text {M E A N E O} \triangleq \max  _ {s, s ^ {\prime} \in [ K ]} \frac {1}{2} \left(| \mathrm {T P R} _ {S = s} - \mathrm {T P R} _ {S = s ^ {\prime}} | + | \mathrm {F P R} _ {S = s} - \mathrm {F P R} _ {S = s ^ {\prime}} |\right). \tag {42}
$$
$$
\operatorname {S P} \text {V I O L A T I O N} \triangleq \max  _ {s, s ^ {\prime} \in [ K ]} \frac {1}{2} \left(| \Pr (\widehat {Y} = 1 | S = s) - \Pr (\widehat {Y} = 1 | S = s ^ {\prime}) |\right). \tag {43}
$$
Next, we offer a brief discussion of various intervention mechanisms used in this paper. The fairness interventions can be categorized into two categories: in-processing and post-processing. In-processing mechanisms incorporate fairness constraints during training. It usually add the fairness constraint to the loss function and outputs a fair classifier. Post-processing mechanisms treat the model as a black box and update its predictions to achieve the desirable fairness constraints [10].
REDUCTION [1], short for exponentiated gradient reduction, is an in-processing technique that reduces fair classification to a sequence of cost-sensitive classification problems, and yields a randomized classifier with the lowest empirical error subject to the desired constraints. This technique achieves fairness with a minimal decrease in accuracy, but it is computationally expensive since it requires re-training multiple models.
REJECT OPTION CLASSIFIER [29] is a postprocessing technique that achieves fairness constraints by modifying outcomes of samples in a confidence band of the decision boundary with the highest uncertainty. It gives favorable outcomes to unprivileged groups and unfavorable outcomes to privileged groups. It outputs a thresholded prediction rather than a probability over the binary labels.
EQODDS [23] is a post-processing technique that formulates empirical risk minimization with fairness constraint as a linear program and modifies predictions according to the derived probabilities to achieve equalized odds.