SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fangxin Wang
Chicago, USA
fwang51@uic.edu
Lu Cheng
lucheng@uic.edu
Ruocheng Guo
ByteDance Research
rguo.asu@gmail.com
zliu234@uic.edu
Philip S. Yu
psyu@uic.edu
We study fair machine learning (ML) under predictive uncertainty to enable reliable and trustworthy decision-making. The seminal work of "equalized coverage" proposed an uncertainty-aware fairness notion. However, it does not guarantee equal coverage rates across more fine-grained groups (e.g., low-income females) conditioning on the true label and is biased in the assessment of uncertainty. To tackle these limitations, we propose a new uncertainty-aware fairness - Equal Opportunity of Coverage (EOC) - that aims to achieve two properties: (1) coverage rates for different groups with similar outcomes are close, and (2) the coverage rate for the entire population remains at a predetermined level. Further, the prediction intervals should be narrow to be informative. We propose Binned Fair Quantile Regression (BFQR), a distribution-free post-processing method to improve EOC with reasonable width for any trained ML models. It first calibrates a hold-out set to bound deviation from EOC, then leverages conformal prediction to maintain EOC on a test set, meanwhile optimizing prediction interval width. Experimental results demonstrate the effectiveness of our method in improving EOC. Our code is publicly available at https://github.com/fangxin-wang/bfqr.
Machine Learning (ML) can bring bias and discrimination even with good intentions [123456]. Fair ML has been developed to counteract unfairness, but the practical use of fair ML models is limited by predictive uncertainty. Predictive uncertainty is the extent to which ML can confidently predict the future. Over- or under-confidence can cause an ML model to be unaware of its own knowledge gaps and make inaccurate predictions [789]. This can lead to unfairness in decision-making. To address this, we can produce predicted intervals for each sample and incorporate uncertainty into fairness to make decisions more reliable and trustworthy.
The idea of "equalized coverage" - an uncertainty-aware notion of demographic parity [10] - was introduced in a study [11] as a way to ensure that every group receives the same level of prediction certainty. It works by generating prediction intervals that cover the true label  $Y$  with a specified probability (e.g., 90%), while also reflecting uncertainty through interval width. However, even with this approach, there are still disparities in coverage rates across groups when conditioning on  $Y$ . For example, we observe from the empirical results (Fig. 1) for Adult dataset [4] that low-income women are less likely to be covered than men in the same income bracket, and high-income men are less likely to be predicted to earn as much as high-income women. Consequently, the widths of prediction intervals for different groups, as an indicator of the uncertainty, are not comparable under
different coverage rates. These disparities can lead to unfair risk assessment for domains like bank loans and taxation. Further, equalized coverage may sacrifice the efficiency of uncertainty estimation for ensuring coverage rate as it produces wider prediction intervals for the group it intends to protect (See Section 5).
Figure 1: Evaluating equalized coverage [11] on the Adult dataset. The protected attribute is gender. The test data is partitioned into 10 equal-mass bins based on the ascending order of salaries. Significant variations in the coverage rates can be observed among different groups within the head and tail bins.
To address the limitations of equalized coverage, we propose a novel uncertainty-based fairness notion Equal Opportunity of Coverage (EOC), extending from the standard fairness notion of equal opportunity [12]. EOC aims to achieve two properties: (1) similar coverage rates for different groups (e.g., female and male) with similar outcomes  $Y$  (e.g., salaries), (2) achieve a desired level of coverage rate (e.g., 90%) for the entire population. Ideally, to provide informative predictions, intervals should be as narrow as possible while still satisfying EOC.
We consider the regression task as it is a more general fair ML setting [13, 14] with minimal assumptions about the underlying data distribution.
Achieving EOC confronts various challenges. Firstly, the majority of prior fair ML approaches are developed for classification problems [12, 15], with only a few for regression [14, 16]. This necessitates developing effective techniques to measure and improve EOC in a regression setting. Secondly, ensuring EOC and marginal coverage rate for test data is difficult when true labels are unknown. Finally, prior works [11, 17, 18, 19] have primarily focused on improving fairness and satisfying coverage rate guarantees, but often neglect the width of prediction intervals during the optimization process. This often results in generating wider intervals that limit the amount of decision-making information available. However, optimizing interval width with fairness constraints is a non-convex problem, and coverage rates are difficult to guarantee on noisy data, making it challenging to consider all these factors simultaneously.
To address these challenges, we propose Binned Fair Quantile Regression (BFQR), a distribution-free post-processing method that improves EOC while maintaining a desired marginal coverage rate and a narrow prediction interval. It consists of three major steps: First, a hold-out (calibration) dataset is calibrated to improve EOC based on true label  $Y$  within discretized bins; second, we leverage conformal prediction [20, 21] to achieve EOC for test data from the calibration results in the first step; and finally, an efficient and robust optimization technique is developed to minimize the mean width of prediction intervals. Experiments on both synthetic and real-world data show that BFQR is more effective in improving EOC fairness than state-of-the-art methods.
Related Work. In regression settings, equal opportunity [12] has been studied mostly by adversarial training [22, 23]. For quantile regression, models with equal opportunity constraint [24, 25] are proposed. However, all mentioned are in-processing methods and have trade-offs between accuracy and fairness [26, 27]. There are several works on uncertainty-aware fairness in regression. The pioneering work equalized coverage [11], is built on the validity of conditional coverage [28], and has several follow-ups [17, 18]. [19] also proposes conformalized fair regression, imposing demographic parity fairness on prediction interval bounds. However, neither of these methods considers the fine-grained group (e.g., low-income females) fairness conditional on true labels as well as the increased width of prediction intervals that provide little information.
Conformal Prediction extends traditional ML by providing a set of prediction intervals around the predicted value, which can be used to assess the level of confidence or uncertainty in the prediction [20]. It is distribution-free and has a rigorous statistical guarantee. A commonly used approach is split conformal prediction [20, 21]. To start, the training data is divided into two sets: the training set  $\mathbb{D}_{tr}$  and the calibration set  $\mathbb{D}_c = \{(X_1,Y_1),\dots,(X_n,Y_n)\}$ . A prediction model  $\hat{f}$  is trained on the training set  $\mathbb{D}_{tr}$ . The key ingredient is the conformity score function  $S(x,y)\in \mathbb{R}$  used to evaluate the model's prediction performance. Given a desired error rate  $\alpha$ , it then calculates the quantile  $\hat{Q}_{1 - \alpha}(S,\mathbb{D}_c)$ , denoting the  $(1 - \alpha)(1 + 1 / |\mathbb{D}_c|)$ -th quantile of the empirical distribution of  $S$  on  $\mathbb{D}_c$ . Finally, for a sample  $X_{n + 1}$  in the test set  $\mathbb{D}_t$ , its prediction set is  $C^S (X_{n + 1}) = \{y:S(X_{n + 1},y)\leq \hat{Q}_{1 - \alpha}(S,\mathbb{D}_c)\}$ . This set contains all possible values of  $y$  for which the conformity score  $S(X_{n + 1},y)$  is less than or equal to the calculated quantile. Given a mild assumption that the test and calibration set are exchangeable, the coverage rate in conformal prediction is guaranteed with a probability of  $P(Y_{n + 1}\in C^{S}(X_{n + 1}))\geq 1 - \alpha$ .
Equalized Coverage. Let  $V \in \mathcal{V}$  be the indicator of whether  $Y$  is covered in the prediction set  $\hat{C}(X)$ , i.e.,  $V = \mathbb{1}[Y \in \hat{C}(X)]$ .  $\mathcal{V} = \{0, 1\}$ . Given a desired error rate  $\alpha \in [0, 1]$ , equalized coverage is satisfied [11] when  $\forall a \in A$ ,  $Pr\{V | A = a\} = Pr\{V\} \geq 1 - \alpha$  and  $Pr\{V\} \geq 1 - \alpha$ . Equalized coverage guarantees equal conditional coverage, i.e.,  $V$  is independent of  $A$  (denoted as  $V \perp A$ ), but fails to ensure equal coverage conditional on  $Y$ , especially for extreme values of  $Y$  (Fig. 1). This is problematic as it can perpetuate discrimination against marginalized groups (e.g., females with low income) in risk assessment.
3 Equal Opportunity of Coverage
Equal Opportunity of Coverage addresses the limitation of equalized coverage and is defined based on the equal opportunity [12]:
Definition 3.1 (Equal Opportunity of Coverage (EOC)). EOC is satisfied when  $Pr\{V|A = a, Y = y\} = Pr\{V|Y = y\}$ , i.e.,  $V \perp A|Y$  and  $Pr\{V\} \geq 1 - \alpha, \forall y \in Y$  and  $\forall a \in A$ .
The definition of EOC requires that (1) conditioned on the target variable  $Y$ , whether a sample is covered in its prediction interval should be independent of its sensitive attribute  $A$ ; (2) the marginal coverage rate is above the desired level. Interestingly, the difference between equalized coverage and EOC in their mathematical forms share similarities with the difference between demographic parity and equalized odds. Note that our focus here is whether the true label is covered in the prediction interval since even though the prediction interval contains false labels,  $V = \mathbb{I}[Y \in \hat{C}(X)] = 1$  is still valid. Therefore, while EOC has a similar formation to equalized odds, it in fact describes equal opportunity which focuses on  $Y = 1$ .
Preferably, both EOC and equalized coverage should be guaranteed. However, the mutual exclusivity theorem below suggests that there is an inherent trade-off between EOC and equalized coverage:
Theorem 3.1 (Mutual Exclusivity). If  $A \not\perp Y$  and  $V \not\perp Y$ , then either equalized coverage or equal opportunity of coverage holds but not both.
Proof. If  $V \perp \perp A$  and  $V \perp \perp A|Y$ , then either  $A \perp Y$  or  $V \perp Y$ .
Unfairness often arises from the fact that features predictive of  $Y$  are also correlated to the protected attribute due to e.g., historical bias in the data [29]. This indicates  $V \perp A$ . For the second condition, though  $V \nparallel Y$  is possible,  $V$  needs to depend on  $Y$  to ensure prediction intervals with reasonable width. Predicting certain values (e.g., extreme values) of  $Y$  can be challenging due to representative bias. If we enforce the predictor to provide high coverage for marginalized groups with these values, it is highly likely to result in extremely wide intervals that offer little guidance in decision-making.
Measuring EOC. Given the underlying distribution  $p \in \Delta(\mathcal{V} \times \mathcal{A} \times \mathcal{Y})$ , where  $\mathcal{V}, \mathcal{A}, \mathcal{Y}$  is the domain of  $v, a, y$  respectively. We can determine how likely  $p$  satisfies EOC by measuring its distance to  $p'$ , the closest distribution that perfectly achieves EOC. Formally, we denote  $P_{EOC}$  as the property of EOC, the set of all distributions in defined space that satisfy EOC, i.e.,  $P_{EOC} := \{p \in \Delta(\mathcal{V} \times \mathcal{A} \times \mathcal{Y}) : (V, A, Y) \sim p, V \perp A|Y\}$ .  $p' \in P_{EOC}$  is the distribution with minimum total variation (TV) distance to  $p$ , i.e.,  $\forall q \in P_{EOC}$ ,  $d_{TV}(p, p') \leq d_{TV}(p, q)$ . The TV distance between  $p$  and  $p'$  is
formally defined as  $d_{TV}(p,p^{\prime}) = \frac{1}{2}\sum_{(v,a,y)\in \Delta (\mathcal{V}\times \mathcal{A}\times \mathcal{Y})}|p(v,a,y) - p^{\prime}(v,a,y)| = \frac{1}{2}\| p - p^{\prime}\|_{1},$  where  $\| \cdot \| _1$  denotes the  $l_{1}$  norm of a distribution. Instead of the commonly-used Kolmogorov-Smirnov (KS) distance [12, 30], TV distance is chosen as the measure due to the significant drawback of the KS distance - its insensitivity to the deviations between  $p$  and  $p^\prime$  at the tails [31]. Whereas in real-world data, we often confront such significant deviation (see e.g., Fig. 1).
It is challenging to directly measure the distance between  $p$  and  $p'$  from observed data when the target variable  $Y$  is continuous due to data sparsity issues for each value of  $Y$ . However, drawing from previous works [32, 33], we can instead use an easy-to-compute statistic  $T$  as a surrogate of  $d_{TV}(p,p')$ .  $T$  is the measure of independence, assessing the weighted summed violation of independence within each discretized bin. The expectation of  $T$ ,  $\mathbb{E}[T]$ , has a fixed upper bound when  $p$  satisfies EOC, and a lower bound that increases with the TV distance between  $p$  and  $p'$ . The intuition here is that when  $Y$  is divided into sufficient bins and  $p_Y \coloneqq Pr\{V,A|Y\}$ , the conditional distribution of  $p$  on  $Y$ , is Lipschitz continuous, enforcing independence within each bin generates a distribution that does not deviate far from  $P_{EOC}$  in terms of TV distance.
Lemma 3.2. [33] Let  $d = \lceil n^{2/5} \rceil$  be the number of bins  $Y$  is discretized into, with each bin having an equal size of samples. Let  $T$  be a sum of independence measures within all discretized bins. Given the Lipschitz continuity of  $p_Y$  in Assumption 2,  $L$  is the Lipschitz constant, we have:
1) When  $p$  satisfies EOC,  $\mathbb{E}[T] \leq \frac{|\mathbb{D}|L^2}{d^2}$ ;
2) When  $d_{TV}(p, p') = \epsilon$ , there exists a constant  $Z$  such that  $\mathbb{E}[T] \geq Z(\epsilon - 3\frac{L}{d})^2$ .
Lemma 3.2 indicates that we can approximately evaluate EOC for  $p$  by measuring the independence within each bin. If  $V \perp A$  almost holds in each bin, such that  $\mathbb{E}[T]$  is small enough, then it is highly possible that  $V \perp A|Y$ , i.e.,  $p$  satisfies EOC. Moreover, an increasing function of  $\epsilon$  is upper bounded by  $\mathbb{E}[T]$ . As  $\mathbb{E}[T]$  increases, it is highly likely that  $p$  is further away from any  $p'$  that satisfies EOC. This lemma serves as a theoretical foundation for the proposed method below to improve EOC and for adopting  $T$  as an evaluation metric in experiments.
In order to calculate  $\mathbb{E}[T]$  from data, we introduce unbiased estimators of  $T$  in Appendix 8.1.1. According to the central limit theorem, we could estimate  $\mathbb{E}[T]$  through a sufficient number of random samplings. Whereas for the sake of efficiency, we prefer to construct  $T$  with bounded variance such that  $\mathbb{E}[T]$  could converge through limited repeated samplings.
4 Improving Equal Opportunity of Coverage
In this section, we introduce a post-processing approach where we have a trained ML model, a calibration dataset, and a test dataset for which we aim to improve EOC. It consists of three steps. First, we enforce the independence of  $A$  and  $V$  within each interval of  $Y$  (i.e., EOC) for the calibration data, and then leverage conformal prediction to achieve EOC for test data. Lastly, we describe an efficient and robust optimization approach that optimizes both EOC and widths of prediction intervals.
4.1 Improving EOC on calibration data
The first step aims to improve EOC on the calibration data where we have ground-truth labels. As the target variable  $Y$  is continuous, the number of samples with certain values of  $Y$  can be extremely small, thus calibrating for each distinct value of  $Y$  is almost impossible. Meanwhile, commonly used methods designed for continuous variables such as adversarial learning [23, 22], which intend to learn a near-optimal  $p$ , are computationally inefficient for post-processing approaches [34].
According to Lemma 3.2,  $\mathbb{E}[T]$  is an upper bound of an increasing function of  $d_{TV}(p,p')$ . Thus, this indicates that if  $\mathbb{E}[T]$  decreases, the maximum distance between  $p$  and  $p'$  is reduced, resulting in an improvement in EOC. Informed by this idea, we first focus on enhancing EOC on the calibration data  $\mathbb{D}_c$ . For simplicity, we employ the framework introduced in [35] as the conformal prediction model, though our post-processing method is applicable to any base model. Specifically, let  $\hat{q}_{\alpha}$  denotes the  $\alpha$ -th conditional quantile regression function, i.e., for  $i$ -th sample  $(X_i,Y_i)$ ,  $\hat{q}_{\alpha}(X_i) \coloneqq \inf \{y \in \Delta \mathcal{Y} : Pr\{Y_i \leq y|X = X_i\} \geq \alpha\}$ . Fix the lower and upper quantiles as  $\alpha_{lo} = \alpha / 2$  and  $\alpha_{hi} = 1 - \alpha / 2$ , then  $\hat{q}_{\alpha_{lo}}(X_i)$  and  $\hat{q}_{\alpha_{hi}}(X_i)$  denote lower and upper quantile regression functions, respectively. The base model is trained on the training data and used for inference on calibration and test data.
Following the discretization idea in Lemma 3.2, we divide the continuous variable  $Y \in [Y_{min}, Y_{max}]$  in  $\mathbb{D}_c$  into  $M$  bins with equal sample sizes, and the  $m$ -th bin is denoted as  $B_m = [Y_m^-, Y_m^+)$ .
To enhance EOC on  $\mathbb{D}_c$ , we can minimize  $\mathbb{E}[T]$  by enforcing the independence of  $A$  and  $V$  within all discretized bins. In particular, we fix the coverage rates within each bin as  $\beta_m \in [0,1]$ , therefore  $V \perp A|Y \in B_m$ . Note that, although the coverage rates are equal across groups within the same bin, the quantile value at  $\beta_m$  are computed separately for each group, as illustrated in Figure[2]. We first calculate the vanilla prediction intervals  $\hat{C}(X_i) = [\hat{q}_{\alpha_{lo}}(X_i), \hat{q}_{\alpha_{hi}}(X_i)]$  obtained from the trained model, and get the conformity score  $S(X_i, Y_i) = \max(\hat{q}_{\alpha_{lo}}(X_i) - Y_i, Y_i - \hat{q}_{\alpha_{hi}}(X_i))$  for all samples in  $\mathbb{D}_c$ . Then for different combinations of bins and protect groups, we calculate the quantile value of conformity scores at coverage rate  $\beta_m$ ,  $\hat{Q}_{\beta}(S, \mathbb{D}_c(a, m))$ , for data in  $\mathbb{D}_c(a, m) = \{i | i \in \mathbb{D}_c, A_i = a, Y_i \in B_m\}$ . To simplify notations, we substitute  $\hat{Q}_{\beta}(S, \mathbb{D}_c(a, m))$  by  $G_{a,m}(\beta_m)$ . Since each bin has an equal sample size and coverage rate  $\beta_m$ , the average coverage rate  $\sum_{m} \beta_m / M$  is then set to  $1 - \alpha$  to keep a coverage rate of  $1 - \alpha$  on the calibration data.
(a) Bin 1
(b) Bin 2
Figure 2: Empirical Cumulative Distribution Functions (ECDF) of conformity scores of men (blue) and women (red) within bins 1,2 and 10 on the Adult dataset. The desired coverage rates within different bins could be set at different levels but should be equal across groups within the same bin, as indicated by the overlapping horizontal dashed lines. Due to the distinct disparity between the ECDF of men and women, the same coverage rate is mapped to different quantile values on the x-axis.
(c) Bin 10
Through the reconstruction of prediction intervals  $C(X_{i})$  with  $G_{a,m}(\beta_m)$ , i.e.,  $\forall i\in \mathbb{D}_c(a,m)$ ,  $C(X_{i}) = [\hat{q}_{\alpha_{l o}}(X_{i}) - G_{a,m}(\beta_{m}),\hat{q}_{\alpha_{h i}}(X_{i}) + G_{a,m}(\beta_{m})]$ , EOC is enhanced and the marginal coverage rate is guaranteed on the calibration data.
4.2 For Coverage and Independence Guarantees
This subsection seeks to preserve EOC and marginal coverage rate for test data based on results for calibration data through conformal prediction. The key is to find out, for a test sample with  $A_{n + 1} = a_{n + 1}$ , which bin  $B_{m}$  it belongs to. We can then calibrate the test sample with the quantile  $G_{a,m}(\beta)$ . However, direct calibration based on the predicted bin would not improve EOC since the prediction result can be biased due to, e.g., skewed distributions for different groups [29].
To address this issue, we propose Binned Fair Quantile Regression (BFQR) (see Algorithm 1 in Appendix 8.2.1). Our method could be treated as a variant of Mondrian conformal prediction [36], where the confidence in each bin is evaluated independently. Suppose that a new data point with feature  $X_{i}$  and protected attribute  $a$  falls into a certain bin  $B_{m}$ , we calibrate it with the corresponding quantile value  $G_{a,m}(\beta_m)$ . Then, we obtain a sub-interval of prediction within bin  $B_{m}$ , i.e.,  $C_m(X_i) = B_m \cap [\hat{q}_{\alpha_{l_o}}(X_i) - G_{a,m}(\beta_m), \hat{q}_{\alpha_{hi}}(X_i) + G_{a,m}(\beta_m)]$ . After computing  $C_m(X_i)$ , a union of all subsets  $C(X_i) = \bigcup_{m} C_m(X_i)$  is then the prediction interval of  $X_{i}$ . Under a mild exchangeability assumption similar to [11], our algorithm provides both the marginal coverage guarantee and fair coverage guarantees within each bin.
Assumption 1 (Exchangeability). All calibration data  $(X_{i},Y_{i})$ ,  $i = 1,\dots,n$  and a sample of test data  $(X_{n + 1},Y_{n + 1})$  are exchangeable conditioned on  $A_{n + 1} = a$  and  $Y_{n + 1}\in B_m$ , and conformity scores  $\{S(X_i,Y_i),i\in \mathbb{D}_c(a,m)\cup \{n + 1\} \}$  are almost surely distinct.
Theorem 4.1 (Bin Coverage and Independence Guarantee). Under Assumption  $\square$ $\beta_{m} \leq Pr\{Y_{n+1} \in C(X_{n+1}) | A_{n+1} = a, Y_{n+1} \in B_{m}\} \leq \beta_{m} + 1 / (|\mathbb{D}_{c}(a, m)| + 1)$ . The expectation of max coverage gap inside  $m$ -th bin is upper bounded by  $\max_{a} \{1 / (|\mathbb{D}_{c}(a, m)| + 1)\}$ .
Theorem 4.2 (Marginal Coverage Guarantee). If we have  $\sum_{m} \beta_{m} / M = 1 - \alpha$ , under Assumption I then  $Pr\{Y_{n+1} \in C(X_{n+1})\} \geq 1 - \alpha$ .
Here we sketch the proofs. Take  $(X_{i},A_{i},Y_{i})$  as new sample from test data,  $i\in \mathbb{D}_t$ . Suppose  $Y_{i}\in B_{m}$ , then the  $p$ -value of the null hypothesis  $Y_{i}\in B_{m}$  is given by  $\hat{u}_{A_i,m} = \frac{1 + |j\in\mathbb{D}_c(A_i,m):S(X_j,Y_j)\leq S(X_i,Y_i)|}{1 + |\mathbb{D}_c(A_i,m)|}$ , which is the proportion of conformity scores that are less than the score of the new sample among all calibration data that have the same protected attribute and fall into the same bin. The prediction interval  $\hat{C}_m(X_i)$  in  $B_{m}$  is the intersection of  $B_{m}$  and the prediction interval calibrated by  $G_{a,m}(\beta_m)$ , which includes the part of  $B_{m}$  in the prediction interval where the  $p$ -value  $\hat{u}_{A_i,m}$  is greater than  $\beta_{m}$ . This guarantees the bin coverage of  $B_{m}$  at level  $\beta_{m}$  in Theorem 4.1. The complete proofs can be found in Appendix 8.2.2.
4.3 Constrained Optimization
With improved EOC and coverage guarantee, we now need to identify  $\beta_{m}$  for each bin so that the mean width of prediction intervals for  $\mathbb{D}_t$  is the smallest. As such, our goal is to solve a constrained optimization problem, where the decision variables are the coverage rates in each bin  $\beta_{m}$ ,  $m = 1,\dots ,M$ , and the objective function is the mean width of prediction intervals in test data:
$$
\min  \sum_ {m, i \in | \mathbb {D} _ {t} |} | C _ {m} (X _ {i}) | / | \mathbb {D} _ {t} |
$$
$$
\begin{array}{l} m, t \in | \mathbb {D} _ {t} | \\ \text {s . t .} \quad \left\{ \begin{array}{l} \sum_ {m} \beta_ {m} / M = 1 - \alpha , \\ \beta_ {m} \in [ 0, 1 ], \quad m = 1, \dots , M. \end{array} \right. \end{array} \tag {1}
$$