Title: Equal Opportunity of Coverage in Fair Regression
Venue: NeurIPS 2023
Accepted: yes
Keywords: Equal Opportunity; Fair Machine Learning; Conformal Prediction; Uncertainty Quantification
Abstract: We study fair machine learning (ML) under predictive uncertainty to enable reliable and trustworthy decision-making. The seminal work of 'equalized coverage' proposed an uncertainty-aware fairness notion. However, it does not guarantee equal coverage rates across more fine-grained groups (e.g., low-income females) conditioning on the true label and is biased in the assessment of uncertainty. To tackle these limitations, we propose a new uncertainty-aware fairness -- Equal Opportunity of Coverage (EOC) -- that aims to achieve two properties: (1) coverage rates for different groups with similar outcomes are close, and (2) the coverage rate for the entire population remains at a predetermined level. Further, the prediction intervals should be narrow to be informative. We propose Binned Fair Quantile Regression (BFQR), a distribution-free post-processing method to improve EOC with reasonable width for any trained ML models. It first calibrates a hold-out set to bound deviation from EOC, then leverages conformal prediction to maintain EOC on a test set, meanwhile optimizing prediction interval width. Experimental results demonstrate the effectiveness of our method in improving EOC.

Equal Opportunity of Coverage in Fair Regression

Abstract

We study fair machine learning (ML) under predictive uncertainty to enable reliable and trustworthy decision-making. The seminal work of 'equalized coverage' proposed an uncertainty-aware fairness notion. However, it does not guarantee equal coverage rates across more fine-grained groups (e.g., low-income females) conditioning on the true label and is biased in the assessment of uncertainty. To tackle these limitations, we propose a new uncertainty-aware fairness - Equal Opportunity of Coverage (EOC) - that aims to achieve two properties: (1) coverage rates for different groups with similar outcomes are close, and (2) the coverage rate for the entire population remains at a predetermined level. Further, the prediction intervals should be narrow to be informative. We propose Binned Fair Quantile Regression (BFQR), a distribution-free post-processing method to improve EOC with reasonable width for any trained ML models. It first calibrates a hold-out set to bound deviation from EOC, then leverages conformal prediction to maintain EOC on a test set, meanwhile optimizing prediction interval width. Experimental results demonstrate the effectiveness of our method in improving EOC. Our code is publicly available at https://github.com/fangxin-wang/bfqr .

1 Introduction

Machine Learning (ML) can bring bias and discrimination even with good intentions [1, 2, 3, 4, 5, 6]. Fair ML has been developed to counteract unfairness, but the practical use of fair ML models is limited by predictive uncertainty. Predictive uncertainty is the extent to which ML can confidently predict the future. Over- or under-confidence can cause an ML model to be unaware of its own knowledge gaps and make inaccurate predictions [7, 8, 9]. This can lead to unfairness in decision-making. To address this, we can produce predicted intervals for each sample and incorporate uncertainty into fairness to make decisions more reliable and trustworthy.

The idea of 'equalized coverage' - an uncertainty-aware notion of demographic parity [10] - was introduced in a study [11] as a way to ensure that every group receives the same level of prediction certainty. It works by generating prediction intervals that cover the true label Y with a specified probability (e.g., 90%), while also reflecting uncertainty through interval width. However, even with this approach, there are still disparities in coverage rates across groups when conditioning on Y . For example, we observe from the empirical results (Fig. 1) for Adult dataset [4] that low-income women are less likely to be covered than men in the same income bracket, and high-income men are less likely to be predicted to earn as much as high-income women. Consequently, the widths of prediction intervals for different groups, as an indicator of the uncertainty, are not comparable under

different coverage rates. These disparities can lead to unfair risk assessment for domains like bank loans and taxation. Further, equalized coverage may sacrifice the efficiency of uncertainty estimation for ensuring coverage rate as it produces wider prediction intervals for the group it intends to protect (See Section 5).

Coverageratesformenandwomenwithsalariesindifferentbins

1.0 -

Men

Rate

0.9

Women

0.8

Coverage!

0.7

0.6

0.5 -

0.4 -

Bin5

Bin10

Salary

To address the limitations of equalized coverage, we propose a novel uncertainty-based fairness notion Equal Opportunity of Coverage (EOC) , extending from the standard fairness notion of equal opportunity [12]. EOC aims to achieve two properties: (1) similar coverage rates for different groups (e.g., female and male) with similar outcomes Y (e.g., salaries), (2) achieve a desired level of coverage rate (e.g., 90%) for the entire population. Ideally, to provide informative predictions, intervals should be as narrow as possible while still satisfying EOC.

We consider the regression task as it is a more general fair ML setting [13, 14] with minimal assumptions about the underlying data distribution.

Achieving EOC confronts various challenges. Firstly, the majority of prior fair ML approaches are developed for classification problems [12, 15], with only a few for regression [14, 16]. This necessitates developing effective techniques to measure and improve EOC in a regression setting. Secondly, ensuring EOC and marginal coverage rate for test data is difficult when true labels are unknown. Finally, prior works [11, 17, 18, 19] have primarily focused on improving fairness and satisfying coverage rate guarantees, but often neglect the width of prediction intervals during the optimization process. This often results in generating wider intervals that limit the amount of decisionmaking information available. However, optimizing interval width with fairness constraints is a non-convex problem, and coverage rates are difficult to guarantee on noisy data, making it challenging to consider all these factors simultaneously.

To address these challenges, we propose Binned Fair Quantile Regression (BFQR), a distribution-free post-processing method that improves EOC while maintaining a desired marginal coverage rate and a narrow prediction interval. It consists of three major steps: First, a hold-out (calibration) dataset is calibrated to improve EOC based on true label Y within discretized bins; second, we leverage conformal prediction [20, 21] to achieve EOC for test data from the calibration results in the first step; and finally, an efficient and robust optimization technique is developed to minimize the mean width of prediction intervals. Experiments on both synthetic and real-world data show that BFQR is more effective in improving EOC fairness than state-of-the-art methods.

Related Work. In regression settings, equal opportunity [12] has been studied mostly by adversarial training [22, 23]. For quantile regression, models with equal opportunity constraint [24, 25] are proposed. However, all mentioned are in-processing methods and have trade-offs between accuracy and fairness [26, 27]. There are several works on uncertainty-aware fairness in regression. The pioneering work equalized coverage [11], is built on the validity of conditional coverage [28], and has several follow-ups [17, 18]. [19] also proposes conformalized fair regression, imposing demographic parity fairness on prediction interval bounds. However, neither of these methods considers the finegrained group (e.g., low-income females) fairness conditional on true labels as well as the increased width of prediction intervals that provide little information.

2 Preliminary

Conformal Prediction extends traditional ML by providing a set of prediction intervals around the predicted value, which can be used to assess the level of confidence or uncertainty in the prediction [20]. It is distribution-free and has a rigorous statistical guarantee. A commonly used approach is split conformal prediction [20, 21]. To start, the training data is divided into two sets: the training set D tr and the calibration set D c = { ( X 1 , Y 1 ) , ..., ( X n , Y n ) } . A prediction model ˆ f is trained on the training set D tr . The key ingredient is the conformity score function S ( x, y ) ∈ R used to evaluate the model's prediction performance. Given a desired error rate α , it then calculates the quantile ˆ Q 1 -α ( S, D c ) , denoting the (1 -α )(1 + 1 / | D c | ) -th quantile of the empirical distribution of S on D c . Finally, for a sample X n +1 in the test set D t , its prediction set is C S ( X n +1 ) = { y : S ( X n +1 , y ) ≤ ˆ Q 1 -α ( S, D c ) } . This set contains all possible values of y for which the conformity score S ( X n +1 , y ) is less than or equal to the calculated quantile. Given a mild assumption that the test and calibration set are exchangeable, the coverage rate in conformal prediction is guaranteed with a probability of P ( Y n +1 ∈ C S ( X n +1 )) ≥ 1 -α .

Equalized Coverage. Let V ∈ V be the indicator of whether Y is covered in the prediction set ˆ C ( X ) , i.e., V = [ Y ∈ ˆ C ( X )] . V = { 0 , 1 } . Given a desired error rate α ∈ [0 , 1] , equalized coverage is satisfied [11] when ∀ a ∈ A , Pr { V | A = a } = Pr { V } ≥ 1 -α and Pr { V } ≥ 1 -α . Equalized coverage guarantees equal conditional coverage, i.e., V is independent of A (denoted as V ⊥ ⊥ A ), but fails to ensure equal coverage conditional on Y , especially for extreme values of Y (Fig. 1). This is problematic as it can perpetuate discrimination against marginalized groups (e.g., females with low income) in risk assessment.

3 Equal Opportunity of Coverage

Equal Opportunity of Coverage addresses the limitation of equalized coverage and is defined based on the equal opportunity [12]:

Definition 3.1 (Equal Opportunity of Coverage (EOC)) . EOC is satisfied when Pr { V | A = a, Y = y } = Pr { V | Y = y } , i.e., V ⊥ ⊥ A | Y and Pr { V } ≥ 1 -α , ∀ y ∈ Y and ∀ a ∈ A .

The definition of EOC requires that (1) conditioned on the target variable Y , whether a sample is covered in its prediction interval should be independent of its sensitive attribute A ; (2) the marginal coverage rate is above the desired level. Interestingly, the difference between equalized coverage and EOC in their mathematical forms share similarities with the difference between demographic parity and equalized odds. Note that our focus here is whether the true label is covered in the prediction interval since even though the prediction interval contains false labels, V = I [ Y ∈ ˆ C ( X )] = 1 is still valid. Therefore, while EOC has a similar formation to equalized odds, it in fact describes equal opportunity which focuses on Y = 1 .

Preferably, both EOC and equalized coverage should be guaranteed. However, the mutual exclusivity theorem below suggests that there is an inherent trade-off between EOC and equalized coverage:

Theorem 3.1 (Mutual Exclusivity) . If A /negationslash ⊥ ⊥ Y and V /negationslash ⊥ ⊥ Y , then either equalized coverage or equal opportunity of coverage holds but not both.

Proof. If V ⊥ ⊥ A and V ⊥ ⊥ A | Y , then either A ⊥ ⊥ Y or V ⊥ ⊥ Y .

Unfairness often arises from the fact that features predictive of Y are also correlated to the protected attribute due to e.g., historical bias in the data [29]. This indicates V ⊥ ⊥ A . For the second condition, though V /negationslash ⊥ ⊥ Y is possible, V needs to depend on Y to ensure prediction intervals with reasonable width. Predicting certain values (e.g., extreme values) of Y can be challenging due to representative bias. If we enforce the predictor to provide high coverage for marginalized groups with these values, it is highly likely to result in extremely wide intervals that offer little guidance in decision-making.

Meauring EOC . Given the underlying distribution p ∈ ∆ ( V × A × Y ) , where V , A , Y is the domain of v , a , y respectively. We can determine how likely p satisfies EOC by measuring its distance to p ′ , the closest distribution that perfectly achieves EOC. Formally, we denote P EOC as the property of EOC, the set of all distributions in defined space that satisfy EOC, i.e., P EOC := { p ∈ ∆ ( V × A × Y ) : ( V, A, Y ) ∼ p, V ⊥ ⊥ A | Y } . p ′ ∈ P EOC is the distribution with minimum total variation (TV) distance to p , i.e., ∀ q ∈ P EOC , d TV ( p, p ′ ) ≤ d TV ( p, q ) . The TV distance between p and p ′ is

formally defined as d TV ( p, p ′ ) = 1 2 ∑ ( v,a,y ) ∈ ∆ ( V × A × Y ) | p ( v, a, y ) -p ′ ( v, a, y ) | = 1 2 ‖ p -p ′ ‖ 1 , where ‖ · ‖ 1 denotes the l 1 norm of a distribution. Instead of the commonly-used Kolmogorov-Smirnov (KS) distance [12, 30], TV distance is chosen as the measure due to the significant drawback of the KS distance - its insensitivity to the deviations between p and p ′ at the tails [31]. Whereas in real-world data, we often confront such significant deviation (see e.g., Fig. 1).

It is challenging to directly measure the distance between p and p ′ from observed data when the target variable Y is continuous due to data sparsity issues for each value of Y . However, drawing from previous works [32, 33], we can instead use an easy-to-compute statistic T as a surrogate of d TV ( p, p ′ ) . T is the measure of independence, assessing the weighted summed violation of independence within each discretized bin. The expectation of T , E [ T ] , has a fixed upper bound when p satisfies EOC, and a lower bound that increases with the TV distance between p and p ′ . The intuition here is that when Y is divided into sufficient bins and p Y := Pr { V, A | Y } , the conditional distribution of p on Y , is Lipschitz continuous, enforcing independence within each bin generates a distribution that does not deviate far from P EOC in terms of TV distance.

Lemma 3.2. [33] Let d = /ceilingleft n 2 / 5 /ceilingright be the number of bins Y is discretized into, with each bin having an equal size of samples. Let T be a sum of independence measures within all discretized bins. Given the Lipschitz continuity of p Y in Assumption 2, L is the Lipschitz constant, we have:

When p satisfies EOC, E [ T ] ≤ | D | L 2 d 2 ;

When d TV ( p, p ′ ) = /epsilon1 , there exists a constant Z such that E [ T ] ≥ Z ( /epsilon1 -3 L d ) 2 .

Lemma 3.2 indicates that we can approximately evaluate EOC for p by measuring the independence within each bin. If V ⊥ ⊥ A almost holds in each bin, such that E [ T ] is small enough, then it is highly possible that V ⊥ ⊥ A | Y , i.e., p satisfies EOC. Moreover, an increasing function of /epsilon1 is upper bounded by E [ T ] . As E [ T ] increases, it is highly likely that p is further away from any p ′ that satisfies EOC. This lemma serves as a theoretical foundation for the proposed method below to improve EOC and for adopting T as an evaluation metric in experiments.

In order to calculate E [ T ] from data, we introduce unbiased estimators of T in Appendix 8.1.1. According to the central limit theorem, we could estimate E [ T ] through a sufficient number of random samplings. Whereas for the sake of efficiency, we prefer to construct T with bounded variance such that E [ T ] could converge through limited repeated samplings.

4 Improving Equal Opportunity of Coverage

In this section, we introduce a post-processing approach where we have a trained ML model, a calibration dataset, and a test dataset for which we aim to improve EOC. It consists of three steps. First, we enforce the independence of A and V within each interval of Y (i.e., EOC) for the calibration data, and then leverage conformal prediction to achieve EOC for test data. Lastly, we describe an efficient and robust optimization approach that optimizes both EOC and widths of prediction intervals.

4.1 Improving EOC on calibration data

The first step aims to improve EOC on the calibration data where we have ground-truth labels. As the target variable Y is continuous, the number of samples with certain values of Y can be extremely small, thus calibrating for each distinct value of Y is almost impossible. Meanwhile, commonly used methods designed for continuous variables such as adversarial learning [23, 22], which intend to learn a near-optimal p , are computationally inefficient for post-processing approaches [34].

According to Lemma 3.2, E [ T ] is an upper bound of an increasing function of d TV ( p, p ′ ) . Thus, this indicates that if E [ T ] decreases, the maximum distance between p and p ′ is reduced, resulting in an improvement in EOC. Informed by this idea, we first focus on enhancing EOC on the calibration data D c . For simplicity, we employ the framework introduced in [35] as the conformal prediction model, though our post-processing method is applicable to any base model. Specifically, let ˆ q α denotes the α -th conditional quantile regression function, i.e., for i -th sample ( X i , Y i ) , ˆ q α ( X i ) := inf { y ∈ ∆ Y : Pr { Y i ≤ y | X = X i } ≥ α } . Fix the lower and upper quantiles as α lo = α / 2 and α hi = 1 -α / 2 , then ˆ q α lo ( X i ) and ˆ q α hi ( X i ) denote lower and upper quantile regression functions, respectively. The base model is trained on the training data and used for inference on calibration and test data.

Following the discretization idea in Lemma 3.2, we divide the continuous variable Y ∈ [ Y min , Y max ] in D c into M bins with equal sample sizes, and the m -th bin is denoted as B m = [ Y -m , Y + m ) .

To enhance EOC on D c , we can minimize E [ T ] by enforcing the independence of A and V within all discretized bins. In particular, we fix the coverage rates within each bin as β m ∈ [0 , 1] , therefore V ⊥ ⊥ A | Y ∈ B m . Note that, although the coverage rates are equal across groups within the same bin, the quantile value at β m are computed separately for each group, as illustrated in Figure 2. We first calculate the vanilla prediction intervals ˆ C ( X i ) = [ˆ q α lo ( X i ) , ˆ q α hi ( X i )] obtained from the trained model, and get the conformity score S ( X i , Y i ) = max(ˆ q α lo ( X i ) -Y i , Y i -ˆ q α hi ( X i )) for all samples in D c . Then for different combinations of bins and protect groups, we calculate the quantile value of conformity scores at coverage rate β m , ˆ Q β ( S, D c ( a, m )) , for data in D c ( a, m ) = { i | i ∈ D c , A i = a, Y i ∈ B m } . To simplify notations, we substitute ˆ Q β ( S, D c ( a, m )) by G a,m ( β m ) . Since each bin has an equal sample size and coverage rate β m , the average coverage rate ∑ m β m /M is then set to 1 -α to keep a coverage rate of 1 -α on the calibration data.

(c) Bin 10

1.00

1.000

1.000

men

men

men

0.80

women

0.858

0.800

women

0.800

women

Beta

8:88

0.40

N

Bet

0.400

0.600

8:666

0.400

0.20

0.200

0.200

0.00

-20000-10000

0

10000200003000040000

0.000

-3000020000100000

10000 20000 30000 40000 50000

0.000

0

000000000000000000000000000000

ConformityScore

Conformity Score

Conformity Score

Through the reconstruction of prediction intervals C ( X i ) with G a,m ( β m ) , i.e., ∀ i ∈ D c ( a, m ) , C ( X i ) = [ˆ q α lo ( X i ) -G a,m ( β m ) , ˆ q α hi ( X i ) + G a,m ( β m )] , EOC is enhanced and the marginal coverage rate is guaranteed on the calibration data.

4.2 For Coverage and Independence Guarantees

This subsection seeks to preserve EOC and marginal coverage rate for test data based on results for calibration data through conformal prediction. The key is to find out, for a test sample with A n +1 = a n +1 , which bin B m it belongs to. We can then calibrate the test sample with the quantile G a,m ( β ) . However, direct calibration based on the predicted bin would not improve EOC since the prediction result can be biased due to, e.g., skewed distributions for different groups [29].

To address this issue, we propose Binned Fair Quantile Regression (BFQR) (see Algorithm 1 in Appendix 8.2.1). Our method could be treated as a variant of Mondrian conformal prediction [36, 37], where the confidence in each bin is evaluated independently. Suppose that a new data point with feature X i and protected attribute a falls into a certain bin B m , we calibrate it with the corresponding quantile value G a,m ( β m ) . Then, we obtain a sub-interval of prediction within bin B m , i.e., C m ( X i ) = B m ∩ [ˆ q α lo ( X i ) -G a,m ( β m ) , ˆ q α hi ( X i ) + G a,m ( β m )] . After computing C m ( X i ) , a union of all subsets C ( X i ) = ⋃ m C m ( X i ) is then the prediction interval of X i . Under a mild exchangeability assumption similar to [11], our algorithm provides both the marginal coverage guarantee and fair coverage guarantees within each bin.

Assumption 1 (Exchangeability) . All calibration data ( X i , Y i ) , i = 1 , ..., n and a sample of test data ( X n +1 , Y n +1 ) are exchangeable conditioned on A n +1 = a and Y n +1 ∈ B m , and conformity scores { S ( X i , Y i ) , i ∈ D c ( a, m ) ∪ { n +1 }} are almost surely distinct.

Theorem 4.2 (Marginal Coverage Guarantee) . If we have ∑ m β m /M = 1 -α , under Assumption 1, then Pr { Y n +1 ∈ C ( X n +1 ) } ≥ 1 -α .

Theorem 4.1 (Bin Coverage and Independence Guarantee) . Under Assumption 1, β m ≤ Pr { Y n +1 ∈ C ( X n +1 ) | A n +1 = a, Y n +1 ∈ B m } ≤ β m +1 / ( | D c ( a, m ) | +1) . The expectation of max coverage gap inside m -th bin is upper bounded by max a { 1 / ( | D c ( a, m ) | +1) } .

Here we sketch the proofs. Take ( X i , A i , Y i ) as new sample from test data, i ∈ D t . Suppose Y i ∈ B m , then the p -value of the null hypothesis Y i ∈ B m is given by ˆ u A i ,m = 1+ | j ∈ D c ( A i ,m ): S ( X j ,Y j ) ≤ S ( X i ,Y i ) | 1+ | D c ( A i ,m ) | , which is the proportion of conformity scores that are less than the score of the new sample among all calibration data that have the same protected attribute and fall into the same bin. The prediction interval ˆ C m ( X i ) in B m is the intersection of B m and the prediction interval calibrated by G a,m ( β m ) , which includes the part of B m in the prediction interval where the p -value ˆ u A i ,m is greater than β m . This guarantees the bin coverage of B m at level β m in Theorem 4.1. The complete proofs can be found in Appendix 8.2.2.

4.3 Constrained Optimization

With improved EOC and coverage guarantee, we now need to identify β m for each bin so that the mean width of prediction intervals for D t is the smallest. As such, our goal is to solve a constrained optimization problem, where the decision variables are the coverage rates in each bin β m , m = 1 , . . . , M , and the objective function is the mean width of prediction intervals in test data:

An easy solution to the optimization problem Eq. 1 can be obtained by adjusting ˆ C ( X i ) , X i ∈ D c with split conformal prediction described in Section 2. However, it is not optimal since β m is determined by ˆ Q 1 -α ( S, D c ) calculated on all calibration data, but different bins have varying costs of width associated with changes in their coverage rates. For instance, if a bin has a coverage rate of 0.98, increasing it to 0.99 would lead to a significant increase in its width, whereas increasing the coverage rate of a bin from 0.50 to 0.51 would result in only a minor increase in width. Therefore, we use the solution of split conformal prediction as the initialization and then optimize it.

Solving Eq. 1 is challenging in that it is computationally expensive or even infeasible to compute the value and the gradient of the objective function. First, the computation of prediction interval C ( X i ) involves multiple intersection and union operations, which is a complex step function of β m . Second, prediction interval C ( X i ) is related to quantile value G a,m , which is estimated from data with noise. Directly using slopes of G a,m as the gradient methods could result in over-fitting to noise, as shown in Section 5.2.4. Third, We cannot assume the objective function's convexity or differentiability as the data may come from any possible distribution, and sorting is involved in calculating G a,m [38]. To address those obstacles, we propose an efficient and robust optimization algorithm (detailed in Algorithm 2 in Appendix 8.3.2) that utilizes a relaxed upper bound and optimizes through approximated subgradients [39].

The steps of our approach are described below. First, to accelerate computing, we use a dummy continuous prediction interval C d ( X i ) = Convex ( ⋃ m C m ( X i )) , i.e., the convex hull of all subintervals, as an upper bound to substitute the original prediction interval C ( X i ) in the objective function. However, C d ( X i ) is still related to noisy G a,m . To address this, in each iteration, we compute the slope for each bin m in increasing and decreasing directions, denoted as ˆ t + m and ˆ t -m , respectively. Since the objective function aims to decrease without changing the marginal coverage rate, we take a greedy strategy by moving up a step η in the bin with the steepest descendent direction max m { ˆ t -m } , meanwhile taking a step η down the slowest ascendant direction min m { ˆ t + m } . We stop until max m { ˆ t + m } ≥ min m { ˆ t -m } +2 ε , where ε is an appropriate estimation error bound related with | D c | [40, 41]. More details can be found in Appendix 8.3.1.

The proposed method can be viewed as a subgradient method, incorporating considerations for quantile value estimation errors and maintaining a constant mean coverage rate. Combining the three steps, we could get prediction intervals with improved EOC, guaranteed marginal coverage, and decreased average width of prediction intervals.

5 Experiments

In this section, we conduct three sets of experiments on both synthetic and real-world data to evaluate (1) the effectiveness of the proposed approach for achieving EOC (Section 5.2) ; (2) the impact of the

key parameter M , the number of bins (Section 5.2.3); and (3) the effectiveness and efficiency of the proposed optimization framework (Section 5.2.4).

5.1 Experimental Setup

We propose two metrics to evaluate the first property of EOC, i.e., whether coverage rates for different groups with similar outcomes are close. With the M discretized bins in Section 4.1, the first metric is the average of maximum difference in coverage rates between groups for all bins, similar to the definition of conditional KS distance in [12]. In main experiments, M is set as 20. The second metric T is introduced in Lemma 3.2, where T is calculated on d = /ceilingleft | D t | 2 / 5 /ceilingright bins. Specifically, d = M for fairness of evaluation, e.g., d = 100 for synthetic data. A smaller T implies a closer distance to the ideal EOC distribution and, therefore a better EOC. Furthermore, for the second property of EOC, we need to ensure the desired marginal coverage rates. The efficiency of uncertainty estimation is measured by the width of the prediction interval. In addition, we check the conditional coverage rates on groups to measure equalized coverage. All metrics related to coverage rates are multiplied by 100 in tables to exhibit significant differences. Every experiment is repeated 100 times on random divisions of data with different seeds, with | D tr | : | D c | : | D t | = 3 : 1 : 1 .

/negationslash

We compare our method with the following state-of-the-art methods: 1) Split Conformalized Quantile Regression (CQR) [35, 21] with only marginal coverage guarantee; 2) Group-conditional Conformalized Quantile Regression (GCQR) [11] with both marginal and conditional coverage guarantee; 3) MultiValid Predictor (MVP) [18] with both marginal and conditional coverage guarantee. Note that the base model for this algorithm is trained on the union of training and calibration data as MVP does not require any calibration data; 4) Conformal Fair Quantile Prediction (CFQR) [19], which guarantees marginal coverage and demographic parity on both upper and lower bounds of the predicted intervals; 5) Label-conditional Conformalized Quantile Regression (LCQR) [42], which is designed for classification problems to provide marginal coverage and equalized coverage for each class. We adapt it to regression tasks. The base model for all compared conformal prediction methods is set as the QR model at the level of 0.05 and 0.95, and the desired marginal coverage is set to 0.9. Considering that for some real-world applications like scoring [43], disjoint prediction intervals make little sense, we evaluate prediction intervals C ( X ) along with their dummy prediction intervals C d ( X ) in Section 4.3, represented as BFQR and BFQR*. In the comparison tables, the best results and the second-best results are highlighted in bold and underlined, and undercovered groups who fail to reject the null hypothesis at 0.05 level in one-tailed t-tests are emphasized in Italian.

5.2 Results

5.2.1 Synthetic data

We generate ten independent and exponentially distributed features with the scale of 1, X = ( X 1 , . . . , X 10 ) ; protected attribute A is randomly selected from { 0 , 1 , 2 } with a probability of 0.1, 0.2, 0.7, respectively. The labels Y for A = 1 follow a random distribution, thus impossible to predict; labels Y for the other two groups are the linear summations of X and A , plus noises that increase with Y . A size of 100,000 samples are generated from this distribution and the data generating process is detailed in Appendix 8.4.1.

We have the following observations from the results in Table 1: 1) All methods have marginal coverage guarantee, which is attributed to statistical guarantee from conformal prediction. 2) Our proposed methods, BFQR with disjoint intervals and BFQR* with joint intervals achieve the best trade-off between EOC and equalized coverage. In particular, our method achieves the second-best EOC (i.e., Mean Max Coverage Gap and T ), meanwhile, the conditional coverage rates are almost equal across different groups. While CFQR has significantly better performance w.r.t. EOC, the conditional coverage for A = 1 (the most challenging case) is extremely low compared to A = 0 and A = 2 . This result aligns with the mutual exclusivity between EOC and equalized coverage formulated in Theorem 3.1. 3) Among all methods, the average interval width of our method is the smallest, validating the effectiveness of the optimization process in Section 4.3. One of the main advantages of BFQR and BFQR* is optimizing through bin coverages. In this process, bins that sacrifice interval width for an over-coverage rate are adjusted to a lower but satisfactory coverage rate. Therefore, we are able to guarantee a smaller average interval width while improving the EOC.

5.2.2 Real-world Data

We further evaluate our method on two benchmark datasets: Adult [44, 45] where gender is the protected attribute and the outcome is salary; MEPS (Medical Expenditure Panel Survey) data [46, 11] where race is the protected attribute and the outcome is the health care system utilization score.

We observe similar results: For Adult data (shown in Table 2), all methods achieve marginal coverage. Our methods achieve the best EOC and smallest mean width of prediction interval while maintaining competitive conditional coverage rates. CFQR with the best EOC on synthetic data does not have consistently good performance, and the mean interval width is greatly larger compared with other methods. The increased width of LCQR implies that as some bins are difficult to predict, enforcing all bins to reach the same high coverage rates generates prediction results with little useful information. For MEPS data (shown in Table 3), the results are similar to those in synthetic data: our method is the second-best w.r.t. EOC, with marginal coverage rates guaranteed and a significantly smaller average prediction width.

5.2.3 The Impact of M

The number of bins, M , is the only primary parameter in our method. Using synthetic data that ideally satisfies Assumption 1, we evaluate how M influences EOC. In our experiment, M varies

among {1, 5, 10, 20, 50}. When bin size is 1, our method degenerates to GCQR. The results are shown in Fig. 3, where the best EOC for BFQR and BFQR* is achieved with 10 bins. When the number of bins is small, the sample size of D a,m increases. According to Theorem 4.1, the upper bound of the expectation of max group coverage rate gap is decreased. However, this only suggests a decrease in the expectation, not for every Y within bins. When the quantile value is calculated on large samples, it fails to characterize the conformity scores of individuals. BFQR* with continuous intervals exhibits better EOC compared to BFQR as its coverage gaps are primarily influenced by the quantile values of the first and last bins, involving less randomness. As M greatly affects the performance of our methods, it should be cautiously chosen for various problem settings.

5.2.4 Comparisons of Optimization Methods

To demonstrate the efficiency of the proposed optimization method, we compare it with three recent optimization methods: 1) BFGS-SQP [47, 48], a constrained nonsmooth optimization method, 2) Augmented Lagrange (AL) [49], another method to solve constrained nonsmooth optimization problems, and 3) Bayesian optimization [50, 51], a global optimization method for solving problems with noisy objective functions. We evaluate these methods based on their performance in terms of EOC, width, and running times. The results on synthetic data are in Table 4. The Bayesian method has extremely slow computational speed and cannot guarantee marginal coverage. In comparison to BFGS-SQP and AL, our method

achieves similar EOC performance but with slightly lower prediction interval widths, which confirms the effectiveness of our subgradient approximation. Additionally, our method is more efficient as it significantly reduces running times.

6 Discussion

EOC or equalized coverage? The results in Section 5.2 clearly demonstrate that perfect EOC and equalized coverage are mutually exclusive, empirically verifying Theorem 3.1. In real-world applications, it becomes crucial to trade off between EOC and equalized coverage. We recommend placing a higher emphasis on EOC when certain labels are considered more favorable, e.g., when individuals labeled as low-salary are the most important subpopulation for decision-making. Under EOC, each group with the same label is treated equally, not only in terms of equal coverage rates (i.e., equal probability of being included in the prediction result) but also in terms of comparable prediction interval widths as a measure of uncertainty. For example, under equalized coverage, the coverage rate for men is around 0.65 but 0.55 for women in the first bin, low-income population. We cannot conclude that the model is more confident in the prediction results for low-income men solely based on the larger prediction intervals of men: this discrepancy might arise from the over-coverage of men. In this sense, our designed metric EOC, not only contributes to evaluating and enhancing fairness under uncertainty but also serves for fair uncertainty quantification, discovering model bias in uncertainty quantification. A possible application of EOC is for guiding sample selection in active learning [52, 53], allowing the applied model to label the most uncertain samples in a fair manner.

Discretizition into equal-mass bins is a favorable strategy to avoid excessively small sizes of D ( a, m ) . While this guarantees a tighter upper bound for the expectation of the maximum coverage gap within bins according to Theorem 4.1, our method can be extended beyond equal-mass bins. When the

number of samples in each bin is unequal, we adjust the constraints in Eq. 1 by incorporating a weighted sum of β m , i.e., ∑ m ∈ [ M ] β m M | D ( m ) | = 1 -α . Consequently, in Algorithm 2, the gradient in the direction of each bin also should be weighted by | D ( m ) | . These adaptations allow our method to maintain its effectiveness when the sample sizes in each class are uneven, e.g., in imbalanced classification problems. Moreover, our method is able to address certain coverage rate concerns within particular bins, such as low-income females, by assigning specific coverage rates to the corresponding bins while allowing flexibility in adjusting the coverage rates of other bins.

Why Better EOC? Our methods have been empirically shown to have better EOC than methods designed for equalized coverage, and here we elucidate the reasons behind this improvement. Given the underlying distribution of the α -quantile conformity scores in group a conditioned on y as Q a,y ( α ) . For every y and a , calibrating with Q a,y ( α ) would generate perfect EOC. However, due to the limitations of available calibration samples, the estimation of Q a,y ( α ) is subject to bias. Since the base model cannot perfectly capture the underlying distribution, the conformity score, as a measure of disagreement, must correlate with Y . While equalized coverage methods generally ignore this correlation, our method tries to handle it. In Section 4.1, we used G a,m ( β m ) to estimate Q a,y ( β m ) for all y ∈ B m , which is the β m -quantile value of conformity scores whose y belong to bin B m . The localized estimation within discretized bins is upper-bounded by the largest β m -quantile value of score conditioned on any y and lower-bounded by the smallest one, regardless of the unknown distribution of Y . Consequently, we could better characterize the conditional distribution within each bin. This enhancement is especially evident when the Lipschitiz constant L in Lemma 3.2 is small, such that the deviation from perfect EOC to the distribution generated by bins is negligible.

Possible Extension. For future work, we aim to further improve our method by seeking a tighter bound to achieve optimal EOC. Possible direction may follow [14, 54], by discretizing Y into finite values and bounding deviation in EOC of the discretized predictor. Additionally, our conformalized method leverages statistical properties to ensure coverage rates in each bin and optimize prediction interval widths, making it particularly appropriate for large datasets. With a better discretization strategy that better utilizes the calibration samples, e.g., ensemble sampling, our method may maintain its effectiveness on a small calibration set.

Potential Exploration beyond the Current Scope. The context of this paper is situated in postprocessing fairness. Though post-processing methods typically underperform in-processing methods, in-processing methods are not applicable in many situations, e.g., the prediction model is a pre-trained black-box regression model, or a flexible and computationally efficient method is required. Whereas, adapting the proposed method to an in-processing setting to obtain better EOC is a possible extension. It would also be interesting to extend our method to the context that sensitive attributes are not available in the test data, possibly through missing data augmentation for the calibration set [55] or prediction-powered inference [56].

7 Conclusion

In this paper, we introduce a new uncertainty-aware fairness notion, equal opportunity of coverage (EOC), which addresses the limitations of the seminal work of equalized coverage [11]. EOC has several desired properties: It guarantees equal coverage rates for groups with the same labels and marginal coverage rate at a pre-determined level. It also ensures a small prediction interval. The theoretical analyses and empirical findings indicate that EOC and equalized coverage are generally incompatible. We suggest using EOC as an alternative to equalized coverage when equal coverage rates and assessment of uncertainty are needed for more fine-grained demographic groups. To improve EOC, we propose a distribution-free post-processing method, BFQR, based on discretization. Experimental results on synthetic and real-world datasets show that BFQR achieves competitive EOC and ensures guaranteed marginal coverage rates with small mean prediction interval widths compared to the state-of-the-art. Moreover, BFQR is adaptable to various settings, such as classification and other decision-making tasks.

Acknowledgments and Disclosure of Funding

This work is supported in part by NSF under grant III-2106758. Lu Cheng is in part supported by the Cisco Research Gift Grant. We thank Xinhua Zhang for the helpful discussion on the optimization

method. We are grateful to the anonymous reviewers at NeurIPS 2023 for providing valuable feedback and suggestions.

References

Osonde A Osoba and William Welser IV. An intelligence in our image: The risks of bias and errors in artificial intelligence . Rand Corporation, 2017.

Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. In Ethics of data and analytics , pages 254-264. Auerbach Publications, 2016.

Ayanna Howard and Jason Borenstein. The ugly truth about ourselves and our robot creations: the problem of bias and social inequity. Science and engineering ethics , 24:1521-1536, 2018.

Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data , 5(2):153-163, 2017.

Lu Cheng, Kush R Varshney, and Huan Liu. Socially responsible ai algorithms: Issues, purposes, and challenges. Journal of Artificial Intelligence Research , 71:1137-1181, 2021.

Henry Peng Zou, Yue Zhou, Weizhi Zhang, and Cornelia Caragea. Decrisismb: Debiased semi-supervised learning for crisis tweet classification via memory bank. In Findings of the Association for Computational Linguistics: EMNLP 2023 , 2023.

Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems , 30, 2017.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321-1330. PMLR, 2017.

Henry Peng Zou and Cornelia Caragea. Jointmatch: A unified approach for diverse and collaborative pseudo-labeling to semi-supervised text classification. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , 2023.

Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference , pages 214-226, 2012.

Yaniv Romano, Rina Foygel Barber, Chiara Sabatti, and Emmanuel Candès. With malice toward none: Assessing uncertainty via equalized coverage. Harvard Data Science Review , 2(2):4, 2020.

Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems , 29, 2016.

Junpei Komiyama, Akiko Takeda, Junya Honda, and Hajime Shimao. Nonconvex optimization for regression with fairness constraints. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 2737-2746. PMLR, 10-15 Jul 2018.

Alekh Agarwal, Miroslav Dudík, and Zhiwei Steven Wu. Fair regression: Quantitative definitions and reduction-based algorithms. In International Conference on Machine Learning , pages 120-129. PMLR, 2019.

Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A reductions approach to fair classification. In International Conference on Machine Learning , pages 60-69. PMLR, 2018.

Jianfeng Chi, Yuan Tian, Geoffrey J Gordon, and Han Zhao. Understanding and mitigating accuracy disparity in regression. In International conference on machine learning , pages 1866-1876. PMLR, 2021.

Shai Feldman, Stephen Bates, and Yaniv Romano. Improving conditional coverage via orthogonal quantile regression. Advances in Neural Information Processing Systems , 34:2060-2071, 2021.

Osbert Bastani, Varun Gupta, Christopher Jung, Georgy Noarov, Ramya Ramalingam, and Aaron Roth. Practical adversarial multivalid conformal prediction. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 29362-29373. Curran Associates, Inc., 2022.

Meichen Liu, Lei Ding, Dengdeng Yu, Wulong Liu, Linglong Kong, and Bei Jiang. Conformalized fairness via quantile regression. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 11561-11572. Curran Associates, Inc., 2022.

Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world , volume 29. Springer, 2005.

Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511 , 2021.

Pieter Delobelle, Paul Temple, Gilles Perrouin, Benoît Frénay, Patrick Heymans, and Bettina Berendt. Ethical adversaries: Towards mitigating unfairness with adversarial machine learning. ACM SIGKDD Explorations Newsletter , 23(1):32-41, 2021.

Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society , pages 335-340, 2018.

Dana Yang, John Lafferty, and David Pollard. Fair quantile regression. arXiv preprint arXiv:1907.08646 , 2019.

Robert Williamson and Aditya Menon. Fairness risk measures. In International Conference on Machine Learning , pages 6786-6797. PMLR, 2019.

Carlos Pinzón, Catuscia Palamidessi, Pablo Piantanida, and Frank Valencia. On the incompatibility of accuracy and equal opportunity. Machine Learning , pages 1-30, 2023.

Han Zhao and Geoffrey J Gordon. Inherent tradeoffs in learning fair representations. The Journal of Machine Learning Research , 23(1):2527-2552, 2022.

Vladimir Vovk. Conditional validity of inductive conformal predictors. In Steven C. H. Hoi and Wray Buntine, editors, Proceedings of the Asian Conference on Machine Learning , volume 25 of Proceedings of Machine Learning Research , pages 475-490, Singapore Management University, Singapore, 04-06 Nov 2012. PMLR.

Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) , 54(6):1-35, 2021.

Kun Zhang, Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Kernel-based conditional independence test and application in causal discovery. arXiv preprint arXiv:1202.3775 , 2012.

Veeranjaneyulu Sadhanala, Yu-Xiang Wang, Aaditya Ramdas, and Ryan J Tibshirani. A higherorder kolmogorov-smirnov test. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 2621-2630. PMLR, 2019.

Clément L Canonne, Ilias Diakonikolas, Daniel M Kane, and Alistair Stewart. Testing conditional independence of discrete distributions. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing , pages 735-748, 2018.

Matey Neykov, Sivaraman Balakrishnan, and Larry Wasserman. Minimax optimal conditional independence testing. The Annals of Statistics , 49(4):2151-2177, 2021.

Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations , 2020.

Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. Advances in neural information processing systems , 32, 2019.

Henrik Boström and Ulf Johansson. Mondrian conformal regressors. In Conformal and Probabilistic Prediction and Applications , pages 114-133. PMLR, 2020.

Matteo Fontana, Gianluca Zeni, and Simone Vantini. Conformal prediction: a unified review of theory and new challenges. Bernoulli , 29(1):1-23, 2023.

Bat-Sheva Einbinder, Yaniv Romano, Matteo Sesia, and Yanfei Zhou. Training uncertaintyaware classifiers with conformalized deep learning. In Advances in Neural Information Processing Systems , 2022.

Torbjörn Larsson, Michael Patriksson, and Ann-Brith Strömberg. Conditional subgradient optimization-theory and applications. European Journal of Operational Research , 88(2):382403, 1996.

Patrizia Berti and Pietro Rigo. A glivenko-cantelli theorem for exchangeable random variables. Statistics & Probability Letters , 32(4):385-391, 1997.

Stephen Boyd, Lin Xiao, and Almir Mutapcic. Subgradient methods. lecture notes of EE392o, Stanford University, Autumn Quarter , 2004:2004-2005, 2003.

Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adaptive coverage. Advances in Neural Information Processing Systems , 33:3581-3591, 2020.

Julia Angwin, Jeff Larson, Lauren Kirchner, and Surya Mattu. Machine bias, May 2016.

Drago Pleˇ cko, Nicolas Bennett, and Nicolai Meinshausen. fairadapt: Causal reasoning for fair data pre-processing. arXiv preprint arXiv:2110.10200 , 2021.

Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. Advances in neural information processing systems , 34:6478-6490, 2021.

Agency for Healthcare Research and Quality. Medical expenditure panel survey. https: //meps.ahrq.gov/mepsweb , 1996.

Buyun Liang, Tim Mitchell, and Ju Sun. Ncvx: A general-purpose optimization solver for constrained machine and deep learning. arXiv preprint arXiv:2210.00973 , 2022.

Frank E Curtis, Tim Mitchell, and Michael L Overton. A bfgs-sqp method for nonsmooth, nonconvex, constrained optimization and its evaluation using relative minimization profiles. Optimization Methods and Software , 32(1):148-181, 2017.

Yu Wang, Wotao Yin, and Jinshan Zeng. Global convergence of admm in nonconvex nonsmooth optimization. Journal of Scientific Computing , 78:29-63, 2019.

Benjamin Letham, Brian Karrer, Guilherme Ottoni, and Eytan Bakshy. Constrained bayesian optimization with noisy experiments. Bayesian Analysis , 14(2):495-519, 2019.

Yang Li, Yu Shen, Wentao Zhang, Yuanwei Chen, Huaijun Jiang, Mingchao Liu, Jiawei Jiang, Jinyang Gao, Wentao Wu, Zhi Yang, et al. Openbox: A generalized black-box optimization service. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 3209-3219, 2021.

Yifan Fu, Xingquan Zhu, and Bin Li. A survey on instance selection for active learning. Knowledge and information systems , 35:249-283, 2013.

Hadis Anahideh, Abolfazl Asudeh, and Saravanan Thirumuruganathan. Fair active learning. Expert Systems with Applications , 199:116981, 2022.

Evgenii Chzhen, Christophe Denis, Mohamed Hebiri, Luca Oneto, and Massimiliano Pontil. Fair regression via plug-in estimator and recalibration with statistical guarantees. Advances in Neural Information Processing Systems , 33:19137-19148, 2020.

Margaux Zaffran, Aymeric Dieuleveut, Julie Josse, and Yaniv Romano. Conformal prediction with missing values. arXiv preprint arXiv:2306.02732 , 2023.

Anastasios N Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I Jordan, and Tijana Zrnic. Prediction-powered inference. arXiv preprint arXiv:2301.09633 , 2023.

Gábor J Székely, Maria L Rizzo, and Nail K Bakirov. Measuring and testing dependence by correlation of distances. The Annals of Statistics , 35(6):2769, 2007.

Gábor J Székely and Maria L Rizzo. Energy statistics: A class of statistics based on distances. Journal of statistical planning and inference , 143(8):1249-1272, 2013.

Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association , 113(523):1094-1111, 2018.

Peer reviews:
comments: SUMMARY:
Authors propose a new uncertainty-aware fairness – Equal Opportunity of Coverage (EOC) since current definition do not guarantee equal coverage rates across more fine-grained groups (e.g., low-income females) conditioning on the true label and is biased in the assessment of uncertainty.

STRENGTHS:
- paper is technically sound 
- experimental results are promising

WEAKNESSES:
- presentation is mostly verbose and unclear
- novelty is limited

QUESTIONS:
The paper is, in my opinion, technically sound with promisin gresults.
Nevertheless I think that the approach is not completely novel.
Many works in the literature of fairness focus on distribution matching (see also NeurIPS papers when they use MMD, Sinkhorn distance, or first order matching) and there in no discussion on the relation with these works and a comparison with the idea of this work.
Authors should clarify and elaborate on this.

LIMITATIONS:
Authors should better clarify the novelty of the work and its impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies fair machine learning under predictive uncertainty. \nThey propose a new notion of fairness named Equal Opportunity of Coverage (EOC), building upon \"equalized coverage.\" While EC ensures that every demographic group receives the same level of prediction certainty, EOC ensures that every demographic group with similar true labels receives the same level of prediction certainty. \nThis extension is similar to the extension of equal opportunity from statistical parity. \nWith EOC, they also ensure that the entire population's coverage can be maintained at a predetermined level. \nTo achieve EOC, they propose a post-processing method called Binned Fair Quantile Regression (BFQR), a distribution-free post-processing method for a given trained model. \n\nThey show the power of their method both on the simulated data and real-world datasets.\n\nSTRENGTHS:\nThis is a very interesting research direction. \n\nFair regression is also a challenging problem, and little research extends equal opportunity (and its variants) to regression settings. \nThis is a promising research direction.\n\nWEAKNESSES:\nI have a few apprehensions regarding the notation and the inadequate introduction of certain mathematical objects. Please see my comments below.\n\nQUESTIONS:\n### Major Comments:\n\n1) Line 83: Why are the calibration and test sets exchangeable?\n2) Line 111: How does this indicate \"V \\perp A\"?\n3) TV definition in Line 122 needs to be more rigorous. Please define what $p$ and $p'$, which space they live, etc. \n4) Up to Line 135, I need clarification on $p$. Is the true data-generating distribution, or is it the empirical distribution? \n5) In Lemma 3.2, what is the sum of independence measures? What is the exact reference of Lemma 3.2 in [30]?\n6) Why do you choose to use TV? (I understand why not KS distance, but I do not understand why you choose TV over other metrics. An example could be Wasserstein distance, MMD, KL divergence etc.)\n7) Line 164: The coverage number $\\beta_m$ is the same for each group, right? It is different across subgroups. Then, I do not understand why $\\beta$ is computed separately for each group. Isn't it computed separately for each bin? I do not understand that part.\n8) The difference between BFQR and BFQR* is not discussed in the main text. What are the differences between the two? \n\n9) What are the conditions that Assumption 1 is violated? How strong is this assumption?\n\n10) Is it possible to include EOC constraints somehow during training?\nDo you think it would be a stronger method if this method could be an in-processing method?\n\n\n\n### Minor Comments:\n\n1) Typo Line 85: \"Pr(\" instead of \"P(\".\n2) In (1), $min$ and $s.t.$ should be aligned, and \"s.t.\" should be a math operator. \n3) Please introduce each object rigorously. It is unclear to me what sort of mathematical objects you are referring to until I see them used. For example, quantile regression. What is $\\hat q$? Is this a function? Is it a vector? What is $\\alpha_{lo}$? This would help the readers a lot if you were to introduce each object appearing in the paper rigorously. \n4) Line 252: \"l\" should be capitalized in lemma\n5) Line 312:  \"t\" should be capitalized in theorem\n6) Please indicate the best results in the experiment with bold font. \n7) Line 224: \"W\" should not be capitalized.\n8) Line 360: \"\\beta_m\" instead of $\\beta m$\n9) Line 272: \"Section\" instead of \"Sec.\" (also 290). \n10) Line 311: \"Figure\" instead of \"Fig.\" \n11) What is the probability distribution $\\mathbb P$ appeared suddenly in Theorem 4.1 and Theorem 4.2? \nI do not think the notation of this paper is unified. Please unify the notation during the rebuttal. Defining notation rigorously and sticking with it throughout the paper is important. Otherwise, the paper becomes unnecessarily hard to follow.\n12) I think it would be safe to drop the \"under no label distribution shift on test data\" assumption from Theorem 4.2. In the beginning, if you were to define the true data-generating distribution and explain how the test data and the calibration data are created from iid samples from this true data-generating distribution, then there is no need to mention this shift. If you could cover the shifted distribution case, then I would suggest mentioning it.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method to achieve same coverage rates between different groups conditioned on similar targets in addition to satisfying a marginal coverage rate on the entire population. The proposed objective is denoted as Equal Opportunity of Coverage (EOC) and the authors propose a two step approach denoted as Binned Fair Quantile Regression (BFQR) to address it. This method partitions the target space into bins and for each bin finds a coverage rate such that EOC is satisfied (i.e., equal coverage across sensitive groups inside the bin and global coverage rate level is achieved.). At test time the prediction intervals are built as the union of the intersections of each bin and its associated conformal prediction interval taking into account the group membership of the test sample.\n\nSTRENGTHS:\nI think the problem presented in this work is well motivated. Figure 1 is a nice graphical example to motivate the importance of considering EOC over Equalized coverage.\n\nThe definition of EOC is clear and as far as I know it was not discussed in other works. Its associated theory is well linked to previous works on Equal Opportunity.\n\nEmpirically, the proposed BFQR method is improving the maximum coverage gap between groups across the bins when compared to almost all the competing approaches. It also seems to achieve similar group coverage (average) across the groups and maintaining smaller interval widths.\n\nWEAKNESSES:\nI think the clarity of the presentation should be improved. In particular, I found that the description of the proposed procedure in Section 4 needs better clarification. For instance, L161 to L164 mentions that to enforce EOC the coverage rate within each bin $\\beta_m$ is the same across groups. The method to achieve this in practice is not clear to me when reading the main manuscript. Paragraph in L116 discusses that TV distance w.r.t. the closest EOC distribution (p’) is measured. However, it is not clear enough how p’ is obtained and how this distance is obtained. I understand the authors are citing the works from [39,40] where an ‘easy-to'-compute statistic T’ is being used but this is an important point in the proposed method and should be clarified here for reproducibility purposes.\n\nI found that Algorithm 1 in Appendix is a better summary of the description provided in last part of Section 4.1 and 4.2. However, this Algorithm builds on top of the previously estimated bin coverage rate $\\beta_m$.\n\nI think the organization of the paper is a bit convoluted since Section 4.3 describes how to obtain $\\beta_m$ but this was already discussed in Section 4.1 and Section 3. Moreover, Section 4.1 and Section 4.2 were relying on this. I think that the description on how $\\beta_m$ is obtained could be improved, it could be made more concise and informative.\n\nFigure 1 provides a nice and intuitive illustration of the problem, it would be nice to see the same plot after the proposed post processing procedure is applied in comparison with the other competing methods.\n\nFrom Algorithm 1 in the Appendix and L172-173 it seems that during test time the sensitive attribute needs to be available. I can think of this as a weak point in the fairness setting since the sensitive attribute needs to be available at test time. However, this is also the case for other existing fairness post-processing approaches.\n\nQUESTIONS:\nIf the sensitive attribute needs to be disclosed for each test sample, I think a reasonable comparison would be against a model that performs a group conditional split conformal approach that conditions on the sensitive attribute and a binning based on the outcomes of the quantile regression model. Is any of the competing approaches doing this?\n\nCFQR seem to be the method achieving the best Max coverage gap in Table 1 and Table 3 by a large margin. In particular, in Table 1 the max coverage gap is 0.19 but group A=1 has an overall coverage of 71.38 versus the other two groups whose coverage are larger than 93. I wonder if this is because across the majority of bins group A=1 is the one with the worse coverage even though by a small amount per bin (i.e., less than 0.19). How many bins where considered to the synthetic example?  I think to achieve that large difference between group coverages with small coverage gaps there has to be many bins.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of algorithmic fairness using uncertainty quantification methods in regression settings. The main novelty of this work over previous work is taking the notion of “equalized coverage” and strengthening it by considering finer groups by additionally conditioning on the target variable. The result is a conformal prediction based post-processing method that provably gives conditional coverage guarantees over sets defined by the demographic groups and target variable, where most prior work just considers sets on the group membership. Another upshot of this method is that the conformal-based post-processing method proposed (BFQR) explicitly focuses on also optimizing interval widths of predictions to improve “the amount of decision-making information available” to downstream decision-makers. \n\nThe paper is organized by presenting the main fairness notion studied - equal opportunity of coverage (EOC), describing the conformal prediction based post-processing approach to improve EOC and optimize for prediction interval widths. The main theoretical results are marginal and conditional coverage guarantees of BFQR. Finally, there is experimental evaluation on one synthetic and two benchmark datasets with comparisons to a few other conformal prediction based regression procedures.\n\nSTRENGTHS:\nThis paper studies an important problem - giving conditional coverage guarantees, while conditioning on the true label of the data, using a general post-processing conformal method, with particular emphasis on minimizing interval width to give more utility to downstream decision makers. The presentation is clear and precise.\n\nWEAKNESSES:\nA main upshot of BFQR as presented is the improvement on interval width, which seems to be modest on 2/3 (synthetic and Adult) datasets. It seems hard to compare this statistic with the other methods and evaluate how it compares to BFQR’s slight degradation in tightness of coverage rates.\n\nQUESTIONS:\nThere are a few things that would be helpful to clarify in your experimental setup. What are the groups / conditioning events for the comparison experiments run with GCQR and MVP? My understanding is that the sets they give conditional guarantees for do not need need to condition on the true label - but can use them if given the information. If tested in this way, could it take some of the bite out of the improvement we see in EOC with BFQR / is that already how the experiments were run?\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}