Title: Conformal Classification with Equalized Coverage for Adaptively Selected Groups
Venue: NeurIPS 2024
Accepted: yes
Keywords: Classification

Conformal Prediction

Equalized Coverage

Fairness
Abstract: This paper introduces a conformal inference method to evaluate uncertainty in classification by generating prediction sets with valid coverage conditional on adaptively chosen features. These features are carefully selected to reflect potential model limitations or biases. This can be useful to find a practical compromise between efficiency---by providing informative predictions---and algorithmic fairness---by ensuring equalized coverage for the most sensitive groups. We demonstrate the validity and effectiveness of this method on simulated and real data sets.

Conformal Classification with Equalized Coverage for Adaptively Selected Groups

Abstract

This paper introduces a conformal inference method to evaluate uncertainty in classification by generating prediction sets with valid coverage conditional on adaptively chosen features. These features are carefully selected to reflect potential model limitations or biases. This can be useful to find a practical compromise between efficiency-by providing informative predictions-and algorithmic fairnessby ensuring equalized coverage for the most sensitive groups. We demonstrate the validity and effectiveness of this method on simulated and real data sets.

1 Introduction

1.1 Uncertainty, Fairness, and Efficiency in Machine Learning

Increasingly sophisticated machine learning (ML) models, like deep neural networks, are revolutionizing decision-making in many high-stakes domains, including medical diagnostics [1], job screening [2], and recidivism prediction [3, 4]. However, serious concerns related to uncertainty quantification [5, 6] and algorithmic fairness [7-10] underscore the need for novel methods that can provide reliable and unbiased measures of confidence, applicable to any model.

Uncertainty quantification is crucial because ML models, although effective on average, can make errors while displaying overconfidence [11]. Consequently, in some situations users may lack sufficient warning about the potential unreliability of a prediction, raising trust and safety concerns. A promising solution is conformal inference [12-14], which enables converting the output of any model into prediction sets with precise coverage guarantees. These sets reflect the model's confidence on a case-by-case basis, with smaller sets indicating higher confidence in a specific prediction.

Algorithmic fairness focuses on the challenges of prediction inaccuracies that disproportionately impact specific groups, often identified by sensitive attributes like race, sex, and age. Among the many sources of algorithmic bias are training data that do not adequately represent the population's heterogeneity and a focus on maximizing average performance. However, fairness is partly subjective and lacks a universally accepted definition [15], leading to sometimes conflicting interpretations [16].

This complexity makes conformal inference with equalized coverage [17] an appealing approach. Equalized coverage aims to ensure that the prediction sets attain their coverage not only on average for the whole population (e.g., above 90%) but also at the same level within each group of interest. While this does not necessarily imply that the prediction sets will have equal size on average across different groups-since it is possible the predictive model may be more or less accurate for different groups-it objectively communicates the possible limitations of a model. This transparency helps decisionmakers recognize when predictions may be less reliable for specific subgroups, allowing them to either avoid unnecessary actions or adopt more cautious strategies in cases of higher uncertainty, thereby minimizing the potential harm from inaccurate predictions.

A limitation of the method developed in [17] for conformal inference with equalized coverage is that it does not scale well to situations involving diverse populations with multiple sensitive attributes. In such cases, it necessitates splitting the data into exponentially many subsets, significantly reducing the effective sample size and leading to less informative predictions. Balancing this trade-off [18] between efficiency -aiming for highly informative predictions with small set sizes-and fairness -ensuring unbiased treatment-is challenging and requires novel approaches. This paper introduces a method to address this by providing equalized coverage conditional on carefully chosen features, informed by the model and data. While it cannot guarantee equalized coverage for all sensitive groups, it seeks a reasonable compromise with finite data sets, mitigating significant biases while retaining predictive power.

1.2 Background on Conformal Inference for Classification

Consider a data set comprising n exchangeable (e.g., i.i.d.) observations Z i for i ∈ D := [ n ] := { 1 , . . . , n } , sampled from an arbitrary and unknown distribution P Z . In classification, one can write Z i = ( X i , Y i ) , where Y i ∈ [ L ] := { 1 , . . . , L } is a categorical label and X i ∈ X represents the individual's features, taking values in some space X . As explained below, we will assume these features include some sensitive attributes. Further, we consider a test point Z n +1 = ( X n +1 , Y n +1 ) , also sampled exchangeably from P Z , and whose label Y n +1 ∈ [ L ] has not yet been observed.

A standard goal for split conformal prediction methods is to quantify the predictive uncertainty of a given 'black-box' ML model (e.g., pre-trained on an independent data set) by constructing a prediction set ˆ C ( X n +1 ) for Y n +1 , guaranteeing marginal coverage at some desired level α ∈ (0 , 1) :

This probability is taken over the randomness in Y n +1 and X n +1 , as well as in the data indexed by D . Intuitively, marginal coverage means the prediction sets are expected to cover the correct outcomes for a fraction 1 -α of the population. However, this is not always satisfactory, especially if the miscoverage errors may disproportionately affect individuals characterized by well-defined features.

To address these concerns, one might consider feature-conditional coverage , P [ Y n +1 ∈ ˆ C ( X n +1 ) | X n +1 = x ] ≥ 1 -α for all x ∈ X . This would ensure consistent coverage for all possible test features X n +1 . However, it is impossible to achieve without additional assumptions, such as modeling the distribution P Z [19] or significantly restricting the feature space X [20]. Given that such assumptions may be unrealistic in real-world settings, exact feature-conditional coverage is typically unachievable.

Equalized coverage [17] seeks a practical middle ground between the two extremes of marginal and feature-conditional coverage, focusing on accounting for specific discrete attributes encapsulated by X n +1 . To facilitate the subsequent exposition of our method, it is useful to recall the definition of equalized coverage with the following notation.

Let K denote the number of sensitive attributes, and for each k ∈ [ K ] let M k ∈ N count the possible values of the k -th attribute. Consider a function ϕ : X × { 0 , 1 } K → N d for any subset A ⊆ [ K ] with | A | = d elements, so that ϕ ( x, A ) is a vector of length | A | representing the values of all attributes indexed by A for an individual with features x . In the special case where A is an empty set, ϕ returns a constant. If A is a singleton, e.g., A = { k } for some k ∈ [ K ] , then ϕ ( x, { k } ) ∈ [ M k ] denotes the value of the k -th attribute; e.g., someone's academic degree. More generally, ϕ ( x, { k, l } ) ∈ [ M k ] × [ M l ] , for any distinct k, l ∈ [ K ] , denotes the joint values of two attributes, characterizing a smaller group, such as 'males with a bachelor's degree.'.

When multiple sensitive attributes are involved, i.e., K > 1 , the concept of equalized coverage introduced by [17] can be naturally extended to exhaustive equalized coverage , defined as:

In words, this says ˆ C ( X n +1 ) has valid coverage conditional on all K sensitive attributes. Prediction sets satisfying (2) can be obtained by applying the standard conformal calibration method separately within each of the M = ∏ K k =1 M k groups characterized by a specific combination of the protected attributes represented by ϕ ( X i , [ K ]) ; see Appendix A1 for details. However, a downside of this approach is that the calibration subsets may be too small if M is large, leading to uninformative predictions for even moderate values of K . This limitation forms the starting point of our work.

1.3 Preview of Our Contributions: Adaptive Equalized Coverage

In practice, for a given model and data set, different groups may not exhibit the same need for rigorous equalized coverage guarantees (2), as conformal predictions may be able to approximately achieve the desired coverage even without explicit constraints. Algorithmic bias typically affects only a minority of the population, so standard prediction sets with marginal coverage (1) may approximately satisfy (2) for most groups. Therefore, we propose Adaptively Fair Conformal Prediction (AFCP), a new method that efficiently identifies and addresses groups suffering from algorithmic bias in a data-driven way, adjusting their prediction sets to equalize coverage without sacrificing informativeness.

AFCP involves two main steps. First, as sketched in Figure 1, it carefully selects a sensitive attribute ˆ A ( X n +1 ) ∈ {∅ , { 1 } , . . . , { K }} , based on X n +1 and the data in D . Although AFCP can be extended to select multiple attributes, we begin by focusing on this simpler version for clarity. Intuitively, AFCP searches for the attribute corresponding to the group most negatively affected by algorithmic bias. It may also opt to select no attribute ( ˆ A ( X n +1 ) = ∅ ) in the absence of significant biases.

Test 1

Selects Race,Race=Black,

CPwithvalid coverage onBlack.

Test 2

Trainedmodel

SelectsNone,

CPwithmarginal coverage.

Test 3

Calibration set

SelectsGender&AgeGroup,

Gender=Male,Age Group=>65

CPwithvalidcoverageonMale&>65.

Next, AFCP constructs a prediction set ˆ C ( X n +1 ) for Y n +1 that guarantees the following notion of adaptive equalized coverage at the desired level α ∈ (0 , 1) :

In words, this tells us ˆ C ( X n +1 ) is well-calibrated for the groups defined by the selected attribute ˆ A ( X n +1 ) . It is worth highlighting the key distinctions between (3) and the existing notions of coverage reviewed above. On the one hand, if AFCP identifies no significant bias, selecting ˆ A ( X n +1 ) = ∅ , then (3) reduces to marginal coverage (1), following the convention that ϕ ( X n +1 , ∅ ) is a constant. On the other hand, exhaustive equalized coverage (2) would correspond to simultaneously selecting all possible sensitive attributes instead of only that identified by ˆ A ( X n +1 ) . To clarify the terminology, in this paper we will say that an attribute is sensitive if it may identify a group affected by algorithmic bias. By contrast, a protected attribute is one for which equalized coverage is explicitly sought.

Figure 2 illustrates this intuition through a simulated example. In this scenario, we generate synthetic medical diagnosis data, considering six possible diagnosis labels, and designate race, sex, and age group as potentially sensitive attributes alongside other demographic factors. Notably, the female group, identified by sex, is characterized by fewer samples and higher algorithmic bias, resulting in marginal prediction sets with low group-conditional coverage. By contrast, the model leads to no significant disparities across races and age groups in this dataset.

For two example patients from the critical group, the standard marginal prediction sets fail to cover the true label. Conversely, sets calibrated for exhaustive equalized coverage are too conservative to be informative. By contrast, AFCP generates prediction sets that are both efficient and fair.

Without additional sample splitting, which would be inefficient, constructing informative prediction sets that satisfy (3) is challenging due to potential selection bias from using the same data for attribute selection and conformal calibration. This paper presents a novel solution to address this challenge.

Marginal

AFCP

Exhaustive

{Skin cancer, Flu,

{Diabetes,Flu

Asthma}

(Sexidentified)

Reallabel:Diabetes

{Skin cancer, Flu,

{Asthma,

{Diabetes,

Diabetes}

Stroke,Asthma}

(Sexidentified)

Reallabel:Stroke

Invalid coverage

Approximatelyvalid

Validbutconservative

andefficient

1.4 Table of Contents

Section 2 presents our AFCP method, focusing on the special case in which at most one sensitive attribute may be selected. Section 3 demonstrates the empirical performance of AFCP on synthetic and real data. Section 4 discusses some limitations and suggests ideas for future work.

Additional content is presented in the Appendices. Appendix A1 reviews relevant details of existing approaches. Appendices A2 and A3 present two extensions of our method, respectively enabling the selection of more than one sensitive attribute and providing valid coverage also conditional on the true test label; both extensions involve distinct technical challenges. Additionally, a variation of AFCP designed for outlier detection tasks is detailed in Appendix A4. Appendix A5 contains all mathematical proofs. Appendix A6 explains how to implement our method efficiently and studies its computational cost. Appendix A7 describes the results of numerous additional experiments.

1.5 Related Works

Conformal inference is a very active research area, with numerous methods addressing diverse tasks, including outlier detection [21-23], classification [24-28], and regression [29-31]. Overcoming the limitations of the standard marginal coverage guarantees (1) is a main interest in this field.

Some works have proposed conformity scores designed to seek high feature-conditional coverage while calibrating prediction sets for marginal coverage [27, 30]. Others attempt to mitigate overconfidence while training the ML model [32, 33], and several have developed calibration methods for non-exchangeable data, accounting for possible distribution shifts [34-38]. These works are complementary to ours, as we focus on guaranteeing a new adaptive notion of equalized coverage.

In addition to [17], several other works have considered constructing prediction sets adhering to various notions of equalized coverage and have empirically investigated the performance of conformal predictors in this regard [39]. In the context of regression, [40] and [41] proposed strategies to enhance conditional coverage given several protected attributes, but they targeted a different notion of equalized coverage designed for continuous outcomes. In classification, a classical approach to move beyond marginal coverage is label-conditional coverage, where the 'protected' groups are defined not based on the features X n +1 but by the label itself, Y n +1 [42-44]. As explained in Appendix A3, the method proposed in this paper can also be extended to provide label-conditional coverage.

More closely related to the notion of equalized coverage [17] are the works of [45, 46], which differ from ours as they do not consider the automatic selection of the sensitive groups. To tackle a related challenge due to unknown biased attributes, [47] studied how to identify unfairly treated groups by establishing a simultaneously valid confidence bound on group-wise disparities. In principle, their approach can be integrated within the selection component of our method. Very recently, [48] proposed an elegant method to obtain valid conformal prediction sets for adaptively selected subsets of test cases. While their perspective aligns more closely with ours, their approach and focus differ as they study different selection rules not specifically aimed at mitigating algorithmic bias.

2 Method

2.1 Automatic Attribute Selection

Given a pre-trained classifier, an independent calibration data set D , and a test point Z n +1 = ( X n +1 , Y n +1 ) with an unknown label Y n +1 , we will select (at most) one sensitive attribute, ˆ A ( X n +1 ) ∈ {∅ , { 1 } , . . . , { K }} , according to the following leave-one-out procedure.

For each y ∈ [ L ] , imagine Y n +1 is equal to y , and define an augmented calibration set D ′ y := D∪{ ( X n +1 , y ) } . For each i ∈ [ n +1] , define also the leave-one-out set D ′ y,i := D ′ y \ { ( X i , Y i ) } , with y acting as a placeholder for Y n +1 . Then, for each i ∈ [ n + 1] , we construct a conformal prediction set ˆ C loo y ( X i ) for Y i given X i by calibrating the classifier using the data in D ′ y,i . Any method can be applied for this purpose, although it may be helpful for concreteness to focus on employing the standard approach seeking marginal coverage (1) using the adaptive conformity scores proposed by [27]. Let E y,i denote the binary indicator of whether ˆ C loo y ( X i ) fails to cover Y i :

After evaluating E y,i for all i ∈ [ n +1] , we will assess the leave-one-out miscoverage rate for the worst-off group identified by each sensitive attribute k ∈ [ K ] . That is, we evaluate

Intuitively, δ y,k denotes the maximum miscoverage rate across all groups identified by the k -th attribute. Large values of δ y,k suggest that the k -th attribute may be a sensitive attribute corresponding to at least one group suffering from algorithmic bias.

To assess whether there is evidence of significant algorithmic bias, we can perform a statistical test for the null hypothesis that no algorithmic bias exists. Note that this test can be heuristic since it does not need to be exact for our method to rigorously guarantee (3). Therefore, we do not need to carefully consider the assumptions underlying this test. As a useful heuristic, we define:

and carry out a one-sided t-test for the null hypothesis H 0 : ˆ q y ≤ α against H 1 : ˆ q y > α .

If H 0 is rejected (at any desired level, like 5%), we conclude there exists a group suffering from significant algorithmic bias, and we identify the corresponding attribute through

Otherwise, we set ˆ A ( X n +1 , y ) = ∅ , which corresponds to selecting no attribute. See Algorithm 1 for an outline of this procedure, as a function of the placeholder label y .

After repeating this procedure for each y ∈ [ L ] , the final selected attribute ˆ A ( X n +1 ) is:

Therefore, an attribute is selected if and only if it is consistently flagged by our leave-one-out procedure for all values of the placeholder label y ∈ [ L ] . This approach minimizes the potential arbitrariness due to the use of a placeholder label and is necessary to guarantee that our method constructs prediction sets achieving (3), as discussed in the next section.

Before explaining how our method utilizes the selected sensitive attribute obtained in (7) to construct prediction sets satisfying (3), we pause to make two remarks. First, as long as n is large enough, ˆ A ( X n +1 , y ) is quite stable with respect to both X n +1 and y , as each of these variables plays a relatively small role in determining the leave-one-out miscoverage rates. Therefore, the selected attribute ˆ A ( X n +1 ) given by (7) is also quite stable for different values of X n +1 . This stability will be demonstrated empirically in Section 3. Second, despite its iterative nature, our method can be implemented efficiently; see Appendix A6. Further, if n is very large, our method could be streamlined using cross-validation instead of a leave-one-out approach.

Algorithm 1 Automatic attribute selection using a placeholder test label.

1: Input : calibration data D ; test point with features X n +1 ; list of K sensitive attributes;

2: Input : pre-trained classifier ˆ f ; fixed rule for computing nonconformity scores; level α ∈ (0 , 1) ;

3: Input : placeholder label y ∈ [ L ] .

4: Assume Y n +1 = y and define the augmented data set D ′ y := D∪{ ( X n +1 , y ) } .

5: for i ∈ [ n +1] do

6: Pretend that ( X i , Y i ) is the test point and D ′ y \ { ( X i , Y i ) } is the calibration set.

7: Construct a conformal prediction set ˆ C loo y ( X i ) for Y i .

8: Evaluate the miscoverage indicator E y,i using (4).

9: end for

10: Perform a one-sided test for H 0 : ˆ q y ≤ α vs. H 1 : ˆ q y > α , with ˆ q y defined as in (5).

11: Select the attribute ˆ A ( X n +1 , y ) using (6) if H 0 is rejected, else set ˆ A ( X n +1 , y ) = ∅ .

12: Output : ˆ A ( X n +1 , y ) , either a selected sensitive attribute or an empty set.

2.2 Constructing the Adaptive Prediction Sets

After evaluating ˆ A ( X n +1 , y ) by applying Algorithm 1 with placeholder label y for Y n +1 for all y ∈ [ L ] , and selecting either an empty set or a single attribute ˆ A ( X n +1 ) using (7), AFCP constructs an adaptive prediction set for Y n +1 that satisfies (3) as follows.

First, it constructs a marginal conformal prediction set ˆ C m ( X n +1 ) targeting (1), by applying the standard approach reviewed in Appendix A1. Then, for each y ∈ [ L ] , it constructs a conformal prediction set ˆ C ( X n +1 , ˆ A ( X n +1 , y )) with equalized coverage for the group identified by attribute ˆ A ( X n +1 , y ) , as if it had been fixed. This is achieved by applying the standard marginal method based on a restricted calibration sample indexed by { i ∈ [ n ] : ϕ ( X i , ˆ A ( X n +1 , y )) = ϕ ( X n +1 , ˆ A ( X n +1 , y )) } ; see Algorithm A1 in Appendix A1 for further details. Therefore, note that ϕ ( X i , ˆ A ( X n +1 , y )) becomes equivalent to ˆ C m ( X n +1 ) if ˆ A ( X n +1 , y ) = ∅ . Finally, the AFCP prediction set for Y n +1 is given by:

See Algorithm 2 for an outline of this procedure.

Note that the AFCP set ˆ C ( X n +1 ) given by (8) always contains the marginal set ˆ C m ( X n +1 ) ; this is essential to prove the validity of our approach. Second, in practice the selection ˆ A ( X n +1 , y ) tends to be very consistent for different values of the placeholder label y , as long as the sample size n is large enough; therefore, the union in (8) will typically not lead to a very large prediction set.

Algorithm 2 Adaptively Fair Conformal Prediction (AFCP).

1: Input : calibration data D ; test point with features X n +1 ; list of K sensitive attributes;

Input ˆ

2: : pre-trained classifier f ; fixed rule for computing nonconformity scores; level α ∈ (0 , 1) . 3: for y ∈ [ L ] do

4: Select an attribute ˆ A ( X n +1 , y ) by applying Algorithm 1 with placeholder label y .

5: Construct ˆ C ( X n +1 , A ) by applying Algorithm A1 with the attribute A = ˆ A ( X n +1 , y ) .

6: end for

7: Construct ˆ C m ( X n +1 ) by applying Algorithm A1 without protected attributes.

8: Output : selected attribute ˆ A ( X n +1 ) given by (7) and prediction set ˆ C ( X n +1 ) given by (8).

The following result, proved in Appendix A5, establishes that the prediction sets ˆ C ( X n +1 ) output by AFCP guarantee adaptive equalized coverage (3) with respect to the adaptively selected attribute ˆ A ( X n +1 ) . It is worth emphasizing this result is not straightforward and involves an innovative proof technique to address the lack of exchangeability introduced by the adaptive selection step.

Theorem 1. If { ( X i , Y i ) } n +1 i =1 are exchangeable, the prediction set ˆ C ( X n +1 ) and the selected attribute ˆ A ( X n +1 ) output by Algorithm 2 satisfy the adaptive equalized coverage defined in (3) .

3 Numerical Experiments

3.1 Setup and Benchmarks

This section demonstrates the empirical performance of AFCP, focusing on the implementation described in Section 2, which selects at most one sensitive attribute. Our method is compared with three existing approaches, which utilize the same data, ML model, and conformity scores but produce prediction sets with different guarantees. The first is the marginal benchmark, which constructs prediction sets guaranteeing (1) by applying Algorithm A1 without protected attributes. The second is the exhaustive equalized benchmark, which constructs prediction sets guaranteeing (2) by applying Algorithm A1 with all K sensitive attributes simultaneously protected. The third is a partial equalized benchmark that separately applies Algorithm A1 with each possible protected attribute k ∈ [ K ] , and then takes the union of all such prediction sets. This is an intuitive approach that can be easily verified to provide a coverage guarantee intermediate between (2) and (3), namely:

However, we will see that these prediction sets are often still too conservative in practice.

In addition, we apply a variation of AFCP that always selects one sensitive attribute, regardless of the outcome of the significance test. This method is denoted as AFCP1 in our experiments.

For all methods considered, the classifier is based on a five-layer neural network with linear layers interconnected via a ReLU activation function. The output layer uses a softmax function to estimate the conditional label probabilities. The Adam optimizer and cross-entropy loss function are used in the training process, with a learning rate set at 0.0001. The loss values demonstrate convergence after 100 epochs of training. For all methods, the miscoverage target level is set at α = 0 . 1 .

3.2 Synthetic Data

We generate synthetic classification data to mimic a medical diagnosis task with six possible labels: Skin cancer, Diabetes, Asthma, Stroke, Flu, and Epilepsy. The available features include three sensitive attributes-Age Group, Region, and Color-and six additional non-sensitive covariates. Color is categorized as Blue or Grey, with 10% and 90% marginal frequencies, respectively. The Age Group is cyclically repeated as < 18 , 18 -24 , 25 -40 , 41 -65 , > 65 , and Region is sampled from an i.i.d. multinomial distribution across { West, East, North, South } with equal probabilities. The six non-sensitive features are i.i.d. random samples from a uniform distribution on [0 , 1] . For simplicity, Color is denoted as X 0 and the first non-sensitive feature as X 1 . Conditional on X , the label Y is generated based on a decision tree model that depends only on X 0 and X 1 , as detailed in Appendix A7. This model is designed so that the diagnosis label for individuals with Color equal to Blue is intrinsically harder to predict, mimicking the presence of algorithmic bias.

Figure 3 shows the performance of all methods as a function of the total sample size, ranging from 200 to 2000. In each case, 50% of the samples are used for training and the remaining 50% for calibration. Results are averaged over 500 test points and 100 independent experiments.

While the marginal benchmark produces the smallest prediction sets on average, it leads to significant empirical undercoverage within the Blue group. In contrast, the exhaustive benchmark, which achieves the highest coverage overall, tends to lead to overly conservative and thus uninformative prediction sets, especially for the Blue group. The partial benchmark, though less conservative than the exhaustive method, still generates prediction sets that are too large when the sample size is small.

Our AFCP method and its simpler variation, AFCP1, not only achieve valid coverage for the Blue group but do so with prediction sets that, on average, are not much larger than the marginal ones. AFCP1 is slightly more robust than AFCP when the sample size is very small, as it never fails to select a sensitive attribute. This is advantageous in scenarios where we know there is a sensitive attribute worth equalizing coverage for, though this may not always be the case in practice. See Figure A1 and Table A1 for detailed results with standard errors.

Figure 4 provides additional insight into our method's performance by plotting the selection frequencies of each sensitive attribute as a function of sample size, within the same experiments described in Figure 3. These results show that our method behaves as anticipated. When the sample size and algorithmic bias are both small, AFCP shows more variability in selecting the sensitive attribute,

Coverage for Blue

Average coverage

Average size

200

500

1000

2000

200

500

1000

2000

200

500

1000

2000

2

3

4

5

6

0.900

0.925

0.950

0.975

1.000

0.6

0.8

1.0

Sample size

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

often selecting no attributes. However, as the sample size grows and the undercoverage affecting the Blue group becomes more pronounced, AFCP consistently selects Color as the most sensitive attribute, correctly identifying the main manifestation of algorithmic bias in these data.

Selects Any Attribute

Selects Region

Selects Age Group

Selects Color

200

500

1000

2000

200

500

1000

2000

200

500

1000

2000

200

500

1000

2000

0.00

0.25

0.50

0.75

1.00

Sample size

Method

AFCP

AFCP1

Additional results are presented in Appendix A7.1.1. Figures A1-A3 and Tables A2-A4 summarize the average coverage and prediction set size conditional on each protected attribute. Figures A4-A7 and Tables A5-A8 study the performance of an extension of our method that also provides valid coverage conditional on the true label of the test point.

3.3 Nursery data

We apply AFCP and its benchmarks to the open-domain Nursery data set [49], which was derived from a hierarchical decision model originally developed to rank applications for nursery schools for social science studies. The data encompass 12,960 instances with eight categorical features: Parents' occupation (3 levels), Child's nursery (5 levels), Family form (4 levels), Number of children (1, 2, 3, or more), Housing conditions (3 levels), Financial standing (2 levels), Social conditions (3 levels), and Health status (3 levels). These features are used to predict application ranks across five categories. The variables 'Parents' occupation", 'Number of children", 'Financial standing", 'Social conditions", and 'Health status" are marked as possible sensitive attributes.

To prepare for model training, we executed several pre-processing steps. First, two instances labeled 'recommend" were removed due to the minimal occurrence of this outcome label. Subsequently, we utilized the LabelEncoder function in the sklearn Python package to numerically encode all features and labels. To make the problem more interesting and allow control over the strength of algorithmic bias, we added independent, uniformly distributed noise to the labels of samples with Parents' occupation in the first category, rounding these perturbed labels to the nearest integer. This makes the group corresponding to the first category of Parents' occupation intrinsically more unpredictable and hence more prone to algorithmic bias. To enhance the challenge of making accurate predictions for this group, it was further down-sampled to 10% of its original size.

Coverage for Occupation 1

Average coverage

Average size

200

500

1000

2000

5000

200

500

1000

2000

5000

200

500

1000

2000

5000

1

2

3

4

0.92

0.96

1.00

0.4

0.6

0.8

1.0

Sample size

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Figure 5 summarizes the performance of all approaches as a function of the total number of training and calibration data points, which varies between 200 and 5000. The results are averaged over 500 randomly chosen test points and 100 repeated experiments. In each experiment, 50% of the samples are randomly assigned for training and the remaining 50% for calibration. The marginal benchmark is heavily biased, leading to prediction sets with very low coverage for samples with Parents' occupation in the first category. The exhaustive benchmark is too conservative and results in very large prediction sets unless the sample size is very large. The partial benchmark performs better than the other two benchmarks, but our AFCP method still outperforms it, producing smaller prediction sets with valid coverage even for the hardest-to-predict group. See Table A9 for detailed results.

The results of additional experiments are presented in Appendix A7.1.2. Figures A8-A12 and Tables A10-A14 detail the average coverage and prediction set size for each sensitive attribute. Additionally, Figures A13-A18 and Tables A15-A20 report on the performance of an extended version of AFCP which also ensured valid coverage conditional on the true test label.

3.4 Additional Numerical Experiments

Figures A19-A21 and Table A21 in Appendix A7.1.3 summarize additional experimental results using the open-source COMPAS dataset [50]. Moreover, Appendix A7.2 demonstrates the empirical performance of our AFCP extension for outlier detection; see Figures A22-A36 and Tables A22-A34. These demonstrations involve both synthetic data and the open-domain Adult Income dataset [51]. The experiments with the real-world Adult Income data also include an AFCP extension that allows for the selection of multiple sensitive attributes at the same time.

3.5 Performance of AFCP with Different Sample Sizes

Constructing informative prediction sets that achieve high conditional coverage is inherently more challenging when dealing with smaller sample sizes. By experimenting with different sample sizes, we demonstrate that our AFCP method consistently performs well across different scenarios. For instance, in the experiments depicted in Figures 3-4, when the sample size is as small as 200 , it is difficult to fit an accurate predictive model, assess conditional coverage, and reliably identify the sensitive attribute associated with the lowest coverage. This difficulty is reflected in the relatively large sizes of the prediction sets produced by all methods and the substantial discrepancies between the nominal and empirical conditional coverage. Despite these challenges, our method often succeeds in selecting the correct sensitive attribute, achieving significantly higher conditional coverage compared to the Marginal benchmark, with only a slight increase in average prediction set size.

Moreover, as the sample size increases, our method becomes highly effective at identifying the attribute associated with the lowest conditional coverage, as illustrated in Figure 4. Consequently, our method is able to achieve high conditional coverage with relatively small prediction sets. Overall, these experiments demonstrate that our method offers distinct advantages over existing approaches in both large-sample and small-sample settings.

3.6 Comparison between AFCP and AFCP1

Both AFCP and AFCP1 outperform the benchmark approaches when applied to datasets with small sample sizes, each excelling in different scenarios. AFCP is better suited to situations where there is uncertainty regarding the presence of significant algorithmic bias, while AFCP1 is more effective when prior knowledge suggests that at least one attribute may be biased. For example, in Figure 3, which illustrates a case where one group (Color-Blue) is consistently biased, AFCP1 achieves slightly higher conditional coverage than AFCP. While AFCP exhibits slight undercoverage for the blue group with small sample sizes, it still outperforms the Marginal approach. The occasional inability of AFCP to select a sensitive attribute in small samples reflects the inherent challenges posed by limited datasets. When the method does not select an attribute, it often signifies a lack of sufficient evidence of algorithmic bias, making it reasonable to calibrate the prediction sets solely for marginal coverage.

4 Discussion

This paper presents a practical and statistically principled method to construct informative conformal prediction sets with valid coverage conditional on adaptively selected features. This approach balances efficiency and equalized coverage, which may be particularly useful in applications involving multiple sensitive attributes. While we believe it offers substantial benefits, a potential limitation of this method is that it does not always identify the most relevant sensitive attribute, particularly when working with limited sample sizes. Nevertheless, our empirical results are quite encouraging, demonstrating that AFCP effectively mitigates significant instances of algorithmic bias when the sample size is adequate. Moreover, our method is flexible, allowing for the integration of prior knowledge about which sensitive attributes might require protection against algorithmic bias.

This paper creates several opportunities for further work. Future research could focus on theoretically studying the conditions under which our method can be guaranteed to select the correct sensitive attribute with high probability. Additionally, future extensions might explore implementing different attribute selection procedures, such as those inspired by [47] - within our flexible AFCP framework to delve into the subtle trade-offs associated with different selection algorithms. Moreover, adapting our approach to accommodate different fairness criteria by adaptively adjusting the coverage rate target for each subgroup is another promising area of study. Extending our method to more efficiently handle scenarios with an extremely high number of possible classes is also worthwhile, potentially drawing inspiration from [43]. Furthermore, investigating extensions for classification tasks where the target variable is ordered could be both intriguing and practically useful. In such cases, a naive modification of our method would involve utilizing the discrete convex hull of all components instead of unions of subintervals on the right-hand side of Equation (8). However, developing a more refined approach would be a valuable contribution for future work. Future extensions of our work could focus on adapting to distributional shifts or enhancing the robustness and efficiency of our method under adversarial attacks or contaminated data, potentially drawing connections with [52-56]. Finally, extending our method to accommodate regression tasks with continuous outcomes presents additional computational challenges, but potential solutions could be inspired by [33].

The numerical experiments described in this paper were carried out on a computing cluster. Individual experiments, involving 1000 calibration samples and 500 test samples, required less than 25 minutes and 5GB of memory on a single CPU. The entire project took approximately 100 hours of computing time, and did not involve preliminary or failed experiments.

Software implementing the algorithms and data experiments are available online at https://github. com/FionaZ3696/Adaptively-Fair-Conformal-Prediction .

References

A. Jones, B. Smith, and C. Johnson. Deep learning models for medical diagnosis. Journal of Medical Imaging , 7(2):021008, 2020.

J. K. Brown et al. Using machine learning for job application screening: A comparative analysis. In International Conference on Artificial Intelligence in HR , pages 12-21, 2021.

Jiaming Zeng, Berk Ustun, and Cynthia Rudin. Interpretable classification models for recidivism prediction. J. R. Stat. Soc. (A) , 180(3):689-722, 2017.

Arun K Kuchibhotla and Richard A Berk. Nested conformal prediction sets for classification with applications to probation data. Ann. Appl. Stat. , 17(1):761-785, 2023.

Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In IEEE Conference on Computer Vision and Pattern Recognition , pages 427-436, 2015.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Int. Conf. Mach. Learn. , pages 1321-1330. PMLR, 2017.

Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Innovations in Theoretical Computer Science Conference , page 214-226. Association for Computing Machinery, 2012.

Richard Zemel, Yu (Ledell) Wu, Kevin Swersky, Toniann Pitassi, and Cyntia Dwork. Learning fair representations. In Int. Conf. Mach. Learn. , 2013.

Moritz Hardt, Eric Price, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. In Adv. Neural. Inf. Process. Syst. , volume 29, 2016.

Richard A Berk, Arun Kumar Kuchibhotla, and Eric Tchetgen Tchetgen. Improving fairness in criminal justice algorithmic risk assessments using optimal transport and conformal prediction sets. Sociological Methods & Research , 2021.

Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Adv. Neural Inf. Process. Syst. , 32, 2019.

Harris Papadopoulos, Kostas Proedrou, Vladimir Vovk, and Alex Gammerman. Inductive confidence machines for regression. In Eur. Conf. Mach. Learn. , pages 345-356, 2002.

Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic learning in a random world . Springer, 2005.

Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. J. Am. Stat. Assoc. , 113(523):1094-1111, 2018.

Deborah Hellman. Measuring algorithmic fairness. Virginia Law Review , 106(4):811-866, 2020.

Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data , 5(2):153-163, 2017.

Yaniv Romano, Rina Foygel Barber, Chiara Sabatti, and Emmanuel Candès. With malice toward none: Assessing uncertainty via equalized coverage. Harvard Data Science Review , 2020.

Jianqing Fan, Xin Tong, Yanhui Wu, and Shunan Yao. Neyman-Pearson and equal opportunity: when efficiency meets fairness in classification. arXiv preprint arXiv:2310.01009 , 2023.

Rina Foygel Barber, Emmanuel J Candès, Aaditya Ramdas, and Ryan J Tibshirani. The limits of distribution-free conditional predictive inference. Information and Inference , 10(2):455-482, 2021.

Yonghoon Lee and Rina Barber. Distribution-free inference for regression: discrete, continuous, and in between. Adv. Neural Inf. Process. Syst. , 34:7448-7459, 2021.

Stephen Bates, Emmanuel Candès, Lihua Lei, Yaniv Romano, and Matteo Sesia. Testing for outliers with conformal p-values. Ann. Stat. , 51(1):149 - 178, 2023.

Ariane Marandon, Lihua Lei, David Mary, and Etienne Roquain. Adaptive novelty detection with false discovery rate guarantee. Ann. Stat. , 52(1):157-183, 2024.

Ziyi Liang, Matteo Sesia, and Wenguang Sun. Integrative conformal p-values for out-ofdistribution testing with labelled outliers. J. R. Stat. Soc. (B) , page qkad138, 2024.

Jing Lei, James Robins, and Larry Wasserman. Distribution-free prediction sets. J. Am. Stat. Assoc. , 108(501):278-287, 2013.

Mauricio Sadinle, Jing Lei, and Larry Wasserman. Least ambiguous set-valued classifiers with bounded error levels. J. Am. Stat. Assoc. , 114(525):223-234, 2019.

Aleksandr Podkopaev and Aaditya Ramdas. Distribution-free uncertainty quantification for classification under label shift. In Uncertainty in Artificial Intelligence , pages 844-853. PMLR, 2021.

Yaniv Romano, Matteo Sesia, and Emmanuel J. Candès. Classification with valid and adaptive coverage. Adv. Neural Inf. Process. Syst. , 33, 2020.

Anastasios Nikolas Angelopoulos, Stephen Bates, Michael Jordan, and Jitendra Malik. Uncertainty sets for image classifiers using conformal prediction. In Int. Conf. Learn. Represent. , 2021.

Jing Lei and Larry Wasserman. Distribution-free prediction bands for non-parametric regression. J. R. Stat. Soc. (B) , 76(1):71-96, 2014.

Yaniv Romano, Evan Patterson, and Emmanuel J Candès. Conformalized quantile regression. In Adv. Neural Inf. Process. Syst. , pages 3538-3548, 2019.

Matteo Sesia and Yaniv Romano. Conformal prediction using conditional histograms. Adv. Neural Inf. Process. Syst. , 34, 2021.

Bat-Sheva Einbinder, Yaniv Romano, Matteo Sesia, and Yanfei Zhou. Training uncertaintyaware classifiers with conformalized deep learning. In Adv. Neural Inf. Process. Syst. , volume 35, 2022.

Ziyi Liang, Yanfei Zhou, and Matteo Sesia. Conformal inference is (almost) free for neural networks trained with early stopping. In Int. Conf. Mach. Learn. , 2023.

Rina Foygel Barber, Emmanuel J Candès, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. Ann. Stat. , 51(2):816-845, 2023.

Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candès, and Aaditya Ramdas. Conformal prediction under covariate shift. Adv. Neural Inf. Process. Syst. , 32, 2019.

Isaac Gibbs and Emmanuel Candès. Conformal inference for online prediction with arbitrary distribution shifts. arXiv preprint arXiv:2208.08401 , 2022.

Yachong Yang, Arun Kumar Kuchibhotla, and Eric Tchetgen Tchetgen. Doubly robust calibration of prediction sets under covariate shift. J. R. Stat. Soc. (B) , page qkae009, 2024.

Ziyi Liang, Tianmin Xie, Xin Tong, and Matteo Sesia. Structured conformal inference for matrix completion with applications to group recommender systems. arXiv preprint arXiv:2404.17561 , 2024.

Charles Lu, Andreanne Lemay, Ken Chang, Katharina Hoebel, and Jayashree Kalpathy-Cramer. Fair conformal predictors for applications in medical imaging. AAAI Conference on Artificial Intelligence , 36:12008-12016, 06 2022.

Fangxin Wang, Lu Cheng, Ruocheng Guo, Kay Liu, and Philip S. Yu. Equal opportunity of coverage in fair regression. In Adv. Neural Inf. Process. Syst , 2023.

Meichen Liu, Lei Ding, Dengdeng Yu, Wulong Liu, Linglong Kong, and Bei Jiang. Conformalized fairness via quantile regression. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Adv. Neural Inf. Process. Syst. , 2022.

Vladimir Vovk, David Lindsay, Ilia Nouretdinov, and Alex Gammerman. Mondrian confidence machine. Technical report, Royal Holloway, University of London, 2003. On-line Compression Modelling project.

Tiffany Ding, Anastasios Nikolas Angelopoulos, Stephen Bates, Michael Jordan, and Ryan Tibshirani. Class-conditional conformal prediction with many classes. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview. net/forum?id=mYz6ApeU4J .

Tuve Löfström, Henrik Boström, Henrik Linusson, and Ulf Johansson. Bias reduction through conditional conformal prediction. Intell. Data Anal. , 19(6):1355-1375, nov 2015. ISSN 1088-467X.

Christopher Jung, Georgy Noarov, Ramya Ramalingam, and Aaron Roth. Batch multivalid conformal prediction. In Int. Conf. Learn. Represent. , 2023.

Isaac Gibbs, John J Cherian, and Emmanuel J Candès. Conformal prediction with conditional guarantees. arXiv preprint arXiv:2305.12616 , 2023.

John J. Cherian and Emmanuel J. Candès. Statistical inference for fairness auditing. arXiv preprint arXiv:2305.03712 , 2023.

Ying Jin and Zhimei Ren. Confidence on the focal: Conformal prediction with selectionconditional coverage. arXiv preprint arXiv:2403.03868 , 2024.

Vladislav Rajkovic. Nursery. UCI Machine Learning Repository, 1997. DOI: https://doi.org/10.24432/C5P88W, License: CC-BY 4.0.

Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. How we analyzed the compas recidivism algorithm. https://www.propublica.org/article/ how-we-analyzed-the-compas-recidivism-algorithm , 2016.

Barry Becker and Ronny Kohavi. Adult. UCI Machine Learning Repository, 1996. DOI: https://doi.org/10.24432/C5XW20, License: CC-BY 4.0.

Bat-Sheva Einbinder, Stephen Bates, Anastasios N Angelopoulos, Asaf Gendler, and Yaniv Romano. Conformal prediction is robust to label noise. arXiv preprint arXiv:2209.14295 , 2022.

Ge Yan, Yaniv Romano, and Tsui-Wei Weng. Provably robust conformal prediction with improved efficiency. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=BWAhEjXjeG .

Asaf Gendler, Tsui-Wei Weng, Luca Daniel, and Yaniv Romano. Adversarially robust conformal prediction. In International Conference on Learning Representations , 2022. URL https: //openreview.net/forum?id=9L1BsI4wP1H .

Shai Feldman, Bat-Sheva Einbinder, Stephen Bates, Anastasios N. Angelopoulos, Asaf Gendler, and Yaniv Romano. Conformal prediction is robust to dispersive label noise. In Harris Papadopoulos, Khuong An Nguyen, Henrik Boström, and Lars Carlsson, editors, Proceedings of the Twelfth Symposium on Conformal and Probabilistic Prediction with Applications , volume 204 of Proceedings of Machine Learning Research , pages 624-626. PMLR, 13-15 Sep 2023. URL https://proceedings.mlr.press/v204/feldman23a.html .

Matteo Sesia, Y. X. Rachel Wang, and Xin Tong. Adaptive conformal classification with noisy labels, 2024. URL https://arxiv.org/abs/2309.05092 .

Glenn Shafer and Vladimir Vovk. A tutorial on conformal inference. J. Mach. Learn. Res. , 9(3), 2008.

Tuve Löfström, Henrik Boström, Henrik Linusson, and Ulf Johansson. Bias reduction through conditional conformal prediction. Intell. Data Anal. , 19(6):1355-1375, 11 2015.

COMPAS Data Set. https://github.com/propublica/compas-analysis . Accessed: July, 2024.

A1 Review of Existing Conformal Classification Methods

Algorithm A1 outlines the standard approach for constructing conformal prediction sets with equalized coverage with respect to a fixed list of protected attributes [17]. In the special case where the list of protected attributes is empty, this method reduces to the standard approach for constructing prediction sets with marginal coverage.

Algorithm A1 Conformal classification with equalized coverage for fixed protected attributes.

1: Input : calibration data D ; test point with features X n +1 ; list of protected attributes A ;

2: Input : pre-trained classifier ˆ f ; pre-defined rule for computing nonconformity scores;

3: Input : nominal level α ∈ (0 , 1) .

4: Define the calibration subset

5: for y ∈ [ L ] do

6: Compute the nonconformity scores ˆ S y i for i ∈ I ( X n +1 , A ) ∪ { ( X n +1 , y ) } using ˆ f .

7: Compute the conformal p-value:

9: Construct a prediction set using ˆ C ( X n +1 ) = { y ∈ [ L ] : ˆ u y ( X n +1 ) ≥ α } .

8: end for

10: Output : a prediction set ˆ C ( X n +1 ) .

In the context of outlier detection, Algorithm A2 reviews the standard approach for computing conformal p-values achieving valid false positive rate (FPR) control conditional a fixed list of protected attributes.

Algorithm A2 Conformal p-value with equalized FPR for fixed protected attributes.

1: Input : calibration data D ; test point Z n +1 ; list of protected attributes A ;

2: Input : pre-trained one-class classifier ˆ f ; pre-defined rule for computing nonconformity scores;

3: Input : nominal level α ∈ (0 , 1) .

4: Define the calibration subset

5: Compute the nonconformity scores ˆ S i for i ∈ I ( Z n +1 , A ) ∪ { Z n +1 } using ˆ f .

6: Compute the conformal p-value:

7: Output : a conformal p-value ˆ u ( Z n +1 ) .

A2 Methodology Extension: AFCP with Multiple Selected Attributes

This section introduces an extension of AFCP that enables the selection of more than one sensitive attribute. For simplicity, we focus on the selection of up to two attributes. The methodology for selecting more than two attributes can be extended in a similar manner, as explained later.

A2.1 Automatic Multiple Attribute Selections

Given a pre-trained classification model, an independent calibration data set D with size n , and a test point Z n +1 = ( X n +1 , Y n +1 ) with an unknown label Y n +1 , Algorithm 1 in Section 2.1 introduces

the AFCP component to select one sensitive attribute according to the leave-one-out procedure. Intuitively, selecting two sensitive attributes requires executing Algorithm 1 twice. However, during the second iteration, the sensitive attribute list is restricted to exclude the most critical protected attribute selected in the first round.

̸

Specifically, for each placeholder label y ∈ [ L ] , assuming Y n +1 = y , Algorithm 1 is run with all K sensitive attributes for the first iteration to obtain the first selected attribute ˆ A 1 ( X n +1 , y ) ∈ {∅ , { 1 } , . . . , { K }} . If ˆ A 1 ( X n +1 , y ) = ∅ , Algorithm 1 is run again using the sensitive attributes [ K ] \ ˆ A 1 ( X n +1 , y ) and one can get the second selected attribute ˆ A 2 ( X n +1 , y ) ∈ {∅ , { 1 } , . . . , { K }}\ ˆ A 1 ( X n +1 , y ) . Therefore, the identified attributes for test feature X n +1 with placeholder y for the test label is the union of the two ˆ A ( X n +1 , y ) = ˆ A 1 ( X n +1 , y ) ∪ ˆ A 2 ( X n +1 , y ) . This procedure is outlined in Algorithm A3.

After repeating this procedure for each y ∈ [ L ] , the final selected attribute ˆ A ( X n +1 ) is:

Algorithm A3 Two attributes selection using a placeholder test label.

1: Input : calibration data D ; test point with features X n +1 ; list of K sensitive attributes;

2: Input : pre-trained classifier ˆ f ; pre-defined rule for computing nonconformity scores;

3: Input : nominal level α ∈ (0 , 1) , placeholder label y ∈ [ L ] .

4: Select the first attribute ˆ A 1 ( X n +1 , y ) by applying Algorithm 1 with placeholder label y and sensitive attributes [ K ] .

̸

5: if ˆ A 1 ( X n +1 , y ) = ∅ then

6: Select the second attribute ˆ A 2 ( X n +1 , y ) by applying Algorithm 1 with placeholder label y and sensitive attributes [ K ] \ ˆ A 1 ( X n +1 , y ) .

7: end if

8: Output : ˆ A ( X n +1 , y ) = ˆ A 1 ( X n +1 , y ) ∪ ˆ A 2 ( X n +1 , y ) , which is a set of an empty set, or a set including one or two selected sensitive attribute(s).

A2.2 Constructing the Adaptive Prediction Sets

After selecting a subset of attributes ˆ A ( X n +1 ) using (A10), which may be empty, or include one or two attributes, AFCP constructs an adaptive prediction set for Y n +1 that satisfies (3) as follows.

First, it constructs a marginal conformal prediction set ˆ C m ( X n +1 ) targeting (1), by applying Algorithm A1 without protected attributes. Then, for each y ∈ [ L ] , it constructs a conformal prediction set ˆ C ( X n +1 , ˆ A ( X n +1 , y )) with equalized coverage for the group jointly identified by attributes ˆ A ( X n +1 , y ) . This can be achieved by applying Algorithm A1 with protected attributes ˆ A ( X n +1 , y ) . Lastly, it constructs a conformal prediction set ˆ C eq ( X n +1 , ℓ ) with equalized coverage separately for each protected attribute ℓ ∈ ∪ y ∈ [ L ] ˆ A ( X n +1 , y ) , by applying the standard approach in Algorithm A1 on the subsets indexed by { i ∈ [ n ] : ϕ ( X i , { ℓ } ) = ϕ ( X n +1 , { ℓ } ) } .

Finally, the AFCP prediction set constructed using up to two selected attributes is given as:

See Algorithm A4 for an outline of this AFCP extension.

Theorem A1. If { ( X i , Y i ) } n +1 i =1 are exchangeable random samples, then the conformal prediction set ˆ C ( X n +1 ) and the selected attributes ˆ A ( X n +1 ) output by Algorithm A4 satisfy the adaptive equalized coverage defined in (3) .

Algorithm A4 AFCP with two selected attributes.

1: Input : calibration data D ; test point with features X n +1 ; list of K sensitive attributes;

2: Input : pre-trained classifier ˆ f ; pre-defined rule for computing nonconformity scores;

3: Input : nominal level α ∈ (0 , 1) .

4: for y ∈ [ L ] do

5: Select attribute(s) ˆ A ( X n +1 , y ) by applying Algorithm A3 with placeholder label y .

6: Construct ˆ C ( X n +1 , A ) by applying Algorithm A1 with protected attribute(s) A = ˆ A ( X n +1 , y ) .

7: end for

8: for ℓ ∈ ∪ y ∈ [ L ] ˆ A ( X n +1 , y ) do

9: Construct ˆ C eq ( X n +1 , ℓ ) by applying Algorithm A1 with protected attribute { ℓ } .

10: end for

11: Construct ˆ C m ( X n +1 ) by applying Algorithm A1 without protected attributes.

12: Define the final selected attribute(s) ˆ A ( X n +1 ) using Equation (A10).

13: Define the final prediction set ˆ C ( X n +1 ) using Equation (A11).

14: Output : ˆ A ( X n +1 ) and ˆ C ( X n +1 ) .

A3 Methodology Extension: AFCP with Label Conditional Coverage

This section extends the AFCP method with adaptive equalized coverage that is also conditional on the true test label. We focus on the main implementation of AFCP, where it can select up to one sensitive attribute.

First, the label conditional counterparts of the marginal coverage, the exhaustive equalized coverage, and the adaptive equalized coverage are defined. The label-conditional counterpart of marginal coverage is defined as:

Intuitively, this coverage ensures that the prediction sets constructed are valid for each group with the test label y for all possible values of y ∈ [ L ] . This coverage offers a stronger assurance than marginal coverage in classification contexts. However, it overlooks scenarios where groups, identified by feature attributes, may suffer adverse effects from prediction biases. To address these concerns, one can aim for label-conditional exhaustive equalized coverage [17], defined as:

Achieving this label conditional exhaustive equalized coverage involves applying the standard conformal classification method (outlined in Algorithm A1) separately within each of the groups characterized by every possible combination of sensitive attributes and test labels. Hence, this approach can become overly conservative when a large number of sensitive attributes or response labels are presented.

Our AFCP method strikes a balance between the two approaches to achieve label-conditional adaptive equalized coverage:

which guarantees that the true test label is contained within the prediction sets with high probability for the groups defined by the selected attribute ˆ A ( X n +1 ) and the test label y for every y ∈ [ L ] .

A3.1 Automatic Attribute Selection

Given a pre-trained classification model and an independent calibration data set D with size n , for each placeholder label y ∈ [ L ] for Y n +1 , the AFCP method with label-conditional adaptive equalized coverage (A14) selects the sensitive attribute ˆ A ( X n +1 , y ) by simply applying Algorithm 1 using the label-restricted calibration data D y = { i ∈ [ n ] : Y i = y } . After repeating this process for each y ∈ [ L ] , the final selected attribute ˆ A ( X n +1 ) is again given by

A3.2 Constructing the Adaptive Prediction Sets

After selecting ˆ A ( X n +1 , y ) by applying Algorithm 1 with placeholder label y based on the labelrestricted calibration data D y = { i ∈ [ n ] : Y i = y } , and selecting a sensitive attribute ˆ A ( X n +1 ) , AFCP constructs an adaptive prediction set for Y n +1 that satisfies (A14) as follows.

For each y ∈ [ L ] , it firstly construct a conformal prediction set ˆ C lc ( X n +1 , y ) by applying Algorithm A1 using the label-restricted calibration set D y without considering protected attributes. Then, it constructs another conformal prediction set ˆ C ( X n +1 , ˆ A ( X n +1 , y )) with equalized coverage for the group identified by both the selected attribute ˆ A ( X n +1 , y ) and label y . This can be achieved by applying Algorithm A1 based on a subset of the calibration samples indexed by I ( X n +1 , y ) = { i ∈ D y : ϕ ( X i , ˆ A ( X n +1 , y )) = ϕ ( X n +1 , ˆ A ( X n +1 , y )) } . Lemma A1 shows that, for any given placeholder label, the prediction set constructed in this step satisfies the label conditional adaptive equalized coverage as long as the selected variable using that placeholder label is fixed.

Lemma A1. If { ( X i , Y i ) } n +1 i =1 are exchangeable and the selected attribute ˆ A ( X n +1 , y ) is fixed for some placeholder label y , then, the prediction set ˆ C ( X n +1 , ˆ A ( X n +1 , y )) constructed by calibrating on I ( X n +1 , y ) satisfies

for any ˜ y ∈ [ L ]

Lastly, the final AFCP prediction set is obtained by:

The procedures to form AFCP prediction sets with the label-conditional adaptive equalized coverage (A14) is summarized in Algorithm A5.

Algorithm A5 AFCP with label-conditional adaptive equalized coverage (A14).

1: Input : calibration data D ; test point with features X n +1 ; list of K sensitive attributes;

2: Input : pre-trained classifier ˆ f ; pre-defined rule for computing nonconformity scores;

3: Input : nominal level α ∈ (0 , 1) .

4: for y ∈ [ L ] do

5: Define the label-restricted calibration set D y = { i ∈ [ n ] : Y i = y } .

6: Select an attribute ˆ A ( X n +1 , y ) by applying Algorithm 1 with placeholder label y on D y .

7: Construct ˆ C ( X n +1 , ˆ A ( X n +1 , y )) by applying Algorithm A1 with protected attributes ˆ A ( X n +1 , y ) on D y .

8: Construct ˆ C lc ( X n +1 , y ) by applying Algorithm A1 on D y .

9: end for

10: Define the final selected attribute ˆ A ( X n +1 ) using Equation (A15).

11: Define the final prediction set ˆ C ( X n +1 ) using Equation (A16).

12: Output : ˆ A ( X n +1 ) and ˆ C ( X n +1 ) .

Theorem A2. If { ( X i , Y i ) } n +1 i =1 are exchangeable random samples, then the conformal prediction set ˆ C ( X n +1 ) and the selected attribute ˆ A ( X n +1 ) output by Algorithm A5 satisfy the label-conditional adaptive equalized coverage defined in (A14) .

A4 Methodology Extension: AFCP for Outlier Detection

Consider a dataset D = { Z i } n i =1 containing n sample points drawn exchangeably from an unknown distribution P Z . Consider an additional test point Z n +1 . In the outlier detection problems, our AFCP method aims to study whether Z n +1 ∼ P Z by constructing a valid conformal p-value ˆ u ( Z n +1 ) conditional on the group identified by the selected attribute ˆ A ( Z n +1 ) , that is:

for any α ∈ (0 , 1) . Intuitively, this guarantees that, on groups defined by the selected attribute ˆ A ( Z n +1 ) , the conformal p-value is super-uniform, therefore controlling the FPR (the probability of rejecting the null hypothesis that Z n +1 is an inlier when it is true) below α .

A4.1 Automatic Attribute Selection

Given a pre-trained one-class classifier ˆ f , an independent calibration dataset D , and a test point Z n +1 , our method selects a sensitive attribute ˆ A ( Z n +1 ) ∈ {∅ , { 1 } , . . . , { K }} according to the following leave-one-out procedure. Sometimes, no attribute may be selected, as denoted by ˆ A ( Z n +1 ) = ∅ .

Define an augmented calibration set D ′ := D ∪ { Z n +1 } . For each i ∈ [ n + 1] , define also the leave-one-out set D ′ i := D ′ \ { Z i } . Then, for each i ∈ [ n +1] , we compute a conformal p-value ˆ u loo ( Z i ) for Z i using the data in D ′ i to test if Z i is an outlier. This can be accomplished by running Algorithm A2, using D ′ i as the calibration data and Z i as the test point, and with the convention that smaller nonconformity scores suggest Z i is more likely to be an outlier. Any nonconformity scores can be utilized here. For concreteness, we focus on using the adaptive conformity scores proposed by [27]. Small ˆ u loo ( Z i ) provides stronger evidence to reject the null hypothesis H 0 ,i : Z i is an inlier. Let E i denote the binary indicator of whether H 0 ,i is rejected:

After evaluating E i for all i ∈ [ n +1] , we will assess the leave-one-out FPR for the worst-off group identified by each sensitive attribute k ∈ [ K ] . That is, we evaluate

Intuitively, δ k denotes the maximum FPR across all groups identified by the k -th attribute, as estimated by the leave-one-out simulation carried out under the assumption that Z n +1 is an inlier. Large values of δ k suggest that the k -th attribute may be a sensitive attribute corresponding to at least one group suffering from algorithmic bias.

To assess whether there is evidence of significant algorithmic bias, we can perform a statistical test for the null hypothesis that no algorithmic bias exists. We define:

and carry out a one-sided t-test for the null hypothesis H 0 : ˆ q ≤ α against H 1 : ˆ q > α .

If H 0 is rejected (at any desired level, such as 5%), we conclude there exists a group suffering from significant algorithmic bias, and we identify the corresponding attribute through

Otherwise, we set ˆ A ( Z n +1 ) = ∅ , which corresponds to selecting no attribute. See Algorithm A6 for an outline of this procedure.

A4.2 Evaluating the Adaptive Conformal P-Value

After selecting either a single attribute or an empty set ˆ A ( X n +1 ) that corresponds to at least one group suffering from algorithmic bias, the next step of our AFCP method is to compute an adaptive conformal p-value for testing whether Z n +1 is an outlier that satisfies (A17). This can be simply achieved by applying the standard conformal method outlined in Algorithm A2 based on a restricted calibration sample indexed by I ( ˆ A ( Z n +1 )) = { i ∈ [ n ] : ϕ ( Z i , ˆ A ( Z n +1 )) = ϕ ( Z n +1 , ˆ A ( Z n +1 )) } , See Algorithm A7 for a summary of AFCP for outlier detection tasks.

Theorem A3. If { Z i } n +1 i =1 are exchangeable random samples, the conformal p-value ˆ u ( Z n +1 ) and the selected attribute ˆ A ( Z n +1 ) output by Algorithm A7 satisfy (A17) .

Algorithm A6 Automatic attribute selection for outlier detection.

1: Input : calibration data D ; test point Z n +1 ; list of K sensitive attributes;

2: Input : pre-trained one-class classifier ˆ f ; pre-defined rule for computing nonconformity scores;

3: Input : nominal level α ∈ (0 , 1) .

4: Define the augmented data set D ′ = D∪{ Z n +1 } .

5: for i ∈ [ n +1] do

6: Pretend that Z i is the test point and D ′ \ { Z i } is the calibration set.

7: Compute a conformal p-value ˆ u loo ( Z i ) .

8: Evaluate the false positive indicator E i using (A18).

9: end for

10: Compute ˆ q using (A20).

11: Perform a one-sided test for H 0 : ˆ q ≤ α vs. H 1 : ˆ q > α .

12: Select the attribute ˆ A ( Z n +1 ) using (A21) if H 0 is rejected, else set ˆ A ( Z n +1 ) = ∅ .

13: Output : ˆ A ( Z n +1 ) , either a selected sensitive attribute or an empty set.

Algorithm A7 AFCP for outlier detection.

1: Input : calibration data D ; test point Z n +1 ; list of K sensitive attributes;

2: Input : pre-trained one-class classifier ˆ f ; pre-defined rule for computing nonconformity scores;

3: Input : nominal level α ∈ (0 , 1) .

4: Select a sensitive attribute ˆ A ( Z n +1 ) by applying Algorithm A6.

5: Evaluate ˆ u ( Z n +1 ) by applying Algorithm A2 with protected attribute ˆ A ( Z n +1 ) .

6: Output : ˆ u ( Z n +1 ) .

A4.3 AFCP for Outlier Detection with Multiple Selected Attributes

AFCP for outlier detection problems can be readily extended to select J sensitive attributes where J > 1 . This is achieved by repeatedly applying the single attribute selection procedure described in Algorithm A6 J times. Each time, the algorithm selects a (possibly empty) subset of attributes ˆ A ( Z n +1 ) j from the list of sensitive attributes excluding the previously selected attribute ˆ A ( Z n +1 ) j -1 . The final set of selected attributes is given by ˆ A ( Z n +1 ) = ∪ J j =1 ˆ A ( Z n +1 ) j . See Algorithm A8 for an outline of this procedure.

Algorithm A8 Multiple attributes selection for outlier detection.

1: Input : calibration data D ; test point Z n +1 ; list of K sensitive attributes;

2: Input : pre-trained one-class classifier ˆ f ; pre-defined rule for computing nonconformity scores;

3: Input : nominal level α ∈ (0 , 1) ; number of selected attributes J .

4: Denote K 0 = [ K ] .

5: for j ∈ { 1 , . . . , J } do

6: Select an attribute ˆ A ( Z n +1 ) j by applying Algorithm A6 with sensitive attributes K j -1 .

7: Update the list of sensitive attributes K j = K j -1 \ ˆ A ( Z n +1 ) j .

8: end for

9: Output : ˆ A ( Z n +1 ) = ∪ J j =1 ˆ A ( Z n +1 ) j , an empty set or a set of selected attributes.

After selecting a set of attributes ˆ A ( Z n +1 ) , which might be empty or include one or more sensitive attributes, AFCP constructs an adaptive conformal p-value satisfying (A17). This can be easily achieved by applying Algorithm A2 with protected attributes ˆ A ( Z n +1 ) . Algorithm A9 summarizes the AFCP implementation for outlier detection that allows selecting multiple protected attributes.

Theorem A4. If { Z i } n +1 i =1 are exchangeable random samples, the conformal p-value ˆ u ( Z n +1 ) and selected attributes ˆ A ( Z n +1 ) output by Algorithm A9 satisfy (A17) .

Algorithm A9 AFCP for outlier detection with multiple selected attributes.

1: Input : calibration data D ; test point Z n +1 ; list of K sensitive attributes;

2: Input : pre-trained one-class classifier ˆ f ; pre-defined rule for computing nonconformity scores;

3: Input : nominal level α ∈ (0 , 1) ; number of selected attributes J .

4: Select up to J sensitive attributes ˆ A ( Z n +1 ) by applying Algorithm A8.

5: Evaluate ˆ u ( Z n +1 ) by applying Algorithm A2 with protected attribute ˆ A ( Z n +1 ) .

6: Output : ˆ u ( Z n +1 ) .

A5 Mathematical Proofs

Proof of Theorem 1. Consider an imaginary oracle that has access to the true value of Y n +1 . Denote ˆ A o ( X n +1 , Y n +1 ) as the sensitive attribute selected by this oracle by applying Algorithm 1 with the true Y n +1 instead of a placeholder label. Let ˆ C o ( X n +1 , ˆ A o ( X n +1 , Y n +1 )) represent the corresponding output prediction set by applying Algorithm A1 with the protected attribute ˆ A o ( X n +1 , Y n +1 ) . Consider also ˆ C m ( X n +1 ) , the standard prediction set with marginal coverage (1).

The main idea of our proof is to connect the output prediction set ˆ C ( X n +1 ) and selected attribute ˆ A ( X n +1 ) from Algorithm 2 to those of the imaginary oracle described above. Throughout this proof, we adopt the convention that ϕ ( X n +1 , ∅ ) = 0 .

To establish this connection, note that the attribute ˆ A ( X n +1 ) selected by Algorithm 2 is either empty, ˆ A ( X n +1 ) = ∅ , or a singleton, ˆ A ( X n +1 ) = { k } for some k ∈ [ K ] . In the latter case, ˆ A ( X n +1 ) = ˆ A ( X n +1 , ˜ y ) , ∀ ˜ y ∈ [ L ] , and thus ˆ A ( X n +1 ) = ˆ A o ( X n +1 , Y n +1 ) almost-surely. Therefore,

where the last inequality follows from the facts that ˆ C m ( X n +1 ) ⊆ ˆ C ( X n +1 ) and ˆ C o ( X n +1 , ˆ A o ( X n +1 , Y n +1 )) ⊆ ˆ C ( X n +1 ) almost-surely. Next, we only need to separately lowerbound by 1 -α the two terms on the right-hand-side of (A22).

The first part of the remaining task is trivial. It is already well-known that P ( Y n +1 ∈ ˆ C m ( X n +1 )) ≥ 1 -α ; see [13, 57].

To complete the second part of the remaining task, note that the oracle-selected attribute ˆ A o ( X n +1 , Y n +1 ) is invariant to any permutations of the exchangeable data indexed by [ n +1] . Therefore, the data points are also exchangeable conditional on the groups defined by ˆ A o ( X n +1 , Y n +1 ) . This means we can imagine the protected attribute ˆ A o ( X n +1 , Y n +1 ) is fixed, and the oracle prediction set ˆ C o ( X n +1 , ˆ A o ( X n +1 , Y n +1 )) is simply obtained by applying Algorithm A1 to exchangeable data

using a fixed protected attribute. This procedure is the same as the main algorithm in [17] and has guaranteed coverage above 1 -α ; see Theorem 1 in [17].

Proof of Theorem A1. Similar to the proof of Theorem 1, consider an imaginary oracle that has access to the true value of Y n +1 . Denote ˆ A o ( X n +1 , Y n +1 ) as the (possibly empty, one, or two) sensitive attribute(s) selected by this oracle by applying Algorithm A3 with the true Y n +1 . Let ˆ C o ( X n +1 , ˆ A o ( X n +1 , Y n +1 )) represent the output prediction set by applying Algorithm A1 with the protected attribute(s) ˆ A o ( X n +1 , Y n +1 ) . Consider also ˆ C m ( X n +1 ) , the standard prediction set with marginal coverage (1), and ˆ C eq ( X n +1 , ℓ ) the prediction set with valid coverage conditional on groups identified by a fixed attribute { ℓ } .

The key idea of our proof is again to connect the practical prediction set ˆ C ( X n +1 ) and selected protected attributes ˆ A ( X n +1 ) of Algorithm A4 to those of the imaginary oracle described above. Throughout this proof, we adopt the convention that ϕ ( X n +1 , ∅ ) = 0 .

To establish this connection, note that the attribute(s) ˆ A ( X n +1 ) selected by applying Algorithm A3 falls into one of the three possible cases almost surely: (a) ˆ A ( X n +1 ) = ∅ , (b) ˆ A ( X n +1 ) = ˆ A o ( X n +1 , Y n +1 ) , and (c) ˙ A ( X n +1 ) = ˆ A ( X n +1 ) ⊂ ˆ A o ( X n +1 , Y n +1 ) . The last scenario happens when the oracle selects two attributes, and the ˆ A ( X n +1 ) contains only one of them. For clarify of the notation, we denote the last case using ˙ A ( X n +1 ) .

Therefore,

where the last inequality follows from the facts that ˆ C m ( X n +1 ) ⊆ ˆ C ( X n +1 ) , ˆ C o ( X n +1 , ˆ A o ( X n +1 , Y n +1 )) ⊆ ˆ C ( X n +1 ) , and ∪ ℓ ∈ ˆ A o ( X n +1 ,Y n +1 ) ˆ C eq ( X n +1 , ℓ ) ⊆ ˆ C ( X n +1 ) almost-surely.

Next, we show how to separately lower-bound by 1 -α the three terms on the right-hand-side of (A23).

The lower bounds of the first and second terms have been proved in theorem 1. We will focus on lowerbounding the coverage of the third term. Because the oracle-selected attribute(s) ˆ A o ( X n +1 , Y n +1 ) are invariant to any permutation of the exchangeable data indexed by [ n +1] , we can treat each of the two elements in ˆ A o ( X n +1 , Y n +1 ) as fixed. Without loss of generality, let ℓ 1 and ℓ 2 denote the first

and the second element respectively, ˙ A ( X n +1 ) = ℓ 1 or ˙ A ( X n +1 ) = ℓ 2 with probability 1 . Then,

where the inequality of the last line of (A24) is proved in [17] with fixed attribute ℓ 1 and ℓ 2 respectively. Lastly, realize that ∪ ℓ ∈ ˆ A o ( X n +1 ,Y n +1 ) ˆ C eq ( X n +1 , ℓ ) ⊆ ∪ ℓ ∈∪ L y =1 ˆ A ( X n +1 ,y ) ˆ C eq ( X n +1 , ℓ ) ⊆ ˆ C ( X n +1 ) almost-surely, the proof is completed.

Proof of Theorem A2. Similar to the proof of Theorem 1, consider an imaginary oracle that has access to the true value of Y n +1 . Denote ˆ A o ( X n +1 , Y n +1 ) as the sensitive attribute selected by this oracle by applying Algorithm 1 based on a subset of the calibration data indexed by D Y n +1 = { i ∈ [ n ] : Y i = Y n +1 } . Let ˆ C o ( X n +1 , ˆ A o ( X n +1 , Y n +1 )) represent the output prediction set by applying Algorithm A1 with the protected attribute ˆ A o ( X n +1 , Y n +1 ) . Consider also ˆ C lc ( X n +1 , Y n +1 ) , the prediction set with label-conditional coverage obtained by running Algorithm A1 using D Y n +1 without any protected attributes.

To establish this connection between the output prediction set ˆ C ( X n +1 ) and the selected attribute ˆ A ( X n +1 ) of Algorithm A5 and these of the imaginary oracle, note that the selected attribute ˆ A ( X n +1 ) must be ˆ A ( X n +1 ) = ∅ or ˆ A ( X n +1 ) = ˆ A o ( X n +1 , Y n +1 ) .

where the last inequality follows from the facts that ˆ C lc ( X n +1 , Y n +1 ) ⊆ ˆ C ( X n +1 ) and ˆ C o ( X n +1 , ˆ A o ( X n +1 , Y n +1 )) ⊆ ˆ C ( X n +1 ) almost-surely.

Next, we only need to separately lower-bound by 1 -α the two terms on the right-hand-side of (A25).

The first part of the remaining task is trivial. It is already well-known that P ( Y n +1 ∈ ˆ C lc ( X n +1 , Y n +1 ) | Y n +1 = y ) ≥ 1 -α ; see [13, 47, 58].

Next, we prove the lower bound for the second term. Fix any y ∈ [ L ] , and assume Y n +1 = y . First, note that the oracle-selected attribute ˆ A o ( X n +1 , Y n +1 ) is invariant to any permutations of the exchangeable data indexed by D y ∪ { X n +1 , y } . Therefore, the data points are exchangeable conditional on the group identified by the oracle-selected attribute ˆ A o ( X n +1 , Y n +1 ) . This means that we can imagine the protected attribute ˆ A o ( X n +1 , Y n +1 ) is fixed, and the oracle prediction set ˆ C o ( X n +1 , ˆ A o ( X n +1 , Y n +1 )) is simply obtained by applying Algorithm A1 to exchangeable data using a fixed protected attribute. This procedure has guaranteed coverage above 1 -α ; see Lemma A1.

Proof of Theorem A3. The strategy of this proof is very standard in the conformal inference literature. We add this proof for completeness.

Recall that I ( ˆ A ( Z n +1 )) denotes a subset of the calibration data D that has the same value of the selected attribute as the test point. To prove (A17), it suffices to show that the nonconformity scores { ˆ S i : i ∈ I ( ˆ A ( Z n +1 )) ∪ { Z n +1 }} are exchangeable. Indeed, if the scores are exchangeable and almost surely distinct (which can be easily achieved by adding continuous random noises), then the rank of ˆ S n +1 is uniformly distributed over the discrete values { 1 , 2 , . . . , I ( ˆ A ( Z n +1 )) + 1 } . Consequently, the conformal p-value ˆ u ( Z n +1 ) constructed by Algorithm A7 follows a uniform distribution Unif ( { 1 |I ( ˆ A ( Z n +1 )) | +1 , 2 |I ( ˆ A ( Z n +1 )) | +1 , . . . , 1 } ) . This implies that P (ˆ u ( Z n +1 ) ≤ α |

ϕ ( Z n +1 , ˆ A ( Z n +1 ))) = α . Even if the nonconformity scores are not almost surely distinct, one can still verify that the distribution of ˆ u ( Z n +1 ) is super-uniform. Combining both cases, we have P (ˆ u ( Z n +1 ) ≤ α | ϕ ( Z n +1 , ˆ A ( Z n +1 ))) ≤ α for any α ∈ (0 , 1) .

We complete the proof by showing that the nonconformity scores { ˆ S i : i ∈ I ( ˆ A ( Z n +1 )) ∪ { Z n +1 }} are exchangeable. Define σ as an arbitrary permutation function applied on D∪{ Z n +1 } and denote the permuted dataset as σ ( D ) . We first run Algorithm A7 based on D to select the sensitive attribute ˆ A ( Z n +1 ) . Next, assume in a parallel world, we repeat Algorithm A7 with the same parameters and seed settings but based on the permuted data σ ( D ) . Denote the selected attribute in the parallel world as ˆ A ′ ( Z n +1 ) . Essentially, ˆ A ′ ( Z n +1 ) = ˆ A ( Z n +1 ) . This is because the computation of conformal pvalues and the procedure of selecting the attribute with the worst FPR in Algorithm A6 are not affected by the order of the calibration and test data. Therefore, the attribute selection process is invariant to the order of D∪{ Z n +1 } . This implies that the attribute selected by Algorithm A6 ˆ A ( Z n +1 ) can be treated as fixed, and the nonconformity scores computed on the subset I ( ˆ A ( Z n +1 )) ∪ { Z n +1 } are simply reordered in the parallel world, i.e.,

where ¯ σ is the permutation obtained by restricting σ on I ( ˆ A ( Z n +1 )) ∪ { Z n +1 } . Hence, we have

where the equality in distribution is implied by D d = σ ( D ) .

Proof of Theorem A4. This proof is the same as the proof of Theorem A3 since the multiple selected attributes ˆ A ( Z n +1 ) are invariant to any permutation of the calibration and test data.

Proof of Lemma A1. This proof is a minor extension of the proof of Theorem 1 in [17], with the difference that we additionally condition on the true test label.

Fix any ˜ y ∈ [ L ] , and suppose Y n +1 = ˜ y . For a placeholder label y , consider a subset of the calibration data D indexed by I ( X n +1 , y ) = { i ∈ D : Y i = ˜ y, ϕ ( X i , ˆ A ( X n +1 , y )) = ϕ ( X n +1 , ˆ A ( X n +1 , y )) } .

Since { ( X i , Y i ) } n +1 i =1 are exchangeable and the protected attribute ˆ A ( X n +1 , y ) is fixed, the nonconformity scores ˆ S i evaluated on the subset I ( X n +1 , y ) ∪ { ( X n +1 , y ) } are also exchangeable.

Further, denote ˆ Q ( X n +1 , y ) as the ⌈ (1 -α ) · | 1+ I ( X n +1 , y ) |⌉ -th smallest value of { ˆ S i } i ∈I ( X n +1 ,y ) . By the quantile lemma [13, 17, 57], for any α ∈ (0 , 1) ,

A6 Computational Shortcuts and Efficient Implementation

A6.1 Outlier Detection

Given a pre-trained one-class classifier, consider a calibration dataset D of size n . Let K denote the number of sensitive attributes, and for each attribute k ∈ [ K ] , let M k ∈ N denote the count of its possible values. Denote M = max k M k as the maximum count across all attributes.

AFCP for outlier detection outlined in Algorithm A7 has the following computational cost.

Analysis for a single test point

The cost of computing the false positive indicators for every sample in D∪{ Z n +1 } takes O ( n · log n ) . Breaking into steps, for every data in D∪{ Z n +1 } , compute their nonconformity scores takes O ( n ) . Their associated conformal p-value can be computed at once by sorting all scores and keeping track of their ranks, which takes O ( n · log n ) .

Then, selecting the sensitive attributes requires O ( n · K · M ) .

Once the attribute is selected, the cost of applying conformal prediction conditional on the group identified by the selected attribute is O (1) because the subset of the group has been found in the last step.

Hence, the total cost of running Algorithm A7 for a single test point is O ( n log n + nKM ) .

Analysis for m test points

The cost of computing the false positive indicators for every sample in D ∪ { Z n + t } t m =1 takes O ( n · ( m +log n )) . This can be derived by rewriting the conformal p-values for all j ∈ D ∪ { Z n + t } and for all t ∈ [ m ] as follows:

Computing the nonconformity scores for all samples in D∪{ Z n + t } t m =1 costs O ( n + m ) , evaluating the ranks takes O ( n log n ) , and comparing ˆ S n + t and ˆ S j costs O ( n · m ) .

Selecting the sensitive attribute costs O ( n · m + M · K · ( n + m )) . Breaking in steps, for each test sample Z n + t , the worst FPR for attribute k , as defined in (A20), can be rewritten as follows:

where D ( k, m ) represents a subset of the calibration data indexed by { i ∈ D : ϕ ( Z i , { k } ) = m } . The trick we use is that the identification of subset D ( k, m ) does not depend on the test sample t and can be reused to calculate the FPR for every test sample { Z n + t } t m =1 . In specific, identifying D ( k, m ) , ∀ k ∈ [ K ] , m ∈ [ M k ] takes O ( n · M · K ) . For each D ( k, m ) , computing ∑ i ∈D ( k,m ) E i,t , ∀ t ∈ [ m ] takes O ( |D ( k, m ) | · m ) , and computing E t,t 1 { ϕ ( Z i , { k } ) = m } takes O ( m ) . Therefore, repeating this process for all D ( k, m ) , k ∈ [ K ] , m ∈ [ M k ] in total takes O ( n · m ) . Lastly, finding the maximum FPR across all attributes takes O ( m · M · K ) .

Hence, the total cost of running Algorithm A7 for m test samples is O ( n log n + nm + MK ( n + m )) .

A6.2 Multi-Class Classification

Given a pre-trained multi-class classifier, consider a calibration dataset D of size n . Let L denote the total number of possible labels to predict, and let K denote the number of sensitive attributes. For each attribute k ∈ [ K ] , let M k ∈ N denote the count of its possible values. Denote M = max k M k as the maximum count across all attributes.

AFCP for multi-class classification outlined in Algorithm 2 has the following computational cost.

Analysis for a single test point

The cost of constructing prediction sets and computing miscoverage indicators within the leave-one-out procedure is O ( L · n · log n ) .

Then, selecting the sensitive attributes requires O ( n · K · M + L · ( n + M · K )) .

Once the attribute is selected, the cost of applying conformal prediction conditional on the group identified by the selected attribute is O (1) because the subset of the group has been found in the last step.

Hence, the total cost of running Algorithm 2 for a single test point is O ( L ( n log n + KM ) + nKM ) .

Analysis for m test points

The cost of computing miscoverage indicators is O ( L · n · ( m +log n )) . This can be derived by rewriting the conformal p-values for all j ∈ D ∪ { ( X n + t , y ) } ∀ y ∈ [ L ] , and ∀ t ∈ [ m ] as follows:

For each placeholder label y ∈ [ L ] , computing the nonconformity scores for all samples in D∪{ ( X n + t , y ) } t m =1 costs O ( n + m ) , evaluating the ranks takes O ( n log n ) , and comparing ˆ S y n + t with ˆ S y j costs O ( n · m ) . This process needs to be conducted for each y ∈ [ L ] , therefore the total cost of this step is O ( L · ( n + m + n log n + nm )) = O ( L · n · (log n + m )) .

Selecting the sensitive attribute costs O ( n · M · K + L · m ( n + M · K ) . Breaking in steps, for each test sample ( X n + t , y ) , y ∈ [ L ] , the worst miscoverage rate for attribute k , as defined in (5), can be rewritten as follows:

where D ( k, m ) represents a subset of the calibration data indexed by { i ∈ D : ϕ ( Z i , { k } ) = m } . The trick we use is that the identification of subset D ( k, m ) does not depend on the test sample t and the placeholder label y , therefore can be reused to calculate the miscoverage

rate for every test sample { ( X n + t , y ) } t m =1 . In specific, identifying D ( k, m ) , ∀ k ∈ [ K ] , m ∈ [ M k ] takes O ( n · M · K ) . For each D ( k, m ) , computing ∑ i ∈D ( k,m ) E y,i,t , ∀ t ∈ [ m ] , ∀ y ∈ [ L ] takes O ( L · |D ( k, m ) | · m ) , and computing E y,t,t 1 { ϕ ( Z i , { k } ) = m } takes O ( L · m ) . Therefore, repeating this process for all D ( k, m ) , k ∈ [ K ] , m ∈ [ M k ] and for all y ∈ [ L ] in total takes O ( L · n · m ) . Lastly, finding the maximum miscoverage across all attributes takes O ( L · m · M · K ) .

Hence, the total cost of running Algorithm 2 for m test samples is O ( n log n + Lnm + MK ( n + Lm )) .

A7 Additional Results from Numerical Experiments

A7.1 AFCP for Multiclass Classification

A7.1.1 Synthetic Data

Recall from Section 3.2 that Color is denoted as X 0 and the first one of the non-sensitive features as X 1 . The distribution of Y conditional on X is modeled by a simple decision tree, where X 0 and X 1 are the only useful predictors for Y , formulated as the following:

The detailed numerical experiments results of AFCP with the adaptive equalized coverage (see Equation (3)) are presented and compared with other benchmark methods.

Table A1: Average coverage and average size of prediction sets for all test samples constructed by different methods as a function of the training and calibration size. All methods obtain coverage beyond 0.9, while our AFCP and AFCP1 methods, along with the Marginal method, produce the smallest, thus, the most informative, prediction sets. Red numbers indicate the small size of prediction sets. See corresponding plots in Figure 3.

Coverage

Size

200

500

1000

2000

200

500

1000

2000

2

3

4

5

6

0.6

0.8

1.0

Sample size

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Attribute: Color

Blue

Grey

Coverage

Size

200

500

1000

2000

200

500

1000

2000

2

3

4

5

6

0.90

0.95

1.00

Sample size

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Attribute: Age Group

<18

>65

18-24

25-40

41-65

Coverage

Size

200

500

1000

2000

200

500

1000

2000

2

3

4

5

6

0.90

0.95

1.00

Sample size

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Attribute: Region

East

North

South

West

Next, we provide numerical experimental results comparing AFCP with the label-conditional adaptive equalized coverage (see Equation (A14)) with other benchmark methods.

Coverage for Blue

Average coverage

Average size

200

500

1000

2000

200

500

1000

2000

200

500

1000

2000

2

3

4

5

6

0.900

0.925

0.950

0.975

1.000

0.6

0.8

1.0

Sample size

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Coverage

Size

200

500

1000

2000

200

500

1000

2000

2

3

4

5

6

0.6

0.8

1.0

Sample size

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Attribute: Color

Blue

Grey

Coverage

Size

200

500

1000

2000

200

500

1000

2000

2

3

4

5

6

0.900

0.925

0.950

0.975

1.000

Sample size

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Attribute: Age Group

<18

>65

18-24

25-40

41-65

Coverage

Size

200

500

1000

2000

200

500

1000

2000

2

3

4

5

6

0.875

0.900

0.925

0.950

0.975

1.000

Sample size

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Attribute: Region

East

North

South

West

A7.1.2 Nursery Data

Coverage

Size

200

500

1000

2000

5000

200

500

1000

2000

5000

1

2

3

4

0.4

0.6

0.8

1.0

Sample size

Attribute: Parents' Occupation

Occupation 1

Occupation 2

Occupation 3

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Coverage

Size

200

500

1000

2000

5000

200

500

1000

2000

5000

1

2

3

4

0.90

0.95

1.00

Sample size

Attribute: Children

1

2

3

more

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Coverage

Size

200

500

1000

2000

5000

200

500

1000

2000

5000

1

2

3

4

0.900

0.925

0.950

0.975

1.000

Sample size

Attribute: Finance

Convenient

Inconvenient

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Coverage

Size

200

500

1000

2000

5000

200

500

1000

2000

5000

1

2

3

4

0.90

0.95

1.00

Sample size

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Attribute: Social

Nonprob

Problematic

Slightly_prob

Coverage

Size

200

500

1000

2000

5000

200

500

1000

2000

5000

1

2

3

4

0.85

0.90

0.95

1.00

Sample size

Attribute: Health

Not-Recommended

Priority

Recommended

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Next, we provide numerical experimental results comparing AFCP with the label-conditional adaptive equalized coverage (see Equation (A14)) with other benchmark methods.

Coverage for Occupation 1

Average coverage

Average size

200

500

1000

2000

5000

200

500

1000

2000

5000

200

500

1000

2000

5000

1

2

3

4

0.900

0.925

0.950

0.975

1.000

0.6

0.7

0.8

0.9

1.0

Sample size

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Coverage

Size

200

500

1000

2000

5000

200

500

1000

2000

5000

1

2

3

4

0.6

0.7

0.8

0.9

1.0

Sample size

Attribute: Parents' Occupation

Occupation 1

Occupation 2

Occupation 3

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Coverage

Size

200

500

1000

2000

5000

200

500

1000

2000

5000

1

2

3

4

0.900

0.925

0.950

0.975

1.000

Sample size

Attribute: Children

1

2

3

more

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Coverage

Size

200

500

1000

2000

5000

200

500

1000

2000

5000

1

2

3

4

0.92

0.96

1.00

Sample size

Attribute: Finance

Convenient

Inconvenient

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Coverage

Size

200

500

1000

2000

5000

200

500

1000

2000

5000

1

2

3

4

0.900

0.925

0.950

0.975

1.000

Sample size

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Attribute: Social

Nonprob

Problematic

Slightly_prob

Coverage

Size

200

500

1000

2000

5000

200

500

1000

2000

5000

1

2

3

4

0.900

0.925

0.950

0.975

1.000

Sample size

Attribute: Health

Not-Recommended

Priority

Recommended

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

A7.1.3 COMPAS data

We extend our experiments to investigate the effectiveness of AFCP using the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) dataset [50]. The COMPAS dataset is widely studied in the fairness literature for multi-class classification tasks, predicting the risk of recidivism across three categories: 'High', 'Medium', and 'Low'. Following the data preprocessing steps outlined in [59], we exclude rows with missing or low-quality data and compute the length of stay in jail. Additionally, we merge the race groups Asian and Native American, both of which have few occurrences, into the 'Others' category. After preprocessing, the dataset comprises 6,172 instances with five categorical features: charge degree of defendants (2 levels), race (4 levels), age category (3 levels), sex (2 levels), and score category of defendants (3 levels). We regard the first four features as potentially sensitive attributes.

Similar to the Nursery dataset case, we utilize the LabelEncoder function to numerically encode categorical features and outcome labels. To increase prediction difficulty and emphasize algorithmic bias, we introduce independent, uniformly distributed noise to the labels of samples identified as African-American. Additionally, we undersample the African-American group to 200 samples, while the Caucasian group contains 2,103 samples, Hispanic 509, and Others 385.

Figure A19 summarizes the performance of all methods as a function of the total number of training and calibration data points, which range from 200 to 1000. Figure A20 narrows the focus by analyzing the performance relative to the number of restricted calibration data points, as defined in Section 2.2. The results are averaged over 500 randomly selected test points and 100 repeated experiments. In each experiment, 50% of the samples are randomly assigned for training and the remaining 50% for calibration. Once again, we conclude that our AFCP methods outperform the other benchmarks considered, resulting in more informative prediction sets with higher conditional coverage for the African-American subgroup.

Figure A21 shows the selection frequencies of the protected attribute Race as a function of sample size within the same experiment described in Figure A19. This plot demonstrates that both AFCP and AFCP 1 consistently select Race as the most sensitive attribute as the number of samples increases. Also, we report the prediction accuracy in Table A21. The results confirm that the African-American group is disproportionately affected by algorithmic bias, not only in terms of uncertainty estimates but also in prediction accuracy, as they experience significantly lower-than-average test accuracy.

Coverage for African-American

Average coverage

Average size

200

300

500

1000

200

300

500

1000

200

300

500

1000

1.2

1.6

2.0

2.4

2.8

0.89

0.91

0.93

0.95

0.97

0.99

0.6

0.7

0.8

0.9

1.0

Sample size

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Coverage for African-American

5

10

30

50

0.6

0.7

0.8

0.9

1.0

Restricted calibration sample size

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Selects Race

200

300

500

1000

0.00

0.25

0.50

0.75

1.00

Sample size

Method

AFCP

AFCP1

A7.2 AFCP for Outlier Detection

A7.2.1 Setup and Benchmarks

This section demonstrates the empirical performance of our AFCP extension for outlier detection. Firstly, we focus on the implementation described in Algorithm A7, which selects at most one sensitive attribute. Similar to the multi-class classification cases, Our method is compared with three existing approaches, which utilize the same data, ML model, and conformity scores but compute conformal p-values with different guarantees. The first is the marginal benchmark, which constructs conformal p-value guaranteeing P (ˆ u marginal ( Z n +1 ) ≤ α ) ≤ α by applying Algorithm A2 without protected attributes. The second is the exhaustive equalized benchmark, which evaluates conformal p-values guaranteeing P (ˆ u exhaustive ( Z n +1 ) ≤ α | ϕ ( Z n +1 , [ K ])) ≤ α by applying Algorithm A2 with all K sensitive attributes simultaneously protected. The third is a partial equalized benchmark that separately applies Algorithm A2 with each possible protected attribute k ∈ [ K ] , and then takes the maximum of all such p values. This is an intuitive approach that can be easily verified to provide a coverage guarantee P (ˆ u partial ( Z n +1 ) ≤ α | ϕ ( Z n +1 , { k } )) ≤ α ∀ k ∈ [ K ] .

In addition, similar to the multiclass classification experiments, we consider AFCP1 - the AFCP implementation that always selects the most critical protected attribute without conducting the significance test.

For all methods considered, the outlier detection model is based on a three-layer neural network, and the outputs from each layer are batch-normalized. The Adam optimizer and the BCEWithLogitsLoss loss function are used in the training process, with a learning rate set at 0.0001. The loss values demonstrate convergence after 100 epochs of training. For all methods, the miscoverage target level is set at α = 0 . 1 . Note that the training and testing data contain both inliers and outliers, while the calibration data only contains inliers.

We assess the performance of the methods based on the False Positive Rate (FPR) and the True Positive Rate (TPR). Ideally, the objective is to achieve a higher conditional FPR for groups experiencing unfairness, thereby maintaining the average FPR below the target threshold of 0.1 while simultaneously achieving a higher TPR. The results presented are averaged over 500 independent test points and 30 experiments.

A7.2.2 Synthetic Data

We employ the same data settings as in the multi-class classification example, designating Color as the sensitive attribute associated with the Blue group, which suffers from undercoverage. While Age Group and Region are also sensitive attributes, they are not subject to biases in this context. The outcome labels Y have two possible values: Y = 0 if the unit is properly treated and Y = 1

otherwise. In our experiment, we treat Y = 1 as an outlier and Y = 0 as an inlier, with the data generation process described as:

Figure A22 and Table A22 illustrate the performance of conformal p-values generated by various methods on synthetic data, showcasing how performance varies with the number of samples in training and calibration datasets. Figures A23-A24 and Tables A23-A25 separately evaluate the False Positive Rate (FPR) and True Positive Rate (TPR) for each protected attribute. Additionally, Figure A26 explores how the severity of bias affecting groups impacts the selection frequency of our method at various levels. The severity of bias is controlled by the varying percentage of Blue samples, with higher levels indicating less samples and more significant biases in the Blue group.

FPR for Blue

Average FPR

Average TPR

200

500

1000

2000

200

500

1000

2000

200

500

1000

2000

0.0

0.2

0.4

0.6

0.8

0.00

0.03

0.06

0.09

0.12

0.0

0.1

0.2

0.3

0.4

Sample size

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Table A22: Average FPR and TPR of conformal p-values for all test samples constructed by different methods as a function of the training and calibration size. All methods control FPR under 0.1, while our AFCP and AFCP1 methods, along with the Marginal method, produce the highest TPR. Red numbers indicate high TPR. See corresponding plots in Figure A22.

FPR

TPR

200

500

1000

2000

200

500

1000

2000

0.00

0.25

0.50

0.75

0.0

0.1

0.2

0.3

0.4

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Attribute: Color

Blue

Grey

Sample size

FPR

TPR

200

500

1000

2000

200

500

1000

2000

0.0

0.2

0.4

0.6

0.8

0.00

0.05

0.10

0.15

Sample size

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Attribute: Region

East

North

South

West

FPR

TPR

200

500

1000

2000

200

500

1000

2000

0.0

0.2

0.4

0.6

0.8

0.00

0.05

0.10

0.15

Sample size

Method

AFCP

AFCP1

Marginal

Partial

Exhaustive

Attribute: Age Group

<18

>65

18-24

25-40

41-65

Selects Any Attribute

Selects Age Group

Selects Region

Selects Color

0

1

2

3

4

5

6

7

8

0

1

2

3

4

5

6

7

8

0

1

2

3

4

5

6

7

8

0

1

2

3

4

5

6

7

8

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

Sensitivity level

Method

AFCP

AFCP1

A7.2.3 Adult Income Data

We apply our method to the open-domain Adult Income dataset [51], a widely utilized resource in fairness studies. In this dataset, individuals with an income exceeding $50,000 are treated as outliers, while those with an income of $50,000 or less are considered inliers. All categorical variables in this dataset are treated as sensitive attributes, with levels that have small sample sizes grouped together during the pre-processing stages. After pre-processing, the sensitive attributes and their associated levels are as follows: Native-country (United-States, Others); Education (Bachelors, Somecollege, HS-grad, Others); Work Class (Private, Non-private); Marital-status (Divorced, Married-civspouse, Never-married, Others); Occupation (Adm-clerical, Craft-repair, Other-service, Sales, Execmanagerial, Prof-specialty, Others); Relationship (Own-child, Husband, Not-in-family, Unmarried, Others); Race (White, Others); and Sex (female, male).

This section evaluates and compares the performance of AFCP, AFCP1, and AFCP+, which is the AFCP implementation capable of selecting multiple protected attributes, along with other benchmark methods described in Appendix A7.2.1. All results presented in this section average over 500 test points and 30 independent experiments.

For this dataset, the groups suffering from algorithmic biases are unknown. Figure A27 evaluates the performance of different methods on several groups that are observed to exhibit higher FPR using the Marginal method. In all such cases, our AFCP, AFPC1, and AFCP+ methods effectively identify and correct for the protected attributes corresponding to at least one group suffering from significantly higher FPR.

Figure A28 and Table A26 present the average FPR and TPR of conformal p-values computed using different methods across all test samples. On average, all methods can control the FPR below the target level of 0.1, while our AFCP methods generally achieve higher TPR.

In addition, Figures A29-A36 and Table A27-A34 separately assess the FPR and TPR of conformal p-values for each protected attribute.

Relationship: Husband

Workclass: Not private

Education: Bachelors

Marital status: Married-civ-spouse

Occupation: Exec-managerial

200

500

1000

2000

200

500

1000

2000

200

500

1000

2000

200

500

1000

2000

200

500

1000

2000

0.00

0.05

0.10

0.15

0.20

0.00

0.05

0.10

0.15

0.20

0.25

0.00

0.05

0.10

0.15

0.20

0.00

0.05

0.10

0.15

0.20

0.25

0.0

0.1

0.2

Sample size

FPR

Method

AFCP

AFCP1

AFCP+

Marginal

Partial

Exhaustive

FPR

TPR

200

500

1000

2000

200

500

1000

2000

0.0

0.1

0.2

0.3

0.4

0.5

0.00

0.03

0.06

0.09

Sample size

Method

AFCP

AFCP1

AFCP+

Marginal

Partial

Exhaustive

FPR

TPR

200

500

1000

2000

200

500

1000

2000

0.0

0.1

0.2

0.3

0.4

0.5

0.00

0.05

0.10

0.15

0.20

Method

AFCP

AFCP1

AFCP+

Marginal

Partial

Exhaustive

Attribute: Workclass

Not private

Private

Sample size

FPR

TPR

200

500

1000

2000

200

500

1000

2000

0.0

0.2

0.4

0.6

0.00

0.05

0.10

0.15

0.20

0.25

Sample size

Method

AFCP

AFCP1

AFCP+

Marginal

Partial

Exhaustive

Attribute: Education

Bachelors

HS-grad

Others

Some-college

FPR

TPR

200

500

1000

2000

200

500

1000

2000

0.0

0.2

0.4

0.00

0.05

0.10

0.15

0.20

0.25

Method

AFCP

AFCP1

AFCP+

Marginal

Partial

Exhaustive

Attribute: Marital status

Divorced

Married-civ-spouse

Never-married

Others

Sample size

Method

FPR

TPR

200

500

1000

2000

200

500

1000

2000

0.0

0.2

0.4

0.6

0.00

0.05

0.10

0.15

0.20

Sample size

AFCP

AFCP1

AFCP+

Marginal

Partial

Exhaustive

Attribute: Occupation

Adm-clerical

Craft-repair

Exec-managerial

Other-service

Others

Prof-specialty

FPR

TPR

200

500

1000

2000

200

500

1000

2000

0.0

0.2

0.4

0.0

0.1

0.2

Method

AFCP

AFCP1

AFCP+

Marginal

Partial

Exhaustive

Attribute: Relationship

Husband

Not-in-family

Others

Own-child

Unmarried

Sample size

FPR

TPR

200

500

1000

2000

200

500

1000

2000

0.0

0.1

0.2

0.3

0.4

0.5

0.00

0.03

0.06

0.09

0.12

Method

AFCP

AFCP1

AFCP+

Marginal

Partial

Exhaustive

Attribute: Race

Others

White

Sample size

FPR

TPR

200

500

1000

2000

200

500

1000

2000

0.0

0.2

0.4

0.00

0.05

0.10

Method

AFCP

AFCP1

AFCP+

Marginal

Partial

Exhaustive

Attribute: Sex

Female

Male

Sample size

FPR

TPR

200

500

1000

2000

200

500

1000

2000

0.0

0.1

0.2

0.3

0.4

0.5

0.00

0.05

0.10

0.15

Sample size

Method

AFCP

AFCP1

AFCP+

Marginal

Partial

Exhaustive

Attribute: Country

Others

United-States

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper discusses the limitations of this work in the Discussion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions are stated in the paper and all proofs are provided in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper fully discloses all the information needed to reproduce the main experimental results. Additionally, code is provided.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The paper provides open access to the code needed to reproduce the main experimental results, and precise information about how the open-domain data were obtained. Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specify all the training and test details necessary to understand the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper reports error bars or standard deviations where appropriate, along with suitable explanations.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper provides information on the computer resources in the Discussion section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: This work is motivated by the need to address algorithmic bias issues, which is a topic with potentially broad impacts, as discussed in the paper. However, this paper focuses on foundational research that is not tied to a particular application.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper describes foundational research and does not release new data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: This paper uses open-domain data, properly crediting the license and creators.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The new code accompanying this paper is well documented.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a conformal inference method to assess uncertainty in classification by generating prediction sets with valid coverage based on adaptively chosen features. Falling between marginal and strictly conditional coverage, the features in the proposed method are adaptively selected to address potential model limitations or biases, balancing the need for informative predictions with ensuring algorithmic fairness.\n\nSTRENGTHS:\n- The paper is well written, with good organization, clear problem statement and easy-understanding method description.\n-  The proposed fairness notion is novel and critical for real-world application. It provides a feasible solution to migitage the tradeoff between fairness and efficiency. It is also impressing that the method works with small sample size.\n- The experiments and theorem are comphrensive and supportive of method and claims.\n\nWEAKNESSES:\n- It seems that *adaptive equalized coverage* requires each group to surpass a given coverage rate. Desirably like RAPS, the coverage rate should be the same at desired level to guarantee exact demographic parity.\n- Though a little out of scope, this paper would be better to provide performance evaluation with presence of distribution shift, especially label shift and shift in protected attributes.\n\nQUESTIONS:\n- Would your method preserve satisfy more rigorous fairness (Q1) with better design? I think this may be related with union of sub-intervals. Adaptively adjust the coverage rate of subgroup may help.\n- Also, the union of sub-intervals may make less sense in some real-world applications. Say, in some data, the target variables are some categorical and also ordered labels, say level 1~5. Predicting a sample to be either level 1 or 5  is confusing. Have this phenomenon be considered and addressed in your paper?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a conformal inference method to assess uncertainty in classification by generating prediction sets with valid coverage, conditional on adaptively chosen features. These features are selected to address model limitations or biases, balancing efficiency and fairness by ensuring equalized coverage for sensitive groups. The paper demonstrates this method's validity and effectiveness on both simulated and real datasets.\n\nSTRENGTHS:\n1. The method efficiently identifies and addresses algorithmic biases, ensuring fair treatment of sensitive groups without sacrificing informativeness. \n2. AFCP provides a practical compromise between efficient, informative predictions and algorithmic fairness by adjusting prediction sets for sensitive groups.\n3. Demonstrated effectiveness on both synthetic and real-world datasets, outperforming traditional methods in terms of both fairness and prediction informativeness.\n\nWEAKNESSES:\n1.\tLimitation on Sensitive Attribute Selection: The current method may not always identify the most relevant sensitive attribute, especially with limited sample sizes or overlapping biases. The ‘Automatic Attribute Selection’ section is somewhat challenging to follow. For instance, in Equation 6, it seems that only the argmax element is included in the set, and the algorithm does not seem to provide a sensitive attribute with formal guarantees. Since much of the paper’s contribution hinges on this algorithm, the lack of clarity in its description makes it hard to be convinced of its effectiveness.\n2.\tSample Size Sensitivity: AFCP’s performance can be constrained by the sample size. Smaller sample sizes may result in less reliable attribute selection and less informative prediction sets. In practical datasets, sample sizes tend to be small when many features are selected. The Sensitive Attribute Selection algorithm may lead to a large label set, resulting in too few samples to accurately determine the thresholds.\n3.\tComputational Complexity: With a large label set, such as in ImageNet which has 1000 labels, this method involves complex procedures for attribute selection and prediction set construction. This complexity may make the method computationally intensive.\n\nQUESTIONS:\n1.\tCoverage for Occupation 1 in Figure 5: The coverage for Occupation 1 in Figure 5 shows a significant increase in the Marginal method when the sample size increases from 2000 to 5000. This rapid increase is puzzling, particularly since the average size of the prediction set also decreases in this interval. Could you clarify why this behavior occurs? It seems counterintuitive, as one would generally expect a consistent relationship between coverage and prediction set size.\n2.\tImpact of Calibration Set Size on Sensitive Attribute Selection and Final Prediction Set: How do you think the calibration set size influences the algorithm for Sensitive Attribute Selection and the final prediction set? Specifically:\n\nLIMITATIONS:\nPlease refer to the weakness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel conformal inference method aimed at generating prediction sets with valid coverage, conditional on adaptively chosen features. This method is intended to address the dual concerns of efficiency and algorithmic fairness by ensuring equalized coverage for the most sensitive groups, thus providing informative predictions while maintaining fairness. The proposed approach, termed Adaptively Fair Conformal Prediction (AFCP), is validated on both simulated and real datasets.\n\nSTRENGTHS:\n1. The paper addresses a significant and timely problem in machine learning—ensuring fairness and reliability in prediction sets. The introduction of AFCP, which dynamically adjusts for biases in a data-driven manner, is an innovative and practical contribution.\n2. The paper provides a strong theoretical basis for AFCP, clearly defining adaptive equalized coverage and offering proofs to support the validity of the method.\n3. The empirical results are robust, covering synthetic and real-world datasets. The comparisons with other benchmarks are comprehensive, demonstrating the practical benefits of AFCP in various scenarios.\n4. The method's steps, including automatic attribute selection and prediction set construction, are well-explained and logically structured.\n\nWEAKNESSES:\n1. While the paper acknowledges the scalability issues associated with current methods for conformal inference with equalized coverage, it does not provide a detailed analysis of the computational complexity of the proposed method. A deeper exploration of scalability, particularly for large-scale datasets, would strengthen the paper.\n2. The method's reliance on leave-one-out procedures for attribute selection could be computationally intensive and potentially unstable with small sample sizes. More discussion on the stability and robustness of the attribute selection process, along with empirical evidence, would be beneficial.\n3. The literature review, while covering relevant works, could be more exhaustive. Incorporating additional recent studies on robust conformal inference and handling distributional shifts would provide a broader context and highlight the novelty of the proposed approach.\n\nQUESTIONS:\nWhat is the computational complexity of the proposed method, and how does it scale with large datasets?\nAre there any optimization strategies to improve efficiency?\n\nHow stable is the attribute selection process across different sample sizes and datasets?\nCan the method handle cases where multiple sensitive attributes need to be considered simultaneously?\n\n\nCan the method be tested on more diverse and larger real-world datasets to validate its scalability and generalizability?\nHow does the method perform in scenarios with highly imbalanced datasets?\n\nHave other attribute selection procedures been considered, and how do they compare with the proposed leave-one-out approach?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on the problem of conformal inference with equalized coverage introduced in [1]. The authors propose a new method, Adaptively Fair Conformal Prediction (AFCP) that (i) adaptively selects a sensitive attribute corresponding to the group most negatively affected by algorithmic bias as evaluated by the miscoverage rate, and (ii) constructs a prediction set that satisfies equalized coverage over groups defined by the selected attribute. The authors perform experiments on synthetic and real data that demonstrate the performance of AFCP relative to methods that guarantee marginal coverage and exhaustive equalized coverage i.e., ​​valid coverage conditional on all sensitive attributes.\n\n[1] Yaniv Romano, Rina Foygel Barber, Chiara Sabatti, and Emmanuel Candès. With malice toward none: Assessing uncertainty via equalized coverage. Harvard Data Science Review, 2020.\n\nSTRENGTHS:\n1. The problem of conformal inference with equalized coverage is an important problem and of interest to the community.\n2. The trade-off between efficiency and equalized coverage addressed in the paper is challenging and of practical significance.\n3. Empirical evaluation demonstrates the performance improvement of AFCP over baselines.\n\nWEAKNESSES:\n1. The clarity of the paper can be greatly improved. Detailed comments:\n- p2 l37: \"A limitation of the current method for conformal inference with equalized coverage....\" -- what is the current method?\n- p2 l41: what do you mean by efficiency and informativeness here? Is it set size? Please make it clear and precise.\n- Section 1.1: Overall, it is hard to understand the motivation from this section. It would be helpful to rewrite this a bit e.g. l32-36: it is not clear how this conveys the rationale for conformal inference with equalized coverage different from rationale for conformal inference more generally.\n- The notations are incorrect in some places and unnecessarily complex\n   - p2 l71-72: $\\phi$ is defined as mapping to $\\mathbb{N}$ whereas the next line says it results in vector of length $|A|$. I believe it should be defined more generally\n   - p2 l77-84: The notion of equalized coverage in [17] is defined as: $\\mathbb{P}(Y_{n+1} \\in C(X_{n+1}, A_{n+1}) | A_{n+1} = a) \\geq 1 - \\alpha$. It is not clear how this is extended to multiple (possibly overlapping) groups defined by $K$ sensitive attributes using $\\phi$\n- It is not clear at some places whether $A$ refers to single attribute or set\n   - Alg 2 l4-5: in line 4, $A$ refers to single attribute, in line 5 it refers to multiple (set of) attributes\n   - p6 l197: (7) returns a final selected attribute while the sentence refers to subset of attributes\n- p6 l204: minor comment, this does not hyperlink to A1\n- typos in table captions (Table A25 onwards)\n   \n2. Insufficient empirical evaluation: While the paper includes detailed analysis on the two selected datasets, both these setups are fairly synthetic. Also, the Nursery data seems to be from 1997; the paper lacks evaluation on any recent and common benchmark datasets in literature. The paper also seems to lack experiments that demonstrate coverage and efficiency performance in the presence of multiple sensitive attributes.\n\n3. The AFCP1 variation of AFCP seems to outperform AFCP in all cases -- when the sample size is small, we still see undercoverage for the Blue group using AFCP (Fig 3) and AFCP1 is more robust by selecting at least one attribute. While the paper mentions this, what would be the advantage of using AFCP? Is there any procedure to select variations for different data regimes?\n\nQUESTIONS:\n1. The main text discusses the AFCP algorithm where at most one sensitive attribute may be selected. If we want equalized coverage over multiple attributes, won’t using AFCP result in similar challenges as exhaustive equalized coverage?\n\n2. How does the size of the restricted calibration sample affect the performance of the algorithm?\n\nLIMITATIONS:\nThe authors discuss the limitations in the Discussion section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}