Conformal Prediction Sets Can Cause Disparate Impact
Authors: Jesse C. Cresswell
Bhargava Kumar
Yi Sui
Mouloud Belbahri
Venue: ICLR 2025 Spotlight (2025)
Keywords: Conformal Prediction
Fairness
Uncertainty Quantification
Trustworthy ML
Human Subject Experiments
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We demonstrate that providing conformal prediction sets to human decision makers can increase the unfairness of outcomes, and that applying Equalized Coverage increases unfairness more than marginal coverage.

Abstract: Conformal prediction is a statistically rigorous method for quantifying uncertainty in models by having them output sets of predictions, with larger sets indicating more uncertainty. However, prediction sets are not inherently actionable; many applications require a single output to act on, not several. To overcome this limitation, prediction sets can be provided to a human who then makes an informed decision. In any such system it is crucial to ensure the fairness of outcomes across protected groups, and researchers have proposed that Equalized Coverage be used as the standard for fairness. By conducting experiments with human participants, we demonstrate that providing prediction sets can lead to disparate impact in decisions. Disquietingly, we find that providing sets that satisfy Equalized Coverage actually increases disparate impact compared to marginal coverage. Instead of equalizing coverage, we propose to equalize set sizes across groups which empirically leads to lower disparate impact.

Published as a conference paper at ICLR 2025
CONFORMAL PREDICTION SETS CAN CAUSE DISPARATE IMPACT
Jesse C. Cresswell Layer Al
Bhargava Kumar" TD Securities bhargava. www.r@tdaaccurttieeecccoo
Sui Layer 6 Al
Mouloud Belbahri
Layer 6 Al
aassellayer6.a1
anydlayar6.at
nouloudElagyere.at
ABSTRACT
Conformal prediction is statistically rigorous method for quantifying uncertainty in models by having them output sets of predictions, with larger sets indicating uncertainty. However, prediction applications require a single output to act on, not several. To overcome this lim- itation, prediction
are not inherently actionable; many
can be provided decision. In any such system it is crucial to ensure the fairness of outcomes across protected groups, and researchers have proposed that Equalized Coverage used as the standard for fairness. By conducting experiments with human participants, we demonstrate that providing prediction sets can lead to disparate impact de- cisions. Disquietingly, we find that providing sets that satisfy Equalized Coverage actually increases disparate impact compared to marginal coverage. Instead equalizing coverage, propose to equalize set sizes across groups which empir- ically leads to lower disparate impact.
human who then makes an informed
INTRODUCTION
Human Accuracy Disparate Impact with Prediction Sets
Conformal prediction (CP) (Vovk et al.) 2005)
Teatment:
Auga Marginal + Canditional
has emerged one of the most promising methods for uncertainty quantification in ma- chine learning because of its wide applicabil- ity, and statistical guarantees based on very few assumptions. The main use of CP is trans- form heuristic notions of uncertainty into rigor- ous ones through a calibration step. The output of conformalized model is prediction set a set of likely outputs. To communicate uncer- tainty, prediction sets are larger when the model is more uncertain about the correct answer.
Accoracy Disparate impartt (pp)
Dataset: FACET
Dataset: BioaBias
Dateset: RAVDESS
3.5
3.4
3.3
Treatment
Figure 1: We measure the increase in accuracy per-group from using prediction sets compared to the control population without model assistance. Disparate impact is the maximum difference of creases between groups, and should be minimized for fairness (Equation 7p. Prediction sets do not benefit all groups equally, while sets with Equal- ized Coverage (Conditional) lead to the most un- fair outcomes. Statistical analyses and signifi- cance are presented in /Sectionn 6
However, there is a clear drawback model that outputs prediction sets. Models are com- monly part of decision pipelines where input data is converted into actions. For classifica- tion, the possible actions are often mutually ex- clusive, such that for given observation we re- quire single action in response, not a set.
Still, there are many cases where decisions are not left to model alone, but require a human in the loop. For example, society does not yet use machine learning models make most medical diagnoses we prefer human doctors for reasons of safety, accountability. and trust. Nevertheless, there are ample opportunities for people like doctors to use machine learning tools to improve their decisions. As uncertainty quantification is crucial component for trust, it has been argued that CP sets are a natural fit for such applications (Lu et al. 2022). Indeed, research has shown that providing conformal sets to humans increases their accuracy decision tasks (Straitouri & Gomez Rodriguez, 2024 Zhang et al. 2024 Cresswell et al. 2024 De Toni et al. 2024).
"Equal Contribution

---

Published as a conference paper at ICLR 2025
Some of the same research pointed towards troublesome trend. Cresswell et al. (2024) observed that when conformal sets were given to humans, their accuracy generally increased compared to a control population, but not always by the same amount across groups in the dataset. For some groups accuracy even decreased. This is a major fairness concern. Were a doctor to use prediction sets to assist with diagnoses, it would be unacceptable for some protected groups have worse outcomes with prediction sets than without.
The fairness of CP has been considered previously by Romano al. (2020a) who focused on the coverage guarantee that error rates will be no higher than a user-specified tolerance. However, CP controls the error rate only on average across the entire data distribution, and may fluctuate between groups within the distribution. Hence, Romano et al. (2020a) proposed a new faimess standard that all groups should have Equalized Coverage. Since then, researchers broadly have accepted that equalizing coverage across groups is fair (Lu et al., 2022 Berk et al., 2023b, Ding et al.) 2024).
In this work, we focus on the fairness of prediction sets used for momaa--n--thelloo decision the fairness of using CP sets
pipelines. Our aim is to provide the first scientific evidence in practical settings. Through pre-registered, randomized controlled trials with human participants, we find that prediction sets can lead disparate impact the increases in accuracy compared to the control population are not equal across groups (igure Ip. This alone would be cause for concern, but, distressingly,
further find evidence that applying Equalized Coverage leads to more unfair outcomes than marginal conformal sets. Hence, we expose a major discrepancy between how fairness has been treated in the literature CP, and fairness outcomes in practice. We advocate that practitioners move away from Equalized Coverage and instead aim for Equalized Set Size, which correlates strongly with reduced disparate impact.
BACKGROUND
We begin with background on CP and other set-predictors, and review relevant fairness notions. For the sake of brevity, we only consider the setting of classification in our review and experiments.
2.1 CONFORMAL SET PREDICTORS
CP (Vovk et al. 2005; Shafer & Vovk 2008) is one of the leading approaches uncertainty quan- tification in machine learning (Soize) 2017 Abdar et al. 202T). and is aligned with efforts to make models more trustworthy. Other uncertainty quantification approaches often rely on strong assump- tions which are unlikely to hold in practice (Gal & Ghahramani, 2016 Lakshminarayanan et al.) 2017). or require modifications to the model architecture (Neal 2012). In contrast, CP applies to black-box models, has dependence and assumes only that test data is drawn from the same distribution as calibration data (Vovk et al.
training data, is distribution free, is valid in finite samples,
1999 Angelopoulos & Bates 2021)
Let us consider inputs I E X' C RDD associated with ground truth classes y E === [m]] where [m] (1...., m. drawn jointly from distribution (z, P. Given arbitrary classifier X' + [0, 11 with softmax outputs, CP creates set-predictor (Grycko) 1993). denoted as
C: X' - 2mm), with the hallmark property that the sets produced by C satisfy a coverage guarantee
Ply E C(x)] 2 1-a,
(1)
for error rate E [0, 1] which can be chosen a priori by the user (Vovk et al. 1999).
CP achieves coverage by allowing the set size [C(=) to vary depending on the model's heuristic confidence, e.g., the softmax outputs of f. Larger sets indicate greater model uncertainty about x. To move from f to C, CP carries out calibration step on a held-out dataset Deat of riaal datapoints drawn independently from P. First, one defines conformal score function larger scores indicating worse agreement between x and y according to the heuristic uncertainty notion, and then computes the [(n=])(1-a]] quantile which we will denote q. With the calibration step done, CP sets can be generated for a test datapoint Trest ~ IP as
X' x y I with
all datapoints (z, 3) Deal. After sorting the scores, one computes
(a == yy E y s(Iusts y) < x).
For any classifier f and any score function s, the CP sets Cj will provide 1 1-a coverage (Equatton
2

---

Published as a conference paper at ICLR 2025
While valid for any 8, the usefulness of CP greatly depends how is designed as well properties of the model f including its accuracy and calibration (Guo et al. 2017). The research community has taken great interest in devising score functions that produce sets with the smallest average size. As long as coverage is maintained, small sets are viewed as more useful for applications including human decisioning (Cresswell al.] 2024). Some of the most efficient score functions include APS [Romano et al. 2020b). RAPS AAgeloppolos et al. 2021). and SAPS (Huang et al. 2024).
2.2
CONDITIONAL COVERAGE MONDRIAN CONFORMAL. PREDICTION
The coverage guarantee in Equation I is not totally satisfactory since it holds marginally across the entire distribution P. Some subgroups within the distribution may receive lower coverage than 1-a, for example minority groups which are less well represented in the training set of f leading to lower model accuracy or worse calibration. Ideally, the coverage guarantee would hold for every subgroup of the distribution, down to individual datapoints so-called conditional coverage (Foygel Barber al. 2020), but unfortunately this has been proved impossible without distributional assumptions [Vovk 2012: Lei & Wasserman 2013). A less strict, and actually attainable goal is group-wiss conditional coverage for pre-specified grouping g X + G = [ng]. where each datapoint is assigned one of ny groups E G, and the coverage guarantee applies to each group separately as
Ply E C(x) gg()) - a] 2 a, VaEg.
(3)
Group-wise conditional coverage can be achieved through Mondrian CP (Vovk et al. 2003 2005). which simply partitions Deal by groups and performs CP as described in Subsection 2.1 on each group independently. This method can suffer from high variance when ng is large, or individual groups have scant representation in Deat. the effective size of cach calibration set could become too small (Angelopoulos & Bates 2021 Methods to circumvent these issues have been explored
(Romano al. 2020c Gibbs al. 2023 Ding al. 2024).
2.3 NON-CONFORMALL SET PREDICTORS
While CP is praised for its distribution-free coverage guarantee, it is not the only way to generate high- quality prediction sets (Chzhen al. (02T). One other method, average-k set prediction (Algorithm T). op- timizes for the lowest eтrоr rate given the constraint E|C|= = k, where k E IR+, Avg-k prediction relies on calibration step where a quantile qi of softmax scores chosen such that 1 1- k/ m of the scores in the calibration set are below qk- Then, prediction sets are formed as
Algorithm 1: Average-k set prediction
Input: Calibration dataset Deat, classifier f. average size k, test datapoint Trest- + []
for (z, y) E Deal do Y exxtend (f(x)) // softmax m numClasses (Deat) k/m
P I + quantile (V,
Yust + f(zrea) // sof smft qk
C.. (Fres) == {v ) f(Irss) > qx
(4)
return
Optionally, labels y such that (Zees) = 9k can be randomly added. The avg-k predictor has optimal error rate under the average size constraint (Lorieul et al.) (02T). Since allowed to range over positive real values, target coverage level 1 - c can be set by computing qu for different on a calibration set, and evaluating the empirical coverage on a validation set as
nal
6z
Σ
Ily E Cg. (x,)].
(5)
While the avg-k predictor does not have all the features of conformal methods, it still quantifies the uncertainty of via variable size prediction sets where larger sets indicate greater uncertainty.
2.4 FAIRNESS OF SET PREDICTORS
In real-world applications of machine learning systems, ensuring fairness is of paramount impor- tance, especially in highly regulated industries such healtheare and financial services. The topic of fairness is incredibly nuanced and cannot be distilled down into a simple set of rules or criteria that should be followed in all cases. Still, it can be useful to delineate high-level approaches.
Two distinct concepts often guide regulatory frameworks: procedural and substantive fairness. Pro- cedural faimess focuses the integrity of the processes involved in developing models and ensures that all subjects are treated in the same way (Grgić- Hlača et al.) 2016). simple example of proce- dural faimesss in machine learning is when group identifiers (and proxies thereof) are scrubbed from
3

---

Published as a conference paper at ICLR 2025
dataset. Hence, any model trained on the remaining data cannot rely on group information at all so called fairness through unawareness (Zemel et al. 2013 Kusner et al. 2017).
In contrast, substantive fairness focuses outcomes, aiming for equitable decisions even if groups are not always treated the same (Dwork et al. 2012). Aligned notions include Equalized Odds and Equalized Opportunity which explicitly use group identifiers to ensure that beneficial outcomes equally likely across groups (Hardt et al. (016). Substantive fairness can be measured by computing disparate impact for a given metric related to outcomes (Feldman et al., 2015 Esipova al. (023). For our set predictors, the relevant metric is the accuracy of humans on a classífication task, and
hence we consider the change in per-group accuracy when prediction sets are supplied to humans compared to unassisted decisions,
,r,a == acc, far E E gg()) = a] acComntrol (1r E D g)(())) = a),
(6)
where / E T denotes the trealment indicating which set prediction method was used. The disparate impact of providing prediction sets be written
max (ōt,a - 8r,6), a.bEd
where the maximization is done over all pairs of groups. A treatment which has the same benefi- cial effect on all groups achieves zero disparate impact and is substantively fair. When comparing
other quantities between groups, like coverage or set size, we use a similar A notation although the comparison is not with reference control treatment, e.g.
Aoov
max (Ply C(x) ((z) a| Ply C(x) g(x) b])- a.EEG
The main definition of fairness used in the context of CP is Equalized Coverage (Romano et al. 2020a) which says that all protected groups in the dataset should receive the same level of coverage. Equalized Coverage, expressed formally as Acov & O, is simply group-wise conditional coverage SSusectton 2.2) over protected groups. Researchers have widely adopted the notion that equalizing coverage is sufficient for ensuring fairness, e.g. Ding el al. (2024) state that with Equalized Cover- age "prediction sets ... are effectively 'fair' with respect all classes, even the less common ones". In our context it is clear that Equalized Coverage aligns with the concept of procedural faimess. Coverage not an outcome - a byproduct of how CP quantifies uncertainty about some under- lying task. As we shall demonstrate, equalizing coverage does not translate into fair outcomes on the underlying tasks, in fact it exacerbates disparate impact.
3 RELATED WORK
Investigations of the fairness of CP methods began with Equalized Coverage which has remained the standard framework. [Romano et al. (2020a) presented case study where Mondrian CP achieved Equalized Coverage on a medical dataset at the cost of an increased discrepancy between per-group set sizes. Recently, Wang et al. (2024) proposed Equalized Opportunity of Coverage, which takes account of the class label when equalizing coverage between groups. Since its focus is on coverage and not actual outcomes, it still falls under the notion of procedural fairness.
Several works have applied CP to sensitive datasets where fairness considerations are necessary (Lu al., 2022] Kuchibhotla & Berk 2023 Berk al., 2023a). For instance, [u et al. (2022) discussed how conformal sets could assist medical doctors and specified Equalized Coverage as desirable prerequisite, although no experiments with doctors were conducted. [Straitouri et al. (2044) considered how restricting humans to choose their answer only from a prediction set can cause harm. Other researchers aspired to make CP more fair, such as [ss et al. (2022) who applied demographic parity to conformalized quantile regression (Romano et al. 2019)), or Deng et al. (202)) who generalized multicalibration THebert-Johnson et al. 2018) to conformal methods, and Zhou & Sesia (2024) who adaptively identified under-covered groups and increased their set sizes to achieve Equalized Coverage.
Researchers have also considered the fairness of uncertainty quantification methods beyond set- predictors. Calibration is an essential prerequisite for uncertainty quantification which stands in tension with fairness requirements (Pleiss et al., 2017). Ali et al.] (2021) advocate that fairness approaches should focus on errors from epistemic uncertainty, Kuzucu et al.] (2023) identify that models may appear fair according to point predictions while providing biased uncertainty estimates, and Mchta et al. (2024) explore the impact of applying fairness methods on uncertainty estimates.
4

---

Published as a conference paper at ICLR 2025
Data
Model
Conformal Prediction
Test Data Samples
Human Accuracy Control
Group e Easy
Marginal Coverage Set Size
A
P
90%
Marginal
f
h
h
90% (OA)
90% in (O.A.O)
Conditional Coverage Set Size
Accuracy
Conditional
90%
Group Hard
90% in (A) (O.00
90% in
h
h
h
Figure 2: Illustration of how unfairness can arise in CP. Given a data distribution IP with groups of differing difficulty, a model f may have inherent bias. Using marginal CP can translate to lower coverage and larger sets for the harder group. To equalize coverage, conditional CP must increase set sizes on the harder class, and reduce them on the easier class. Since human accuracy correlates strongly with set size, not coverage, outcomes become more unfair with Equalized Coverage.
4 HYPOTHESES & METHOD
4.1 HYPOTHESES
Suppose / was trained dataset with some inherent bias, or that groups naturally vary in difficulty such that f has different accuracy on two groups e and à ("easy" and "hard"). Marginal CP using sensible score function will tend to overcover examples from and undercover examples from h. To achieve group-wise conditional coverage instead of marginal, the coverage on e must decrease, while the coverage increase coverage on / is to increase the number of classes added to prediction sets for / - in other words, average set size must increase for and correspondingly decrease for If set sizes were comparable across groups for marginal sets, then conditionally covered sets will tend to have more unequal average set sizes across groups. Essentially, there is tradeoff between equalizing coverage, and equalizing set size.
h increases. For fixed model and score function 8, the only way to
On its own this tradcoff is not directly connected to the substantive fairness of using prediction sets. However, it has been observed in previous studies that it is set size, not coverage, which has the greatest influence on human accuracy, and therefore on outcomes (Cresswell et al. 2024). Together, these facts imply that equalizing coverage will create more imbalanced set sizes which will propagate into more biased human performance. The proposed mechanism by which equalizing coverage with Mondrian (conditional) CP causes disparate impact is illustrated in Figure 2
Based on the above discussion, we propose two hypotheses:
Hypothesis 1 Prediction sets supplied human decision makers cause disparate impact in the human's 's performance.
In particular, a model has biased performance across groups, marginal CP will tend to produce unequal set sizes which will in turn provide uneven utility to human decision makers.
Hypothesis 2 Prediction sets created with Mondrian CP to ensure Equalized Coverage will cause greater disparate impact than marginal prediction sets.
The of Mondrian CP or similar methods to equalize coverage will benefit groups that have higher model accuracy more than groups with low accuracy, producing a Matthew effect where "the rich get richer", instead of more equitable Robin Hood effect (Ganev al. 2022).
The remainder of this paper investigates whether these hypotheses can be supported by data, along with the claim that set size has greater influence human accuracy than coverage.
5

---

Published as a conference paper at ICLR 2025
4.2 METHOD
We conduct randomized controlled trials with human subjects to discern the impact of various set prediction methods on per-group accuracies, and measure potential disparate impacts. Our hypothe- ses, experiments, and analysis plans were pre-registered unteers are asked to complete challenging task and are provided prediction sets from trained machine learning model (or for the control, no assistance is given). The sets are constructed either with marginal coverage guarantees (Equation 1}, group-wiss conditional coverage through Mon- drian CP on a specified grouping (Equation 3), or via the avg-k method (Equation 4]. In all cases the same model is used, aiming for the same coverage level over Deat. We call these four options treatments, and refer to them as control, marginal, conditional, and avg-k.
each experiment, paid human vol-
We conduct independent experiments three distinct tasks. For each task, N unique participants are recruited and are randomly but evenly partitioned into the four treatments. During the experi- ments we conduct M sequential trials, each where the participant is shown a datapoint from Drest drawn i.i.d from the same distribution as Deat. along with a prediction set and stated coverage guarantee (except for the control). No other information from the model such as softmax scores is given. The tasks are each designed as forced choices where participants must select one class out of m available classes, and only one class is considered correct.
Given data collected in this way we can compute the accuracy improvements (Equation 6) and disparate impact (Equation 7) for each treatment directly. However, this direct analysis would not account for the design of the experimental method which could result in overconfident inferences (Yarkoni 2022). In particular, each participant provides multiple responses, participants may have different inherent ability levels on their task, and samples drawn from Diest can also have different inherent difficulties. Hence, our formal statistical analysis models human performance on the tasks using Generalized Estimating Equations (GEEs) GEEs can be seen as Generalized Linear Models for clustered responses that take into account intra catra-pprticcpantt response correlations (Liang & Zeger 1986 Hence, GEEs are suitable choice since each participant represents a cluster of responses.
Formally, we denote the response variable by correct,,j. indicating whether participant selected the correct label for trial j. Additionally, we consider the following covariates: (i) treat, denoting the treatment participant i is randomly assigned to; (ii) group,, which is the protected group for C trial, j that participant sees; and (iii) diff,., the marginal conformal set size of trial j shown to participant i, which acts as proxy for the inherent difficulty of that datapoint.
For each task, we fit GEEs estimate a marginal model for the effect of treat on the participants correct responses, controlling for group and diff covariates. The model can be expressed as
logit (B)oorrcc))) treat; x groupi,j + diff
for i 1, ,,..., N and j l,..., M, where log log and the notation A x B indicates an interaction between covariates A and B.
Alongside empirical estimates of the standard errors, it is straightforward to derive odds ratios (ORs) from the GEE model. ORs allow us estimate how much more likely participant in treatment t is to give the correct answer than if they were in the control, which is another way of expressing the expected accuracy improvement due to (c.f. [qquation 6p. For the sake of clarity, we give the formal description below. Let Pt.a := P'correct a correct answer under the GEE model, conditional on treatment and group. For treatment t, and for every protected group a, the OR of versus control is given by
treat = f, group = a) be the probability of
pt.a/ (1 Pt.a)
OR Peontrol, / (1 Peontrolla)
(10)
The interpretation of OR,,a is simple. For example, OR,, = 1.4 means that for trials representing group a, the odds for participants assigned to treatment t to give the correct answer are 40% higher compared with participants assigned to the control treatment.
To assess the disparate impact caused by any treatment for |Hypothesis 1 between groups We define the ratio of ORs (RORs) for treatment t. and groups a and as
we can compare ORs
'The pre-registration is viewable at osf. 10/75hm9
We use the GEE implementation from the stat statelmdels python package.

---

Published as a conference paper at ICLR 2025
ROR == /OR When ROR & 1, treatment provides the same advantage over the control for group a as it does for group b which is a fair outcome. Hence, a value of ROR,,.b6 greater than for some pair (a. indicates that causes disparate impact, and so report the maximum ROR between pairs of groups as a descriptive statistic (c.f. Equation 7.
maxROR, max ROR,
(11)
a.PEg
Finally, to assess whether equalizing coverage increases unfairness for Hypothesis 2 maxRORMare to maxRORcond If the latter value exceeds the former, it indicates that equaliz- ing coverage amplifies disparate impact relative to the marginal treatment.
we compare
EXPERIMENTS & EVALUATION
5.1 TASKS, DATASETS, AND MODELS
Using open-access datasets from the machine learning fairness literature, we created three tasks where human decision makers could potentially take advantage of model assistance.
Image Classification Image classification is performed by humans daily in impactful settings, for example by radiologists who view X- X-ray images from patients of all ages. For a similar setting we used the FACET dataset (Gustafson et al. 2023) of images of people, labeled by their occupation with grouping by age. We used the most common classes and split the dataset into Deat, D Peulval, and Pest stratified by class. The age annotations come in four pre-defined groups: Younger, Middle, Older, and Unknown. We used CLIP ViT-L/14 (Radford et al. 2021) as a zero-shot classifier.
Text Classification Text classification is often used to organize large amounts of text data, for instance in recruiting where gender bias can easily manifest. As a surrogate task, we employed the BiosBias dataset -eeAArteaaa et al. 2019) which contains personal biographies classified by occupa- tion, and grouped by binary gender. We selected 10 of the most common occupations and then split the dataset into Durain DDat. Deat. Dealval, and Diest, ensuring class balance. Dirain and Dvaa for classifier training, while the remaining splits were used for prediction set construction. For clas- sification, we generated representations the biographies using pre-trained BERT model (Devlin) itt al. 2019 Huggingface, 2024), then trained linear classifier on these representations.
used
Audio Emotion Recognition Emotion recognition is performed naturally in human communica- tion. but emotional expression can vary greatly between speakers of different genders. We utilized audio recordings from the RAVDESS dataset (Livingstone & Russo 2018) where female and male actors convey 8 emotions using the same short phrases. We partitioned the dataset into Deat. Dealval and Dies ensuring stratification by class (emotion) and group (binary gender), and used a fine-tuned Fadel] 2023) for emotion classification.
wav2vec2 model (Baevski et al. 2020
For all three tasks, we aimed compare disparate impact between avg-k, marginal, and conditional prediction sets with target 90% coverage. For greater diversity in approaches to CP, FACET and RAVDESS used the RAPS score function (Angelopou- et al. (021), while BiosBias used SAPS (Huang al. 2024). The hyperparame- ters of these score functions were tuned on Deatval with 50 iterations to minimize av- erage set size using Optuna (Akiba et al.) 2019a). Then, each method used Dew to calibrate its threshold(s) for set prediction. Table 1 displays metrics computed on Diea for the mod- els used in each task, including the top-1 accuracy, and for each set prediction method the empirical coverage and average set sizes. We note that all methods achieve coverage close to 90%, while con- ditional CP tends to have larger set sizes due to its more stringent requirements. Also shown are the maximum per-grop ger-group differences in model accuracy, coverage, and set size defined as in Equation 8 We note that conditional CP is the only method that achieves Equalized Coverage (Acov O), but at the cost of larger difference in set sizes between groups.
Table 1: Model metrics on Destt
Task
Top-1
App Method Cov. Size Aco
Asize
Avg-k 90.2 2.52
5.8
1.10
FACET
70.0
22.2
Marg. 89.7 2.62
11.4
0.75
Cond. 90.0 2.76
2.6
3.16
Avg-k 88.2 1.52
3.3
0.01
BiosBias
78.9
2.7
Marg. 89.8
1.69
2.7
0.03
Cond. 89.1 1.70
0.0
0.43
Avg-k
91.4 1.97
2.8
0.12
RAVDESS 71.1
7.8
Marg. 90.8 1.94
3.9
0.07
Cond. 91.4 2.01
0.6
0.71
Additional details on tasks, datasets, models, and set prediction methods are given in Appendix Our code is available at cttecuu..coomlayerrra---- lataab//ooooaaallooooooaaaa.alla http://oooooaappppppoooooooooeae

---

Published as a conference paper at ICLR 2025
5.2 EXPERIMENT DESIGN
We created experiments and hosted them online with participants recruited through Prolific (Prolific) 2024). To ensure high-quality data, participants were trained on their specific task and given fi- nancial incentive to answer correctly. Each ex- periment consisted of task (FACET, BiosBias, or RAVDESS), and a treatment (control, avg-k, marginal, or conditional). We recruited 600 partic- ipants (N 200 per task, 50 per experiment), and paid them on average 9.75 GBP per hour, totaling 1500 GBP for participant payment.
the image below, solect most ap approp option.
Bupgest ons: There 50% probabiley the anower one
Officrr
Schespamaon
Boatmar
Cullorist
Off
Compuner User 6, Gymmast
15, Matoroyclst
16 Eaand
8. How mesccheeeer
14. the otee
10.
Sowekber
Farmer
10. Hone Rider
15, Repaimma
20.
Ternis Player
Enter value betweon 1 and
During each experiment, participants were first shown consent form detailing how their data would be collected and used. Then, participants were intro- duced the task and trained on 20 practice trials which were not used in our analysis. The testing phase proceeded with M 50 more trials, an ex- ample of which is shown in Figure for FACET, while detailed task descriptions can be found in Appendix B For each trial, participants were presented with a datapoint along with all m class labels and were asked to classify x. For the avg-k, marginal, and conditional treatments participants were also shown prediction set, accompanied by the expected coverage. There was no time limit for responses, however for FACET the image x was only displayed for one second to increase task difficulty. After the class was selected for each trial, the correct answer was revealed.
The 20st, brower is Guard
Press BPNEBAR to continue
Figure Main trial screen shown to partic- ipants for FACET with marginal conformal set treatment. The correct answer is given only after the participant responds.
6 RESULTS
6.1 DISPARATE IMPACT MEASUREMENT IN HUMAN SUBJECT EXPERIMENTS
Figure I directly visualizes the data we collected, showing the disparate impact on accuracy A calculated as in [qquation 7, treating each trial as an independent observation. We observe that disparate impact is present for most tasks and treatments (A, > 0), notably with the conditional treatment consistently worse. As discussed in [Subsection 4.2] each trial is not independent, so we present Figure I only for intuition, and not as formal statistical analysis.
Instead, we applied the more rigorous statistical analysis with GEEs and present the results in Table which shows the ORs of human accuracy for each task and treatment as compared to the control, along with maximum RORs across groups We interpret the findings in the context of our two hypotheses from Subsection 4.1
Hypothesis I supposes that biases from a model can propagate to humans using prediction sets as an aid and result in disparate impact. Tabll I demonstrates that the models we used have some biases across groups, with non-zero A values, which translated to non-zero Acov for the avg-k and marginal sets. Conditional sets do equalize coverage (Acov s 0), but at the cost of greater discrepancies in set size, Asize From Table 2 we see that in most cases prediction sets were useful to participants versus the control (OR,,a a > 1), supporting previous findings (e.g. Cresswell et al.) 2024)). However, it is clear that the beneficial effect is not experienced equally across groups. For FACET, the Younger group had the highest model accuracy (Figure 5). and experienced the most improvement across all treatments. The Older and Unknown groups having lower model accuracy and minority representation (see Table 4jin/Appendix A) saw less improvement, and even some harm with conditional sets where human performance decreased (OR,,a < 1). Hence, all three treatments caused disparate impact (max > 1). Similar results are seen for BiosBias. Even though the model itself was much less biased (Talle 1) and both Female and Male groups benefited when sets were provided, the amount of benefit was not equally shared. Both FACET and BiosBias show that providing prediction sets can cause disparate impact, in support of Hypothesis I RAVDESS provides an example that disparate impact need not always occur. Again, both groups benefited from each treatment, but for the avg-k and marginal treatments the increase actually was commensurate across groups. For these two treatments the prediction sets did not have equal coverage, but did have close to the same average set sizes across groups
8

---

Published as a conference paper at ICLR 2025
Table Summary statistics from the GEE models: OR, (Equatinn 10) where values greater than indicate treatment t had a positive effect on human accuracy for group and the and superscripts indicate significance at 5% and 10% levels respectively; maxROR; (Equation IT]. where values
greater than indicate disparate impact treatment t benefited one group more than another.
Dataset
Group
OR
Marg
Ood MaxRORAvg
max Marg maxROR Cond
Younger
1.19
1.34
1.37
Middle
1.12
1.20
1.19
FACET
1.11
1.26
1.51
Older
1.17
1.08
0.91
Unknown
1.07
1.06
0.91
Female
1.91*
1.46*
1.91"
BiosBias
1.34
1.12
1.33
Male
1.42*
1.63*
1.44
Female
1.32*
1.34"
1.43*
RAVDESS
1.02
1.01
1.28
Male
1.30
1.36*
1.12
Hypothesis
proposes that equalizing coverage with the conditional treatment will cause more dis- parate impact than the marginal treatment. Our data in (Table 2 strongly supports this conclusion where we find that the value max is consistently higher than maxROR Even for RAVDESS where the marginal treatment did not show disparate impact, the conditional treat- ment introduced noticeable bias.
6.2 INSIGHTS
Key factors impacting the fairness of human performance We aim to identify the factors con- tributing A., disparate impact in accuracy improvement, visualized in Figure I] A, measures the difference in accuracy improvement over the control group between the most and least improved groups. To investigate this, plot in FFiuure the differences between these groups across four key factors: coverage, adoption, average set size, and singleton frequency. Adoption measures the fraction of answers participants selected from the prediction set, while the singleton frequency refers to how often singleton set was provided, both of which were identified by Cresswell et al. (2024) as factors contributing to how helpful prediction sets are to humans. We note that the differences for these four quantities can be negative because we select the max and min groups based on accuracy improvement- - they are not necessarily the max and min groups respectively for the four quantities.
It is clear that the conditional treatment achieves coverage differences that are closest to zero (Equal- ized Coverage). However, this does not translate to lower A,. Instead, experiments with relatively large (in magnitude) coverage difference, showed the most fair outcomes (c.g. RAVDESS LAVDESS-Marginal where the best and worst groups saw coverage levels differ by 4 percentage points, but still had the same accuracy). Additionally, there is no apparent correlation between coverage difference and A which implies coverage differences do not drive human accuracy outcomes. One might suppose that if the participants relied on sets more for some groups than others, disparate impact could arise. However, like for coverage, we see no strong correlation between the adoption difference and A.. The experiment with the single most fair outcomes also had the largest adoption difference.
By contrast, we see strong correlation between set size differences and A. The most fair outcomes occur when sizes are similar across all groups, for RAVDESS RAVDESS-Marginal, or FACET-Ave-E K. The conditional treatment which increases set size differences in its effort to equalize coverage also increases disparate impact. Similar to set size, there also clear correlation between singleton fre- quency and A. [Cresswell et al. (2024) observed that human accuracy on tasks was highest when the model expressed certainty through singleton sets. Hence, groups with more singletons could expect higher accuracy, so it is plausible that equalizing how often singletons occur would encourage fair outcomes, which is what we observe. Users should be aware that group cropiiconditional modifications such equalizing singleton frequency are susceptible to Yule effects (Ruggieri al. 2024)..
In summary, our results show that simply equalizing coverage, as is the aim of the conditional treatment, is not effective way to promote fair outcomes. Instead, focusing on Equalized Set Size or Equalized Singleton Frequency could be more useful standards of fairness for set predictors.
The role of group difficulty in disparate impact Figure 5 illustrates accuracy the FACET dataset across different age groups for both the model and humans (control), and when broken down by treatment. We observe that both the model and humans are consistent in which groups they

---

Published as a conference paper at ICLR 2025
Avg
BiosBias-Arg B BicsBias-Conditionn Conditional
RAVDESS Avg
FACET-Marginal FACET-Conditional
RAVDESS-Marginal
Accuracy Disparate Impact.
RAVDESS-Condiöon
2
4
15 Singleton Frequency Difference
60
Coverage
Adoption
Set Size Difference
Difference
Difference
Figure 4: Accuracy disparate impact A, compared the difference between the most and least improved groups across key factors for various datasets and treatments. From left to right: coverage, adoption, average set size, and singleton frequency.
RACET
performance improwment FACET
80
Performazze Imprr improsmeentt
Group
L,77
Younger
40
Middle
Older
20
Unknown
Aug-KK
Marginal
Figure 5: Accuracy by group for FACET. Left: Human accuracy (Control) and model Top-1 Accu- racy across groups. Right: Human accuracy across groups and treatments.
find more or less difficult, with model showing greater variance between groups. Providing marginal conditional prediction sets tends to improve human performance more for the group the model had highest accuracy on (Younger), while offering less benefit- and in some cases even worsening performance- for the lowest model accuracy group (Older). This analysís demonstrates how prediction sets may contribute to disparate impact by disproportionately helping easy groups and harming hard groups, widening the gap for outcomes.
7 CONCLUSIONS & LIMITATIONS
In this work we have presented the first experimental study on the fairness of conformal prediction sets human decisioning aid. Our data runs contrary the prevailing wisdom in the research community we find that equalizing coverage (Romano et al., 2020a) across groups actually in- creases unfairness outcomes. Based these results, we instead recommend that practitioners aim for Equalized Set Size or Equalized Singleton Frequency across protected groups, as both set size and singleton frequency correlate much more strongly with outcomes than coverage does.
Still, any study, especially those involving humans, will have its limitations. While we recruited 600 unique participants and collected 30,000 individual responses, some the statistical measures in Table 2 did not achieve significance at the 5% level due to insufficient observations. This is consequence of the inherent imbalance between groups the datasets we used, which is expected to occur in real-world data, and is crucial aspect for our discussions on fairness. In particular, the FACET dataset was divided into four groups compared to only two for BiosBias and RAVDESS, which means fewer examples from each FACET group were shown to participants, and hence our conclusions have less statistical significance on them. The low significance on individual groups mitigated by the consistent results together provide clear evidence for our hypotheses.
across three independent datasets, which when taken
Additionally, experiments only used classification tasks, whereas conformal prediction has also been applied to regression (Romano et al., al. (201)), time-series (Stankeviciute et al. (021). and natural language tasks (Mohri & Hashimoto| (2024). Other instantiations of conformal prediction may be susceptible to the same unfairness mechanism we proposed in Figure 2] where difficult groups tend be undercovered, requiring larger sets for Equalized Coverage which disadvantage. but this is out-of out-of-scope for our study.
10

---

Published as a conference paper at ICLR 2025
Ethics Statement Throughout our research, took steps to ensure that research ethics up- held. We acknowledge that ethical considerations are extremely important for any research involving human subjects, and we have ensured that research meets the Code of Ethics for ICLR 2025. We also followed the ethical standards of our institution. Although does not have an internal review board (IRB) process, the steps took to meet its standards included: understanding the protocols and procedures for ethical research at our institution; curating datasets that only contained content suitable for showing to participants; performing the experiments on ourselves authors; providing participants with clear consent form that detailed the data that would be collected as well as how would be used and how long would be retained; paying all recruits a fair wage; and ensuring we were aware of potential biases in the data we collected by reviewing the demographic distribution of recruits (Table 8).
Reproducibility Statement We have made two specific efforts to promote the reproducibil- ity of our results. First, we have released our codebase and the data we collected at github. com/layer6aaillab/ conf oorforomallao prrprorrrrreeiiiiiiiio.ooaaaaaaa searchers to perform the same set prediction methods, generate the same dataset splits, and perform the same statistical analysis. Second, we have extensively described experimental setup in Ap- pendix B) including images the screens that were shown to human participants throughout the experiment. The description is complete enough for third party faithfully recreate and repro- duce our experiment.
which will enable other
ACKNOWLEDGMENTS
We would like to thank Noël Vouitsis for comments on a draft this manuscript.
REFERENCES
Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra Acharya, Vladimir Makarenkov, and Saeid Nahavandi. A review of uncertainty quantification in deep learning: Tech- niques, applications and challenges. Information Fusion, 76:243-297, 2021. ISSN 1566-2535. doi: https://doi.org// 10.1/1/000.......... (t000//..ifffu..20000 2nffus.2221.05.0000
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In The 25th ACM SIGKDD Interna- fional Conference on Knowledge Discovery & Data Mining, 2019a.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama, Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '19, Associ- ation for Computing Machinery, 2019b. ISBN 978 9781450362016. doi: 10.1145/399250033330701 URL https: /tt.://oo.iooo 100 1145/ 00055/229500022 3330701
Junaid Ali, Preethi Lahoti, and Krishna P. Gummadi. Accounting for model uncertainty in algo- rithmic discrimination. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, 2021. ISBN 978 9781450384735. doi: N0.1145/4461002.346223
Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-fret uncertainty quantification. urXhv:2107.07511, 2021.
Anastasios Nikolas Angelopoulos, Stephen Bates. Michael Jordan, and Jitendra Malik. Uncertainty sets for image classifiers using conformal prediction. In International Conference on Learning Representations, 2021.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, wav2vec 2.0: A frame- work for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems, volume 33, 2020.
Richard A. Berk, Arun Kumar Kuchibhotla, and Eric Tchetgen Tchetgen. Improving faimess in criminal justice algorithmic risk assessments using optimal transport and conformal prediction sets. Sociological Methods Research, 2023a. doi: 10. 10.1177/00912242211 1558883.
11

---

Published as a conference paper at ICLR 2025
Richard A. Berk, Arun Kumar Kuchibhotla, and Eric Tchetgen Tchetgen. Fair risk algorithms.
Anmual Review of Statistics and Its Application, 10:165-187 2023b. ISSN 2326-831X, doi:
https://doi.org/11.11
144/ammurex- wtt.://.orrrro....oo/oooeeeeeeee
Evgenii Chzhen, Christophe Denis, Mohamed Hebiri, and Titouan Lorieul. cessification-veervves via a unified framework. arXiv:2102.1 12318, 2021.
Set-valued
Jesse C. Cresswell, Yi Sui, Bhargava Kumar, and Noël Vouitsis. Conformal prediction sets im- prove human decision making. In Proceedings of the 41th International Conference Machine Learning, 2024.
Maria De-Arteag, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* '19. ACM, January 2019, doi: 10. 10.1145//287500.32875555
URL htttht//p/dxx.oo doi org/ 11411 32877560
Giovanni De Toni, Nastaran Okati, Suhas Thejaswi, Eleni Straitouri, and Manuel Gomez-Rodriguez. Towards Human-AI Complementarity with Predictions Sets. arXiv:2405.17544, 2024.
Zhun Deng, Cynthia Dwork, and Linjun Zhang. HappyMap A Generalized Multicalibration Method. In 14th Innovations in Theoretical Computer Science Conference, volume 251 of Leibniz International Proceedings Informatics, 2023.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Confer-
ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), 2019. doi: 10.18653/v 1/119//1/N11-1433,
Tiffany Ding, Anastasios Angelopoulos, Stephen Bates, Michael Jordan, and Ryan J Tibshirani. Class-conditional conformal prediction with many classes. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Con- ference, 2012. ISBN 9781450311151, doi: 10. 10.114555 147114499022662200002
Maria S. Esipova, Atiyeh Ashari Ghomi, Yaqiao Luo, and Jesse C Cresswell. Disparate impact in differential privacy from gradient misalignment. The Eleventh International Conference on Learning Representations, 2023.
Wiam Fadel. Fine-tuned XLSR-53 nition on ravdess dataset, 2023.
large model for speech emotion recog-
URL https: h/wg///ggggggg.//e/e/e//o/o/ co/Wi
las2vec2-1argee -eweer----ennnih-----eeeeeeeeeee naveetusnddrddesss--vve Accessed 2024-09-25.
Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubra- manian. Certifying and removing disparate impact. Proceedings of the 21th ACM SIGKDD In- ternational Conference Knowledge Discovery and Data Mining, 2015. ISBN 9781450336642. doi: 10.1145/27832582278333
Rina Foygel Barber, Emmanuel Candès, Aaditya Ramdas, and Ryan J Tibshirani. The limits of distribution-fiet free conditional predictive inference. Information and Inference: A Journal of the IMA, 10(2), 08 2020. ISSN 2049-8772. doi: 10. 00.1093/mmaiaiiisa001..
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference Machine Learning, pp. 1050-1059.
PMLR, 2016.
Georgi Ganev, Bristena Oprisanu, and Emiliano De Cristofaro. Robin Hood and Matthew Effects: Differential Privacy Has Disparate Impact on Synthetic Data. In Proceedings of the 39th Interna- tional Conference on Machine Learning, volume 162, pp. 6944 6959, 2022.
Isaac Gibbs, John Cherian, and Emmanuel Candès. Conformal prediction with conditional guar- antees. arXiv:2305. 22..22055./66,, 2023.
12

---

Published as a conference paper at ICLR 2025
Nina Grgié- Hlača, Muhammad Bilal Zafar, Krishna P Gummadi, and Adrian Weller. The case for process fairness in learning: Feature selection for fair decision making. In Symposium on Machine Learning and the Law at the 29th Conference Neural Information Processing Systems, 2016.
Eugen Grycko. Classification with set-valued decision functions. In Information and Classification, pp. 218-224, Springer Berlin Heidelberg, 1993. ISBN 978-3-642-509742.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. PMLR, 2017.
Laura Gustafson, Chloe Rolland, Nikhila Ravi, Quentin Duval, Aaron Adcock, Cheng-Yane Fu, Melissa Hall, and Candace Ross. Facet: Fairness in computer vision evaluation benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20370-20382, 2023.
Moritz Hardt, Eric Price, Erie Price, and Nati Srebro. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems, volume 29, 2016.
Eyuaityy
Ursula Hebert-Johnson Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Cal- ibration for the Commptaaiioalll--denfffaa Conference on Machine Learning, 2018 8.
masses. In Proceedings of the 35th International
Jianguo Huang, Huajun Xi, Linjun Zhang, Huaxiu Yao, Yue Qiu, and Hongxin Wei. Conformal pre- diction for deep classifier via label ranking. Proceedings of the 41 International Conference Machine Learning. 2024.
Huggingface. Bert base model (uncased), 2024. URL httptt/s//nggggggc..eoo///////o/ Accessed 2024-09-25.
Arun K Kuchibhotla and Richard A Berk. Nested conformal prediction sets for classification with applications to probation data. The Annals of Applied Statistics, 17(1):761-785, 2023,
Matt Kusner, Joshua Loftss, Chris Russell, and Ricardo Silva. Counterfactual fairness. In Advances in Neural Information Processing Systems, volume 30, 2017.
Selim Kuzucu, Jiaee Cheong, Hatice Gunes, and Sinan Kalkan. Uncertainty-based faimesss mea- sures. arXiv:2312.11299, 2023.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30, 2017.
Jing Lei and Larry Wasserman. Distribution-free Prediction Bands for Non-parametric Regression. Journal of the Royal Statistical Society Series B: Statistical Methodology, 76(1):71-96, 07 2013. ISSN 1369-7412. doi:
Kung-Yee Yee Liang and Scott L Zeger. Longitudinal data analysis using generalized linear models. Biometrika, 73(1):13-2, 1986.
Meichen Liu, Lei Ding, Dengdeng Yu, Wulong Liu, Linglong Kong, and Bei Jiang. Conformalized fairness via quantile regression. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 1561-11177.. Curran Associates, Inc., 2022.
Steven R. Livingstone and Frank A. Russo. The Ryerson Visual Database of Emo- tional Speech and Song (RAVDESS), April 2018. /uR.attter//abi..o0//000222////0 1188976
Titouan Lorieul, Alexis Joly, and Dennis Shasha. Classification under ambiguity: When is average-k better than top-k? 1xXi::211200885. 2021.
Charles Lu, Andréanne Lemay, Ken Chang, Katharina Höbel, and Jayashree Calpathy-Cramer Fair conformal predictors for applications in medical imaging. In Proceedings of the AÅAI Conference on Artificial Intelligence, volume 36, pp. 2008-12016 2022.
13

---

Published as a conference paper at ICLR 2025
Raghav Mehta, Changjian Shui, and Tal Arbel. Evaluating the fairness of deep learning uncertainty estimates medical image analysis. In Medical Imaging with Deep Learning, 2024.
Tanushree Mitra, C.J. Hutto, and Eric Gilbert. Comparing Person and Process-Centrry Strategies for Obtaining Quality Data on Amazon Mechanical Turk. Pruceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 1345-1354, 2015. doi: 10.1145/ 2702123.2702553
Christopher Mohri and Tatsunori Hashimoto. Language models with conformal factuality guaran- tees. In Proceedings of the 4lst International Conference on Machine Learning. 2024.
Radford M Neal. Bayesian Learning for Neural Networks, volume 118. Springer Science & Busi- Media, 2012.
Pavlovia. https//wwww.avllovia.rrg. 2024. URL 01-24.
org/ Accessed 2024-
Jonathan Peirce, Jeremy R. Gray, Sol Simpson, Michael MacAskill, Richard Höchenberger, Hi- royuki Sogo, Erik Kastman, and Jonas Kristoffer Lindeløv. PsychoPy2: Experiments in be- havior made easy. Behavior Research Methods, ((():115-203, Feb 2019. doi: 10.3758/
113428-018-01193--
Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. On fairness and calibration. In Advances in Neural Information Processing Systems, volume 30, 2017.
Prolific. https://www.polffic.comm/ 2024. URL 01-24.
com/
Accessed 2024-
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the 38th International Conference on Machine Learning, volume 139, pp. 8748-8763. PMLR, 18 24 Jul 2021.
Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. In Advances in Neural Information Processing Systems, volume 32, 2019.
Yaniv Romano, Rina Foygel Barber, Chiara Sabatti, and Emmanuel Candès. With Malice To-
ward None: Assessing Uncertainty via Equalized Coverage. Harvard Data Science Review, (2), 2020a.
Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adaptive cover- age. In Advances in Neural Information Processing Systems, volume 33, pp. 3581-31111 1581-3591, 2020b.
Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adaptive cover- age. In Advances in Neural Information Processing Systems, volume 33, 2020c.
Salvatore Ruggieri, Jose M. Alvarez, Andrea Pugnana, Laura State, and Franco Turini. Can We Trust Fair-AI? Proceedings of the AAAI Conference on Artificial Intelligence, 37(13):15421- 15430, Jul. 2024. doi: 10.1609/aaai 1111:////aaa..33111..6667799 v37:13.26798.
Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008
Christian Soize. Uncertainty Quantification Springer, 2017.
Kamile Stankeviciute, Ahmed M. Alaa, and Mihaela van der Schaar. Conformal time-series fore- casting. In Advances in Neural Information Processing Systems, volume 34, 2021.
George Stein, Jesse C. Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Leigh Ross, Valentin Vil- lecroze, Zhaoyan Liu, Anthony L. Caterini, Eric Taylor, and Gabriel Loaiza-Ganena Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. In Teirth-seventh Conference Neural Information Processing Systems, 2023.
14

---

Published as a conference paper at ICLR 2025
Eleni Straitouri and Manuel Gomez Rodriguez. Designing decision support systems using counter- factual prediction sets. In Proceedings of the 41 International Conference on Machine Learning, volume 235, 2024.
Eleni Straitouri, Suhas Thejaswi, and Manuel Gomez Rodriguez. Controlling counterfactual harm in decision support systems based on prediction sets. arXiv:2406.00677, 2024.
Vladimir Vovk. Conditional validity of inductive conformal predictors. In Proceedings of the Asian Conference on Machine Learning, volume 25, 2012.
Vladimir Vovk, David Lindsay, Ilia Nouretdinov, and Alex Gammerman. Mondrian confidence machine. Technical Report, 2003.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in Random World. Springer, 2005.
Volodya Vovk, Alexander Gammerman, and Craig Saunders. Machine-learning applications of al- gorithmic randomness. In International Conference on Machine Learning, pp. 444 453, 1999.
Fangxin Wang, Lu Cheng, Ruocheng Guo, Kay Liu, and Philip Yu. Equal opportunity of coverage in fair regression. Advances in Neural Information Processing Systems, 36, 2024.
Tal Yarkoni. The generalizability crisis. Behavioral and Brain Sciences, 45:e1, 2022.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In Proceedings of the 30th International Conference on Machine Leurning, volume 28, 2013.
Dongping Zhang, Angelos Chatzimparmpas, Negar Kamali, and Jessica Hullman. Evaluating the utility of conformal prediction sets for ai-advised image labeling. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, 2024. ISBN 9798400703300. doi: 10.1145/3613904.36444
Yanfei Zhou and Matteo Sesia. Conformal classification with equalized coverage for adaptively selected groups, 2024.
A IMPLEMENTATION DETAILS
Our code for curating datasets and performing calibration and set prediction is available at gi tithub. cwm/ wal.araaallaas/eeeeaaaaaaaaaaaee ooaanforalleee prrceooooaleeeeeeeieeeaaaaaaaea In this section we detail the steps that were taken.
A.1 DATASET PRE- PRE-PRCCESSING
High-level information for the calibration-valiiationn, calibration, and test datasets shown in Table 3] The following sections provide detailed explanation the pre-processing steps applied each dataset. Dataset pre-processing and prediction does not require extensive computing resources. We used an Intel Xeon Silver 41 14 CPU and TITAN V GPU, which took total less than hour to process all three datasets.
Table 3: Dataset Information
Dataset [D
[Deall
[Dest
Total Classes Used Classes (m) Groups (ng)
FACET
1400
4000
1400
52
20
4
BiosBias
5000
10000
2000
10
RAVDESS
240
840
360
2
15

---

Published as a conference paper at ICLR 2025
Table 4: FACET Group Counts
Table 5: BiosBias Group Counts
Table 6: RAVDESS Group Counts
Group
Dealval
Deal
Dus
Group Deaival
Deal
Dust
Group
Deatval Deal
Dust
Younger
254
711
276
Female
2424 4887 969
Female
120 420
180
Middle
772
2144
729
Male 2576 5113 1031
Male
120 420 180
Older
103
299
91
Unknown
271
846
304
The FACET dataset (Gustafson et al. 2023) created to investigate bias in image clas- sification tasks, with images of people annotated by human labelers, and sensitive attributes includ- ing perceived age, gender, and skin tone. We used the occupation annotations as class labels, and age annotations for groups Age groups are categorized as 'Younger' (perceived to be < 25 years old), 'Middle' (25-65) 65), 'Older' (> 65), and 'Unknown', . Preprocessing involved filtering for images con- taining a single person and a single occupation label. Images were resized to 224 x 224 pixels and center-cropped for consistency. We focused on the top m 20 most common occupations which were: [backpacker, boatman, computer user, craftsman, farmer, guard, guitarist, gymmast, hair- dresser, horseman, laborer, lawman, motorcyclist, painter, repairman, seller, singer, skateboarder, speaker, tennis player]. Since some of these labels are unintuitive, we clarified them for presentation to participants by renaming them to: [Backpacker, Boatman, Computer User, Craftsman, Farmer, Guard, Guitarist, Gymnast, Hairdresser, Horse Rider, Laborer, Officer, Motorcyclist, Painter, Re- pairman, Salesperson, Singer, Skateboarder, Speaker, Tennis Player). For conformal prediction, the remaining dataset split into 1400 ciaibration-valddation samples, 4000 calibration samples, and 1 1400 test samples, using stratified sampling to ensure balanced occupation representation across subsets. Groups, however, were not stratified or balanced, and followed the natural distribution in the dataset (see Table 4). The calibration cuabratioo-validattinn validation, calibration, and test sets were treated identi- cally, making them i.i.d. for conformal prediction. FACET is released under a custom license that allows it to be used "for the purposes of measuring or evaluating the robustness and algorithmic fairness of Al and machine-learnng vision models, and solely on a non-commercial and research basis".
FACET
BiosBias The BiosBias dataset (De-Arteaga et al. 2019) contains public biographies labeled by occupation and annotated with binary gender groups. Biosbias created occupation labels automat- ically by extracting information from the biographies. They also extracted likely binary gender information from 3rd person pronoun use. We used occupations as class labels, with two groups, "Female" and "Male". To preprocess BiosBias data, we removed non-ASCII characters, URLs, emails, phone numbers, and redundant punctuation, then truncated the biographies to 400 charac- ters to limit variance due to vastly different text lengths. We generated text representations of cach biography using a pre-trained BERT model (Devlin et al. 2019), and trained a linear classifier to predict occupations using training set of 50,000 samples with validation set of 5,000 samples for manual hyperparameter selection and evaluation. From the original 28 occupations, we focused on m 10 of the most common: [Professor, Physician, Photographer, Journalist, Psychologist, Teacher, Dentist, Surgeon, Painter, and Model]. These were not strictly the 10 most common occu- pations since we found that the data quality of some occupations was noticably worse than others; for example we excluded the textitNurse class because it contained hundreds of near-duplicate biogra- samples, 10,000 calibration
phies. For conformal prediction, we used 5,000 calibration-validaut samples, and 2,000 test samples, using stratified sampling to ensure balanced representation across all occupations. Groups were not stratified, but were already roughly balanced in the dataset which is reflected in our splits (Table 5). This method of splitting ensured the calibration and test sets could be treated as i.i.d. for conformal prediction purposes. BiosBias is released under a MIT license.
RAVDESS For RAVDESS (Livingstone Russo, 2018), dataset of audio-visual recordings where actors express emotions, we worked with 1,440 audio-only samples exhibiting m = 8 emo- tions: [Happy, Angry, Calm, Fearful, Neutral, Disgust, Sad and Surprised]. The dataset contains recordings from 24 actors, 12 female and 12 male, so we used emotion the class label and gender as the protected group. Each actor recorded 60 snippets of approximately 4 seconds in length each. The 60 recordings of 8 emotions. However, the "Neutral" emotion used only a single intensity. We split the dataset into 240 for calibration vatbration-validation, 840 samples for calibration, and 360 for testing using stratified
broken down as 2 re-recordings each of 2 intensities each of 2 statements
16

---

Published as a conference paper at ICLR 2025
sampling over both emotion and gender. Given that each split treated the same way they can be considered i.i.d. for conformal prediction. To process the audio samples, we used a wav2 model [Bevvsk et al. (2020) available on Huggingface (Fadel 2023). We resampled each file to 16kHz as the base model was pre-trained on that frequency. For audio with multiple channels, we changed it to mono by averaging channels. The model's feature extractor then turned the raw audio signals into feature vectors ready for the model to use, which were subsequently padded to ensure consistent input length for batch processing by the model. RAVDESS released under a Creative -SarreeAl--aareAkkkkee 4.0 International license.
Commons
A.2 CONFORMAL PREDICTION METHODS AND HYPERPARAMETERS
When generating conformal prediction sets for our human experiments, we aimed to diversify the settings by testing two different score functions. This helps demonstrate that our hypotheses and conclusions are not a result of using one particular score function. We chose two highly-performan methods for our tasks:
RAPS Regularized Adaptive Prediction Sets (Angelopoulos et al. 2021) builds off of APS (Ro- mano et al., 2020b) by penalizing the score for sets with more elements than threshold keeg. In addition RAPS uses temperature scaling on the model's logits before the softmax layer with value T,, and a weight À on the regularization term. First, RAPS defines (z()) - Σy= f(r) l[f(x) > [()yy] as the total probability mass of the labels which have higher softmax values than for input a, and 0z (z(y) = (v/ E Y f()) 2 y, as the ordinal ranking of y among all labels, again based on softmax values. RAPS constructs prediction sets as
Cę(x)= (v (=(()) + f(x), + M(o - ) ) S SG)-
(12)
where u ~ U|0 1] is a uniform random variable. Here, the score function s(x, y) has three terms that use the probability mass of classes more likely than y, the probability of y with a random weighting, and regularization term that penalizes adding more than kreg classes.
SAPS Sorted Adaptive Prediction Sets (Huang al. 2024) is a more recent procedure that aims to produce compact prediction sets with better conditional coverage compared to other methods. SAPS has one primary hyperparameter, X, which governs the weight given to the ranking information. As with RAPS. temperature hyperparameter T is optimized to scale logits before applying the softmax. SAPS discards all probability values except the maximum softmax probability, while preserving the ranking of the labels. The score function is defined as
s(r, v) = {u ((x)) max f(x) + X(oz()) - 2 2 u), otherwise,
if (x) 1,,
(13)
where U00 1] is again a uniform random variable. This score function minimizes the effect of unreliable small softmax values while retaining enough ranking information to adjust the prediction set size based on instance difficulty.
Hyperparameter Tuning For prediction set construction, we used three splits of the data: calibration-validation Deaval, calibration Deat, and test Dreat Deatall was employed for hyperpa- rameter tuning, Deat was used to calculate conformal thresholds with the tuned hyperparameters, and finally Diest was used to generate the prediction sets used in our human experiments. Table presents the final hyperparameters after automated tuning.
For hyperparameter optimization with CP methods we employed Bayesian optimization via the Op- tuna library (Akiba et al., 2019b). utilizing the TPESampler for efficient search over 50 iterations aiming to minimize average set size. Specifically, for each parameter setting, the conformal thresh- old ) was determined using D then prediction sets were generated on Denlval where their average size was computed. For Mondrian CP, within every iteration of tuning the same hyperparameter settings were used each group.
Tuning for avg-k is necessarily different because the main hyperparameter & is the target average set size. Hence, instead of tuning for minimal average set size which would be trivial we tuned k to achieve the desired 90% coverage rate, matching the CP methods. & was adjusted through a binary search by computing the threshold qu Deat, then generating sets on Deaival and evaluating their empirical coverage. We refined the average set size k through binary search up to five decimal points of precision to ensure that the coverage Deatall closely matched the predefined target coverage.
17

---

Published as a conference paper at ICLR 2025
Table 7: Hyperparameter Settings for Each Dataset After Tuning
Marginal
Conditional
Dataset
Score Function
T
Kwg
T
FACET
RAPS
0.32
0.17
0.38
0.46
BiosBias
SAPS
0.53
0.21
0.53
0.26
RAVDESS
RAPS
0.26
10.49
0.20
0.43
3
B HUMAN SUBJECT EXPERIMENTS
In this section we give complete details experiments involving humans which extends the descriptions in Section 4 and SSection
Participant Recruitment We designed our human subject experiments using PsychoPy (Peirce et al. (019) and made them available on Pavlovia (Pavlovia 2024). Participant recruitment was conducted through Prolific (Prolific 2024). We required participants volunteering for the study to be fluent in English, the language we used throughout the experiments, and to use desktop or laptop computer, but otherwise did not filter candidates on any criteria. In total, 600 unique participants were recruited, and each was randomly assigned to single (task, treatment) pair, of which there were 12 total for 50 participants each. Hence, each experiment involved a disjoint set of participants. Participants were paid according to the platform guidelines enforced by Prolific participants were given a flat rate for completing their assigned experiment calculated based on the median completion time of their cohort. We also offered participants bonus pay in the amount of 0.01 GBP for every correct answer they gave on the 50 test trials, which incentivizes high quality answers. On average, participants received pay at rate of 9.75 GBP/hour which totaled 1500 GBP in participant renumeration across all tests.
Across the experiments, 599 participants gave consent for their demographic data be collected and shared in aggregate, while one participant withdrew their consent. As shown in Taall 8) the study population is fairly balanced in terms of gender. Participants in the study represented various age groups and ethnicities, with more being in their 20s and identifying as White respectively.
Table 8: Demographics of Participants
Group
Participants
< 20
31
20-29
324
30-39
144
Age group
40-49
64
50-59
26
60
Unknown
Female
287
Gender
Male
311
Unknown
2
White
346
Black
179
Mixed
34
Ethnicity
Asian
23
Other
15
Unknown
3
Experiment Details Our experimental design follows that of'Cresselll et al. (2024). al. of the screens that were presented to participants is shown in Figure 7 Jsing the BiosBias task with marginal treatment as an example. We will now walk through the progression of these screens, describing the test. For each experiment, participants were first shown consent form detailing what data we were collecting, how we intended use it, and how long we would retain it. Participants who did not consent could remove themselves from the study rather than proceeding. To limit
An overview
18

---

better Published as a conference paper at ICLR 2025
risks participants, we did not collect any personally identifiable information like name, birth date, or address. As noted above, some demographic information had been provided by participants to the Prolific platform, and nearly all participants consented to obtaining this information. We determined that there were no risks to participants taking the study that needed to be disclosed.
We proceeded present participants with instructions about how take the test and about the classification task they would be performing. We provided one labeled example datapoint from each class for participants
familiarize themselves with, and then conducted 20 practice trials that used the same format as the real test trials. For example, the main trial screen for BiosBias is shown in Figure 7] while Figure shoos FACET. and Figure shows the RAVDESS task. The practice phase was for training participants to ensure they would achieve high accuracy on the real test trials, so we did not use data collected on the practice trials. Example and practice datapoints were sourced from Diest, and were not reused for the real test trials. After the practice trials, 50 test trials were given, and the data collected here comprises what we used for analysis. For each task we randomly sampled sets of 50 datapoints from Prest without replacement using one of 10 fixed seeds. Each participant was shown one of these random sets. The use of only 10 seeds was intentional so that multiple participants across treatments would see the same datapoints. Since datapoints can have different inherent difficulty, using the same examples across treatments reduces variance from random data selection. Overlap between the examples shown to different participants also allows model their inherent skill using our GEF analysis.
The main trial screens all show a participant ipants had unlimited time to select one class, after which the correct answer was shown following previous studies in machine learning (Stein al. (003). Providing answers after each trial has been shown to increase the quality of collected data (Mitra et al. 2013) and allows the participant to con- tinually optimize their decision strategy. This is desirable, as we want participants to give the most accurate answers possible, even if their strategy evolves over the course of the test.
datapoint æ, and all classes Y options. Partic-
For the au recorting, select the most appropriate emotion.
Press the kay to play the mecorcnng again.
Suggestions: 90% probabilly Neutral
1. Neutral 2. Calm Hepoy Sad Fearl Diegi et & Surpried
5 Anggy
answer
Press SPACEBAR contrue
Figure 6: Main trial screen shown to participants for RAVDESS with marginal conformal set treat- ment. When the screen was first displayed, the audio recording was played, with the model sug- gestions appearing and ability enter response activated after the recording completed (roughly seconds). Participants were able to replay the recording as many times as needed. The correct answer was shown after the participant responded.
19

---

Published as a conference paper at ICLR 2025
In atadyy we edll colne your praovrr to quetiiions and your unanymiced dermogu raghi data provided internation ridentiiy than your Pirdt Bc prommnnt rograame dara pendaat
you cansapating dt.dy
Hurarr pensongge regiing aogeesbore trome cnmeeaatiinn ariifonal reri perce
clicting
uaud
In sludy, would lee duaemn ew the turress burnan cocilion makking a impociiod by auggessionn
siorec for years improduaiinn
Throughoul thi expertuent you wdll sae encrt bisgraphe on people be asradd detarrere occupation, usent your decaions,
but to oiked aalo rookto
enneert thi it Pruilo
will use heyboont to respand cistion bitotec commajonccng
you undesstand above into motion and givn your concert undersaan infermadonn preas
Press SPNCEBAR continu
choose you ahot bogauph posinn Your hoo persan to deeciioec
ssaidd your deciaion i
brancd
mosel provid axgpenders niine timn, one the anownns excnsici coneet
higgapplees tin profeniinaal and de commas conanst inmeced orfend you.
badgemnnti bricwcr out all options, dver if disagio with Rippeedliee
The 10 optira unall ucko you will como in food uocor The chort it will be procontsd the somo order. appeediine
you
side Loograp
A KlSayoa Twessa ichs aubltty seeii
Paper
"wemeer
fave unn the reat toallls resped sour silon pounton
Eroure undlesdanc
Prede BPNC ESFA 30 500 conthu)
Theas SolcoDAf whar wady
nost proip cocupation
Gora hicaaaaue the Fuand
Partggase, Sporish and un Hie worr hus
The will moro trals.
Nao
Mother Clast,
flcuusnss so phosse conu them ozu tecitions
Soggestons probutiliyy anawel
Pantoysapher Jiamooker
A
Zecwers you prowido, bringit the mooo mum poy piiluding
propent
sonnect
1. Pryeeaan 2. Fttekggaptee 3. Juarroii Phychoooepe Teacher Dental Surgon Parlee Pooczsor
Pheas SENCCDAA
bagin stucd
The bad snanet Prohopauhhe
Pheas SINCESAM
Figure 7: Screens displayed to participants during our experiment using marginal conformal sets on BiosBias, with similar template for other experiments. The seventh panel (left to right, top to bottom) shows an example of how the practice and test trials appear, with the correct answer displayed after the participant entered their response. The fourth panel was not shown to Control group as they were not given suggestions from the model during the trials.
20


Reviews:
[{"review_id": "gUWzlun6Fq", "reviewer": "Reviewer_5hcH", "summary": "This work is motivated by the well-known problem of group-conditional coverage in conformal prediction: when an underlying predictive model is more inaccurate on some subgroups than others, achieving equalized coverage over subgroups requires providing prediction sets that vary in size across subgroups. The work conducts a human study in which human decision-makers are presented with prediction sets as a decision aid; the study suggests that their accuracy improves at different rates across subgroups.", "questions": "* Fig 1: what is the x-axis? \n* Table 1: is the \"coverage\" column for the conditional method computed on average? \n* Table 2: significance levels are for the null of OR = 1?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The authors rightly identify \"equal coverage\" as a limited notion of fairness for prediction sets, in the sense that the end-goal of providing prediction sets is ultimately to improve a downstream task -- i.e., equalized coverage in itself ought to be desirable only to the extent that it improves the utility of predictions in general. Evaluation via human study is also an important perspective; I see this work as playing a similar role as the work that conducted human evaluations of explainability methods. I also appreciate that authors mention (non conformal) methods for providing prediction sets.", "weaknesses": "* The main weakness of this work to me is that its hypotheses are a priori unsurprising. If some subgroup A requires larger set sizes for the same coverage level, it is tautologically true that there is higher uncertainty for subgroup A and that any class in the prediction set is less likely to be the ground-truth label. Even if a person relied 100% on the prediction set and simply picked uniformly at random among the predictions, one would expect performance to be worse on groups with bigger set sizes. \n* Therefore, I find the results to be less 'disquieting' than the work seems to suggest. Over the baseline of no assistance, it seems that all methods approximately improve accuracy, and I don't really see why it's a bad thing that some groups benefit more; the paper does not make a case _against_ using conditional CP methods for bias reasons. (That is, why artificially reduce accuracy, i.e. worsen outcomes, for the easier subgroups? It's not clear to me how that would be normatively defensible, e.g. in a high stakes medical task, and I would push back on the language of \"increases unfairness\".) It would certainly be a bad thing if conditional CP somehow made human prediction worse than marginal CP; a different question that could/would have been interesting to ask is not whether max disparity increases, but rather whether accuracy actually improves from seeing conditional prediction sets or not. That is, something like H0: for all subgroups, accuracy using marginal and accuracy using conditional prediction sets is the same; H1: for at least some subgroups accuracy increases or decreases. \n* It is also surprising to me that the baseline is no assistance rather than providing a point (top-1) prediction; the latter seems like the more natural comparison for a prediction set and could potentially illustrate more significant per-subgroup disparities, especially with more variation in top-1 accuracy. \n* Finally, minor comment on experimental design - I should preface this comment by saying I'm not an expert in this kind of study design (though I am familiar with other approaches to randomized experiments and hypothesis testing). However, I am not confident that the numerical results (i.e. those summarized in Table 2) lend themselves to clear statistical interpretations - e.g., given the computed point-estimate ORs, how are we meant to interpret (max) ROR, and where does that uncertainty propagate? How are significance thresholds computed? This is especially concerning given that for FACET, the only dataset where set sizes differ by more than 1, all the ORs themselves essentially all hover around 1/ just barely at significance. That said, I think the primary value of this work is the high level characterization of how humans use and interpret prediction sets (as opposed to applications of RCTs in econ/medicine/etc, where precise interpretations and therefore rigorous experimental design are more consequential). \n\n======\n\nMinor presentation notes:\n\n* Exchangeability - should probably be formally defined. Saying that it is \"realistic in practice\" (086) is a pretty strong statement, in my opinion. I personally consider the exchangeability/iid distinction to be more formal than practical - I've very rarely seen practical applications that are exchangeable but not iid. This is not central to the claims of the paper and I'm not necessarily counting it as a weakness; I just found that statement to be distracting. \n* Eq 8 - assuming `Cov` means `Coverage` but would probably be nice to define it formally first\n* L241 - minor typo in this sentence I think.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qIUrfTIAtv", "reviewer": "Reviewer_Ptux", "summary": "The authors design a randomized trial to study the impact of utilizing conformal prediction (CP) sets in machine learning decision pipelines with a human in the loop. They find that utilizing a \"fair\" notion of CP (a notion which ensures coverage for each sensitive group) can actually exacerbate disparate impacts in the pipeline.", "questions": "* Could the authors provide further discussion on the implication of using conformal set size as a proxy for difficulty? I believe this is a reasonable proxy for difficulty but obviously it is also related to the treatment, does this have any effect on the analysis?\n* A clarification question on equation 9. Does trial j refer to an entire task (e.g. text classification) or one sample from an entire task?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The background on conformal prediction provided by the authors is well written, as are the justifications for the models picked at inference time.\n* The authors pre-registered the hypothesis they tested\n* I think the findings are generally novel and interesting. As figure 5 indicates, the fact that utilizing Mondrian CP can reduce accuracy for on certain groups has important implications for fairness practices when prediction sets are required.\n* Generally, this feels like a paper that could spark some new work on fairness in CP, so I believe it will be of interest to the ICLR community.", "weaknesses": "* The authors primary focus is to study disparate impact as measured by improvement *gain* from a given CP treatment (as compared to a control). This is fine, but I would argue that if a CP treatment improves accuracy on each sub-group (even if at a different amount) then this isn't necessarily a bad thing, for example the minimax fairness would be improved. In this sense, I actually found figure 5 the most interesting observation, as it indicates how Mondrian CP can actually leave the worst-case groups, off worse. In general I would suggest more discussion on the trade-offs of the specific fairness definitions the authors use.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lJSEAvpDDX", "reviewer": "Reviewer_27ZU", "summary": "This paper study conformal prediction setup, and want to ensure group fairness in outcome, as measured by disparate impact (final decision made). They study via human experiments, how two different fairness metrics on the prediction sets (fairness on the coverage or of the size) and show that i) the former worsens the actual fairness of the outcome and ii) the latter does improve it.\n\nWhile I was not familiar with the conformal  prediction setup, I found the paper easy to read and it sparked my curiosity. From the research perspective, I believe that this paper is extremel valuabley to ensure that metrics are well grounded in actual outcomes.", "questions": "I am curious about the relationship between the Fairness in Conformal predictions and Fairness in Classifiers. Let's imagine a 3rd experiment, where the base classifier is fair and the conformal prediction setup is applied without any fairness constraint. What would the results look like>?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "As mentioned, I find that such connection between metrics on the set and end-behavior are needed. Paper is quite easy and claims are well supported by evidence.", "weaknesses": "Abstract could be a bit clearer, in particular for people less familiar with the subject.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uckx4jdJY5", "reviewer": "Reviewer_EuTp", "summary": "The paper presents a user study highlighting concerns regarding conformal prediction (CP) in practice. The authors show that $(i)$ CP can exacerbate fairness concerns when humans make decisions; $(ii)$ using equalized sets rather than equalized coverage (as previously proposed) leads to fairer outcomes in practice.", "questions": "I have a couple of questions for the authors:\n\nq1) in the background section, I think the definition of conformal and non-conformal set predictors can coincide, depending on how $s()$, $f()$ and the relative quantiles are defined. Am I correct?\n\nq2) regarding the previously highlighted shortcomings, can the authors elaborate on W1, W2 and W3?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strengths of the paper are:\n\nS1) the empirical evaluation is strong enough, involving different data types and settings;\n\nS2) the proposed user study sheds light on some of the limitations of conformal prediction and provides an interesting counter-example to traditional approaches mitigating unfairness in conformal prediction;\n\nS3) the work is overall well-written and easy to follow.", "weaknesses": "I do not have significant concerns regarding the work. I list here a few comments/suggestions regarding a few potential limitations worth discussing in the current version of the paper:\n\n1) When considering selective prediction (see e.g., [1] for a survey), Bondi et al., (2022) [2] show that human performance is affected by how and which information is passed to them. Their experiments consider different settings and show that human performance increases when humans receive only the \" deferral \" information, and the ML model is good at solving the task. Similar considerations could also be taken into account in this setting.\n\n2) As noted in De Toni et al. (2024), conformal prediction is mainly beneficial when the classification task regards several classes. When the task is binary, most methods fail to provide useful sets, as when both classes are returned, this does not help humans. I wonder whether this consideration might hinder the external validity of the paper's findings, as the current experiments concern only multiclass settings ($m\\geq8$ for all three datasets).\n\n3) I would be more cautious regarding lines 477-479. Ruggieri et al. (2023) [3] show that performing conditional corrections at the group level does not ensure an overall improvement in terms of fairness due to the Yule Effect. If my understanding is correct, generating more singletons per each protected group is equivalent to performing \"conditional corrections\"; hence, the Yule Effect might still occur. This means equalizing the number of singletons across groups might not always be the right solution.\n\n[1] - Hendrickx, Kilian, Lorenzo Perini, Dries Van der Plas, Wannes Meert, and Jesse Davis. \"Machine learning with a reject option: A survey.\" Machine Learning 113, no. 5 (2024): 3073-3110.\n[2] - Bondi, Elizabeth, Raphael Koster, Hannah Sheahan, Martin Chadwick, Yoram Bachrach, Taylan Cemgil, Ulrich Paquet, and Krishnamurthy Dvijotham. \"Role of human-AI interaction in selective prediction.\" In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 5, pp. 5286-5294. 2022.\n[3] - Ruggieri, Salvatore, Jose M. Alvarez, Andrea Pugnana, Laura State and Franco Turini. \"Can we trust fair-AI?.\" In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 13, pp. 15421-15430. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)