Controlling Counterfactual Harm in Decision Support Systems Based on Prediction Sets
Authors: Eleni Straitouri
Suhas Thejaswi
Manuel Gomez Rodriguez
Venue: NeurIPS 2024 poster (2024)
Keywords: Counterfactual Harm
Prediction Sets
Decision Support Systems
Primary area: human-AI_interaction

Abstract: Decision support systems based on prediction sets help humans solve multiclass classification tasks by narrowing down the set of potential label values to a subset of them, namely a prediction set, and asking them to always predict label values from the prediction sets. While this type of systems have been proven to be effective at improving the average accuracy of the predictions made by humans, by restricting human agency, they may cause harm---a human who has succeeded at predicting the ground-truth label of an instance on their own may have failed had they used these systems. In this paper, our goal is to control how frequently a decision support system based on prediction sets may cause harm, by design. To this end, we start by characterizing the above notion of harm using the theoretical framework of structural causal models. Then, we show that, under a natural, albeit unverifiable, monotonicity assumption, we can estimate how frequently a system may cause harm using only predictions made by humans on their own. Further, we also show that, under a weaker monotonicity assumption, which can be verified experimentally, we can bound how frequently a system may cause harm again using only predictions made by humans on their own. Building upon these assumptions, we introduce a computational framework to design decision support systems based on prediction sets that are guaranteed to cause harm less frequently than a user-specified value 
using conformal risk control. We validate our framework using real human predictions from two different human subject studies and show that, in decision support systems based on prediction sets, there is a trade-off between accuracy and counterfactual harm.

Controlling Counterfactual Harm in Decision Support Systems Based on Prediction Sets
Eleni Straitouri Max Planck Institute for Software Systems Kaiserslautern, Germany estraitouriCmpi org
Suhas Thejaswi Max Planck Institute for Software Systems Kaiserslautern, Germany j www.pi-ss..or
Manuel Gomez Rodriguez Max Planck Institute for Software Systems Kaiserslautern, Germany Somueelg@@miisssee org
Abstract
Decision support systems based on prediction sets help humans solve multiclass classification tasks by narrowing down the set of potential label values to a subset of them, namely a prediction set, and asking them always predict label values from the prediction sets. While this type of systems have been proven to be effec- tive at improving the average accuracy of the predictions made by humans, by restricting human agency, they may cause harm-a human who has succeeded predicting the ground-truth label of instance on their own may have failed had they used these systems. In this paper, our goal is to control how frequently a decision support system based on prediction sets may cause harm, by design. To this end, we start by characterizing the above notion of harm using the theoretical framework of structural causal models. Then, we show that, under a natural, albeit unverifiable, monotonicity assumption, we can estimate how frequently system may cause harm using only predictions made by humans on their own. Further, we also show that, under weaker monotonicity assumption, which can be verified experimentally, we can bound how frequently system may cause harm again using only predictions made by humans their own. Building upon these assumptions, we introduce a computational framework to design decision support systems based prediction sets that are guaranteed to cause harm less frequently than user- specified value using conformal risk control. We validate our framework using real human predictions from two different human subject studies and show that, in decision support systems based on prediction sets, there is a trade-off between accuracy and counterfactual harm.
Introduction
The principle of "first, do no harm" holds profound significance in variety of professions across multiple high-stakes domains. For example, in the field of medicine, doctors swear an oath to prioritize their patient's 's well-being or, in the legal justice system, preserving the innocence of individuals is paramount. As a result, in all these domains, rules and guidelines have been established to prevent decision makers-doctors doctors or judges- from making decisions that harm suspects. In recent years, it has been increasingly argued that similar principle should apply to decision support systems using machine learning algorithms in high-stakes domains [1-4].'
or
The definition of harm is not unequivocally agreed upon, however, the most widely accepted definition is the counterfactual comparative account of harm (in short. counterfactual harm) [5-7], which we adopt in our work. Under this definition, an action causes harm an individual they would have
'The European Unions' Al act mentions the "harm" more than 35 times and points out that, its crucial role in the design algorithmic systems must be defined carefully.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

been in a worse state had the action been taken. Building upon this definition, we say that a decision support system causes harm to an individual if a decision maker would have made a worse decision about the individual had they used the system.
In machine learning for decision support, one of the main focus has been classification tasks. Here, the most studied setting assumes the decision support system uses a classifier to predict the value of (roound label of interest and a human expert uses the predicted value to update their own prediction [8-12]. Unfortunately, in this setting, it is yet unclear how to guarantee that the (average) accuracy of the predictions made by an expert who uses the system higher than the accuracy of the predictions made by the expert and the classifier on their own, what is often referred to as human-Al complementarity [1]. In this context, a recent line of work [17, 18] have argued, both theoretically and empirically, that an alternative setting may enable human AI complementarity. In this alternative setting, the decision support system helps a human expert by providing a set of label predictions, namely a prediction set, and asking them to always predict label value from the set. The key principle that, by restricting human agency, good performance does not depend on the expert developing good sense of when to predict label from the prediction set. In this context, is also worth noting that Google has recently developed a tool that uses patient history and skin condition images to provide decision support using prediction sets [19], and study by Jain et al. [20] has found that physicians and nurses using this tool improved diagnoses for 1 in every 8 to 10 cases.
In this work, we argue that the same principle that enables human- AI complementarity on decision support systems based on prediction sets may also cause counterfactual harm- a human expert who has succeeded at predicting the label of an instance on their own may have failed had they used these systems. Consequently, our goal is to design decision support systems based on prediction sets that are guaranteed to cause, in average, less counterfactual harm than a user-specified value.
Our contributions. We start by formally characterizing the predictions made by decision maker using a decision support system based on prediction sets using a structural causal model (SCM) and, based on this characterization, formalize our notion of counterfactual harm. In general, since counterfactual harm lies within level three in the "ladder of causation" [21]. it is not (partially) identifiable-it cannot be estimated (bounded) from data. However, we show that, under a natural counterfactual monotonicity assumption on the predictions made by decision makers using decision support systems based on prediction sets, counterfactual harm is identifiable. Further, we show that, under weaker interventional monotonicity assumption, which can be verified experimentally, the average counterfactual harm is partially identifiable. Then, building upon these assumptions, we develop computational framework to design decision support systems based on prediction sets that are guaranteed to cause, average, less counterfactual harm than a user-specified value using conformal risk control [22]. Finally, we validate our framework using real human predictions from two different human subject studies and show that, in decision support systems based on prediction sets, there is a trade-off between accuracy and counterfactual harm.
Further related work. Our work builds upon further related work on set-valued predictors, critiques of prediction optimization, counterfactual harm and algorithmic triage.
Set-valued predictors output set label values, namely a prediction set, rather than single labels [23]. However, set-valued predictors have not been typically designed nor evaluated by their ability to help human experts make more accurate predictions [24-29]. Only very recently, emerging line of work has shown that conformal predictors, a specific type of set-valued predictors, may help human experts make more accurate predictions [17, 18, 30-33]. Within this line of work, the work most closely related to ours is by Straitouri et al. [17, 18), which has introduced the setting, and counterfactual and interventional monotonicity properties we build upon.
Prediction optimization has been recently put into question the context of decision support [34-36]. More specifically, it has been argued that optimizing decision support systems to improve prediction accuracy does not always translate to better becision-making Our work aligns with this critique since we argue that improving prediction accuracy may come at the cost of counterfactual harm.
The literature on counterfactual harm in machine learning is still quite small and has focused on traditional machine learning settings in which machine learning models replace human decision makers and make automated decisions [1-4]. Within this literature, the work most closely related to ours is by Richens et al. [1], which also uses a structural causal model to define counterfactual harm. However, their definition of counterfactual harm differs in a subtle, but important, way from ours. In
2

---

our setting, their definition of counterfactual harm would compare the accuracy of a factual prediction made by an expert using the decision support system against the counterfactual prediction that the same expert would have made on their own. That means, under their definition, one would need to deploy the system to estimate the harm may cause, potentially causing harm. contrast, under our definition, one does not need to deploy the system to estimate (or bound) the harm it may cause, as it will become clear in Section 4, and thus we argue that our definition may be more practical.
Learning under algorithmic triage seeks to develop classifiers that make predictions for given fraction of the samples and leave the remaining ones to human experts, as instructed by a triage policy [37-42]. In contrast, in our work, for each sample, a classifier is used to construct a prediction set and human expert needs to predict label value from the set. In this context, is also worth noting that learning under algorithmic triage has been extended to reinforcement learning settings [43-46].
2 Decision support systems based on prediction sets
We consider a multiclass classification task which, for each task instance, a human expert has to predict the value of ground tround-ttuthh label y E y {I,..... ,.....,,,,, L}. support system C: X + 2" that, given a set of features I E X. helps the expert by narrowing down the label values to subset of them C(x) S V. namely prediction set. Here, we focus on setting in which the system asks the expert to always predict a label value ý from the prediction set C(r)). Note that, by restricting the expert's 's agency, good performance does not depend on the human expert developing a good sense of when to predict a label from the prediction set. Moreover, we assume that the set of features, the ground-trun -truth label and the expert's ''s prediction are sampled from an unknown fixed distribution", Le., V, y ~ P(X, Y) and ÿ ~ P(L X, CYY))())
Then, our goal is to design decision
Further, similarly as in Straitouri et al. [17, 18], we consider that, given a set of features x E X, the system constructs the prediction set C(x) using the following set-valued predictor [23]. First, the set-valued predictor ranks each potential label value y E y using the softmax output of a pre-trained classifier my(xx) [0, 1]. Then, given user-specified threshold À [0, 1]. it uses the resulting ranking to construct the prediction set C(z) = Cx(z) as follows:
Cx(x)= = (Mo)) with k = 1+ Σ 1 (z) (1)
where '(1) denotes the i-th label value in the ranking. Here, note that, for X 0, the prediction set contains just the top ranked label value, for x 1, it contains all label values.3
Given the above parameterization, one may just focus on finding the optimal threshold X under which the human expert achieves the highest average accuracy as in Straitouri al. [17, 18), i.e.
X -argmx A(A), with A(A) E Fx,x P [ = =]] AE(O.1]
However, this focus does not prevent the resulting system Cx from causing harm-an an expert may succeed to predict the value of the ground-truth label on their own on instances in which they would have failed had they used Cx. In this work, our goal is to design computational framework that, given a user-specified bound a E [o, 1), finds the set of À values which are all guaranteed to cause less harm, in average, than the bound
Remark. We would like clarify that, if one sets the value of the threshold À to be roughly the 1 quantile of the empirical distribution of the scores 1 - 1-m, (2) in a calibration set, then, the set valued predictor defined by Eq. is equivalent a vanilla conformal predictor with nonconformity scores 1 - --my(x) and coverage 1 - a. Under this view, it becomes apparent that, by searching for À values in [o, 1] that are harm controlling, we are essentially searching for vanilla conformal predictors that are harm controlling. In this context, we would like to further clarify that our framework is agnostic to the choice of nonconformity score or more generally, the set-valued predictor, used to construct the prediction sets [47-49]. Motivated by this observation, in Appendix E, we include additional experiments where we evaluate our framework using a more complex set-valued predictor [49].
We denote random variables with capital letters and realizations random variables with lowercase letters. The assumption that my (x) E [0, 1] and X [0, 1] is without loss of generality.
3

---

Edema
Fibrosis
Edema
Edema
Effuusion
Ground Truth Label
Set of Features
Prediction Set
Expert Prediction
Y
(X))
Y
Exogenous Variables
V
A
U
Figure 1: Our structural causal model M. Circles represent endogenous random variables and boxes represent exogenous random variables. The value each endogenous variable is given by a function of the values of its ancestors, as defined by Eq. 3. The value of each exogenous variable is sampled independently from a given distribution.
3 Counterfactual harm of decision support systems
To formalize our notion of harm, we characterize how human experts make predictions using decision support system C via structural causal model (SCM) [21], which we denote M. More specifically, similarly as Straitouri et al. [18]. we define M by the following set of assignments:
X = (VV,
Sv(v,,
=
Ý (= (3)
where A, U and are exogenous random variables and fx. fy, fc and fo are given functions.* The exogenous variables A, U and V characterize the user-specified threshold, the expert's individual characteristics and the data generating process, respectively. The function fc is directly defined by Eq. Le. fc(A, X) CA(X) Further, as argued elsewhere [21], we can always find a distribution for the exogenous variables A, and as well as a functional form for the functions fx, fy and fy such that the distributions of the features, the ground-truth label and the expert's ss prediction introduced in Section 2 are given by the observational distribution entailed by the SCM M. For ease of exposition,
assume that, under no interventions, the distribution of the exogenous variable A is P(A) = ( = }} and thus human experts make predictions on their own. Figure shows a visual representation of our SCM M.
Building upon the above characterization, are now ready to formalize the following notion of counterfactual harm, which essentially compares the accuracy of factual prediction made by an expert on their own against the counterfactual prediction that the same expert would have made had they used a decision support system Cx:
Definition (Counterfactual Harm)
PM, the counterfactual harm that a deci- sion support system C2 would have caused, if deployed, is given by
For any I. z.
1xx (2, y. 9) (.w.aaaa 8-0.x-a.Y-u(ý) [max (0, 1 1 = y} - 118 =
where do(A X) denotes (hard) intervention on the exogenous variable A.
Here, note that counterfactual harm can only be nonzero the expert has made successful prediction on their own, i.e., I = Otherwise, the expert's prediction ý could not have become worse had they used the decision support system Cx.
"The functions fx. fr. fc and fy are causal mechanisms not equations that can manipulated [50]. 'To denote interventions in counterfactual distribution, we follow the notation by Peters et al. [51]. Refer to Appendix for comparison Pearl's notation [50].

---

Given the above definition of counterfactual harm and user-specified bound a E [0. 1], our goal is find the largest harm-conee com--oontolling set of values A(a) S [o. such that, for each X A(a:, it holds that the counterfactual harm is, in expectation across all possible instances, smaller than o, i.e.
A(a) = {> E [0, H(A) = E XxXx=> (X, Y, 80 S sa
(5)
At this point, we cannot expect to find the set A A(a) because counterfactual harm lies within level three in the "ladder of causation" [21] and thus it is not identifiable from observational data without further assumptions. However, in what follows, we will show that, under certain assumptions, the average counterfactual harm (A) (partially) identifiable, i.e., it can be estimated (bounded) using observational data.
Comparison to Richens's definition of counterfactual harm. Richens et al. [1] define counterfactual harm as follows:
V.
(m 1 y) - 1 y}H
where x, ys P.M:do(A=A) and À is considered to be the default action in the language of Richens et al. [1]. This definition implicitly assumes that the system C2 is deployed and compares the factual prediction made by expert using the deployed system against the counterfactual prediction Y had the expert made on their own. On the contrary, our definition does not assume that the system Cx is deployed and instead it compares the factual prediction ý made by an expert on their own against the counterfactual prediction Y had the expert made using the system Cx.
4 Counterfactual harm under counterfactual and interventional monotonicity
In this section, we analyze counterfactual harm h> (z. y, 9). as defined in Eq. 4, from the perspective of causal identifiability under two natural monotonicity and interventional monotonicity. Both of these assumptions, which were first studied by Straitouri et al. [18], formalize the intuition that increasing the number of label values in prediction set increases its difficulty.
couumtioons-cuuttfffacci
monotonicity
Under counterfactual monotonicity, for any x E X' and X, X E [0, 1] such that Y E Cx(x) S (x), if expert has succeeded at predicting Y using Cx. they would have also succeeded had they used Cx and, conversely, if they have failed at predicting Y using Cx. they would have also failed had they used Cx, while holding "everything else fixed". More formally, counterfactual monotonicity is defined as follows:
Assumption (Counterfactual Monotonicity) Counterfactual monotonicity holds if and only if. for any x E X and any X, E [0, 1] such that Y E (x)) S (x))). we have that
1ff u, Cx()) = Y) 2 ((fx(M,,, =Y
(6)
for any u ~ PM) and u ~ PM(V X = x).
Under interventional monotonicity, for any X' and X, X' [0, 1] such that Y Cx(x) C (x) the probability that experts succeed at predicting Y using C^ is equal or greater than using Cy. More formally, interventional monotonicity is defined as follows::
Assumption (Interventional Monotonicity) Interventional monotonicity holds if and only if. for any E X. y E V and X, X [0, such that y E Cx(x)) S (x), we have that
PM: X) (ÿ Y x = :M:: = = x =
(7)
where the probability is over the exogenous random variables U and V characterizing the expert's 'ss individual characteristics and the data generating process, respectively.
In what follows, first show that, under the counterfactual monotonicity assumption, counterfactual harm is identifiable (we provide all proofs in Appendix B):
'In Straitouri et al. [18], interventional monotonicity is originally defined unconditionally of the value of ground ttttt label Y.
5

---

Proposition 1 Under the counterfactual monotonicity assumption, for any v. D ~ PM, the counterfactual harm that a decision support system Cx would have caused, if deployed, given by
v. = LÚ - A y & Cx(z)).
(8)
As an immediate consequence, can conclude that, under counterfactual monotonicity, the average counterfactual harm (A), as defined in Eq. 5, is identifiable and can be estimated using observational data sampled from P.M. However, since the inequality condition of the counterfactual monotonicity assumption compares counterfactual predictions and thus cannot be experimentally verified, one should be cautious about using the above proposition estimate counterfactual harm in high-stakes applications.
Next, show that, under the interventional monotonicity assumption, the average counterfactual harm is partially identifiable:
Proposition 2 Under the interventional monotonicity assumption, the average counterfactual harm (A) that a decision support system Cx would have caused, if deployed, satisfies that
E Ex PXXxX-MXXXMX = =Y $ CX)XH & H())
S SXXXY-PMXXXY -Y= (x(X)) + +VVVE.II))
(9)
Importantly, note that, in the above proposition, the lower bound on the left hand side of Eq. matches the average counterfactual harm under the counterfactual monotonicity assumption and thus, holding "everything else fixed", the average counterfactual harm under interventional monotonicity is always greater or equal than the average counterfactual harm under counterfactual monotonicity. Moreover, further note that the inequality condition of the interventional monotonicity assumption compares interventional probabilities and thus we can experimentally verify it (see Appendix F), lending support to using the above proposition to bound average counterfactual harm in high-stakes applications.
Controlling counterfactual harm using conformal risk control
In this section, we develop computational framework that, given a decision support system Cx and user-specified bound a, aims to find the largest harm-controlling set A(x), as defined Eq. 5. In the development of our framework, we will first assume that counterfactual monotonicity holds and, later on, we will relax this assumption, and assume instead that interventional monotonicity holds.
Our framework builds upon the the idea of conformal risk control, which has been introduced very recently by Angelopoulos et [22]. Given any monotone loss function ((Cx Y) with respect to À and a calibration set {(X,, ) with (X,, Y2) ~ P(X, Y). conformal risk control finds a value of X under which the expected loss of test sample (Xm+1. Yn+1)
P(X, Y) does not exceed a user-specified bound i.e., B[[(Cx (X,-11 Yn+1)] I a. However, in our framework, we re-define the loss l so that it does not only depend on the prediction set and the label value but also on the expert's s prediction on their own.
Under the counterfactual monotonicity assumption, we the value of the loss using the expression of counterfactual harm in Eq. 8 and, using a similar proof technique as in Angelopoulous et al., first prove the following theorem:
Theorem Ler {(X,, Y,, 1= be a calibration set, with (,,,Y, (,),,,, E 10 be user-specified bound, and
((a) inf {x= x n n+I
1 n n+1 S
of where HA( Ee 11; = Y,XY & Cx(X.))
H (A)
(10)
If counterfactual monotonicity holds, a test sample (X Yn+ Yn-1) ~ PM (M,(X,Y satisfies that
E 110Y=1= A Yat1 & CMas (Xn- Sa,
where the expectation is over the randomness in the calibration set used to compute the threshold {(x)) and the test sample.

---

Then, we leverage the above theorem and the fact that, under counterfactual monotonicity, the counterfactual harm nonincreasing with respect X, shown in Lemma 3 in Appendix B.5, to recover the largest harm-controlingg set Maa):
Corollary Let a E [0, 1] be user-specified bound. Then, under the counterfactual monotonicity assumption, holds that A(a) = (A E [0, 1||> Ma)), where X(x)) given by Eq. 10.
Under the interventional monotonicity assumption, rather than directly controlling the counterfactual harm, we will control the upper bound given by Eq. 9. Consequently, rather than recovering the largest harm-controlling set A(a), we will recover a harm-controlling set A'(a) C A(a). However, since the expression inside the expectation of the upper bound nonmonotone with respect to we cannot directly use it to set the value of the loss l in conformal risk control. That said, since the first term of the expression nonincreasing and the second term is nondecreasing, as shown in Lemmas and 4 in Appendix B.5, we can apply conformal risk control separately for each term. For the first term, we use Theorem because the term matches the counterfactual harm under the counterfactual monotonicity assumption. For the second term, we prove the following theorem:
Theorem Let D {(x,, Yú, Yo) be user-specified bound, and
calibration set, with (Xa, Yi, Y;) ~ pM, a
X(a) = sup { {: n+1 Gm(A) n. n+
sa) where Gn(A) E 1( YYAY
(11)
If interventional monotonicity holds and À exists, a test sample (Xn+1, 1 Pnt1) PM((,Y, satisfies that
1(82+1 Yn41 A Ya41 CMay (X)= Sa.
where the expectation is over the randomness the calibration set used to compute the threshold X(a)) and the test sample (Xu+1, Ya+1, Yn+1).
Finally, we leverage the above theorems and the fact that the first term inside the expectation of the upper bound of counterfactual harm in Eq. 9 nonincreasing and the second term is nondecreasing with respect to À to recover a harm-controlling set A (a) C A(a):
Corollary 2 Let E [0, 1] be user-specified bound. Then, under the interventional monotonicity assumption, for any choice of a' I a, the set
A((a) (A E [0. (0.1111(c)) ((a)) SX ((a-
where Maa) is given by Eq. 10 and i(a a') is given by Eq. 11 satisfies that A'(a) C A(a).
Note that, in practice, the above corollaries can be implemented efficiently since, by definition, the functions ft (A) and Gn (A) are piecewise constant functions of with n different pieces.
Experiments
In this section, we use data from two different human subject studies to: a) evaluate the average counterfactual harm caused by decision support systems based on prediction sets; b) validate the the- oretical guarantees offered by our computational framework (i.e., Corollaries I and 2); c) investigate the trade-off between the average counterfactual harm caused by decision support systems based on prediction sets and the average accuracy achieved by human experts using these systems. In what follows, we assume that counterfactual monotonicity holds. In Appendix G, we conduct experiments where we relax this assumption and assume instead that interventional monotonicity holds.?
Experimental setup. We first experiment with the ImageNet16H dataset by Steyvers al. [52], which comprises 32,431 predictions made by 145 human participants on their own about noisy
All experiments ran on Mac OS machine with an MI processor and 16GB Memory. An open-source implementation of our methodology is publicly available at http://github.com/Neewwott hwwp://aooooooooooooooooooooooon Leaming/conttollingg counterfactual harm- seam---rdiction-sett
7

---

0.01
0.05
1.
0.9
VGG19
VGG19
0.8
0.8
0.8
0.9
0.9
De166
DenseNe1161
Averge Aocuracy
Average Accuracy
0.8
0.8
0.6
0.85
0.85
GoogleNet
GoogleNet
0.75
0.75
0.4
0.85
0.85
ResNot152
ResNet152
0.75
0.75
0.2
0.8
AlexNet
0.8
AlexNet
0.7
0.0
0.00
0.05
0.10
0.15
0.20
0.00
0.35
0.10
0.15
0.20
Average Counterfact ual Harm
Average Counterfactual Harm
0.01
(b) 0.05
Figure 2: Average accuracy estimated by the mixture of MNLs against the average counterfactual with step 0.001
harm for images with w 110. Each point corresponds to a À value from and the coloring indicates the relative frequency with which each À value is in A(a) across random samplings of the calibration set. Each row corresponds to decision support systems CA with a different pre-trained classifier with average accuracies 0.846 (VGG19), 0.830 (DenseNet), 0.722 (GoogleNet), 0.727 (ResNet152),, and 0.691 (AlexNet). The average accuracy achieved by the simulated human experts on their own is 0.771. The results are averaged across 50 random samplings of the test and calibration set. In both panels, 95% confidence intervals are represented using shaded areas and always have width below 0.02.
images created using 1,200 unique natural images from the ImageNet Large Scale Visual Recognition Challenge (ILSRVR) 2012 dataset [53]. More specifically, each of the natural images was used to generate four noisy images with different amount of phase noise distortion w E {80, 95, 110, 125) and with the same ground-truth label from a label set )' of size L = 16. Here, the amount of phase noise controls the difficulty of the classification task-the higher the noise, the more difficult the classification task. In our experiments, we use (all) the noisy images with noise value {80, 95, 110} because, for the noisy images with 125, humans perform poorly; moreover. we stratify these images (and human predictions) with respect to their amount of phase noise.
For each stratum of images, apply our framework decision support support systems C2 with different pre-trained classifiers, namely VGG19 [54], DenseNet161 [55], GoogleNet [56], ResNet152 [57] and AlexNet [58] after 10 epochs of fine-tuning, provided by Steyvers al. [52]. 8 To this end, we randomly split the images (and human predictions) into a calibration set (10%). which we use to find the harm-controoling sets A(a) by applying Corollary 1, and test set (90%). which we use to estimate the average counterfactual harm (A) caused by the decision support systems Cx as well as the average accuracy A(A) of the predictions made by a human expert using Cx. Here, since the dataset only contains predictions made by human participants on their own, we use the mixture of multinomial logit models (MNLs) introduced by Straitouri et al. [17] to estimate the average accuracy A()). Refer to Appendix C for more details regarding the mixture of MNLS and to Appendix D for additional experiments studying the relationship between average counterfactual harm (A), average prediction set size and empirical coverage".
Then, experiment with the ImageNet16H--PS 10taser by Straitouri et al. [18], which comprises 194,407 predictions made by 2,751 human participants using decision support systems CA about the
$ All classifiers and images are publicly available at https//oosfoo/nurf "By empirical coverage, we refer the fraction the test samples for which the prediction sets include the ground truth label value. TThe dataset publicly available

---

Real Predicted
0.01
0.05
1.0
0.90
0.90
0.8
Averaee Accuraey
Average Arcumey
0.6
0.85
0.85
0.4
0.80
0.2
0.0
0.00
0.02
0.04
0.06
0.08
0.00
0.02
0.04
0,06
0.08
Average Connerfactual Harm
Average Counterfactull Harm
(a) 0.01
(b) 0.05
Figure 3: Average accuracy estimated using predictions by human participants (Real) and using the mixture of MNLs (Predicted) against the average counterfactual harm for images with 110. Each point corresponds to a À value from 0 to with step 0.001 and the coloring indicates the relative frequency with which the À value is in A(a) across random samplings of the calibration set. The decision support systems Cx use the pre-trained classifier VGG19. The results are averaged across 50 random samplings of the test and calibration set. In both panels, 95% confidence intervals are represented using shaded areas and always have width below 0.02.
set of noisy images with 110 described above. More specifically, for each noisy image, the dataset contains human predictions made under any possible prediction set that can be constructed using Eq. I with (the softmax output of) VGG19 after 10 epochs of fine-tuning, as provided by Steyvers et al. [52]. Here, similarly as in the ImageNet 16H dataset, we randomly split the images (and human predictions) into calibration set (10%), which
use to find A(a), and test (90%), which we use to estimate (A) and (A). However, in this case, we can estimate A()) using the predictions made by human participants using C2 from the dataset, and we can compare this empirical estimate to the one using the mixture of MNLs.
In both datasets, calculate confidence intervals and validate the theoretical guarantees offered by Corollary by repeating each experiment 50 times and, each time, sampling different calibration and test sets.
Results. Figure 2 shows the average accuracy A()) achieved by human participant using Cx. as predicted by the mixture of MNLs, against the average counterfactual harm H (A) caused by Cx for E {0.01, 0.05) on the stratum of images with w 110 from the ImageNet 6H dataset. Refer to Appendix D for results on other strata. Here, each point corresponds to different À value and its coloring indicates the empirical probability that included in the harm-controlling set A(ar). The results show several interesting insights. First, we find that, as long À < 1, the decision support systems CΛ always cause some amount of counterfactual harm This suggests that, while restricting human agency enables human-Al complementarity. it inevitably causes (some amount of) counterfactual harm. Second, we find that the sets values provided by our framework are typically harm-coe corm-controlling. i.e., they do not include À values such that (A) > However, the sets are often conservative and do not include all the harm-controllling X values due to estimation error in the the empirical estimate (A) of the average counterfactual harm using data from the calibration set. In Appendix D. we show that using larger calibration sets reduces the above mentioned estimation error and results in sets between accuracy and counterfactual harm and this trade-off off is qualitatively consistent across decision support systems using different pre-trained classifiers, In fact, for a = 0.01, the decision support systems C.- offering the greatest average accuracy A(X*) are honm-controlling.
values that are less conservative. Third, we find that there is a trade-off
Figure shows the average accuracy A(A) achieved by human participant using Cx. as estimated using predictions made by human participants using Cx. against the average counterfactual harm
For À = 1. the decision support system Cs does not cause harm because the prediction sets always contain all label values and thus human participants always make predictions their

---

(A) caused by Cλ for a E {0.01, 0.05) on the ImageNet16H-PS. dataset. Here, the meaning and coloring each point the same as Figure 2. The results also support the findings derived from the experiments with the ImageNetl 6H dataset-the the decision support systems CĄ always cause some amount of harm, our framework succeeds at identifying harm-controlling sets, and there a trade-off between accuracy and counterfactual harm. Further, they also show that, while there is a gap between the average accuracy estimated using the mixture of MNLs and the average accuracy estimated using predictions made by human participants using Cx, they follow the same trend and support the same qualitative conclusions.
Discussion and limitations
In this section, we highlight several limitations of our work, discuss its broader impact, and propose avenues for future work.
Assumptions. We have assumed that the data samples and the expert predictions are drawn i.i.d. from fixed distribution and the calibration contains samples with noiseless ground truth labels and expert label predictions. It would be very interesting to extend our framework to allow for distribution shifts and label noise. Moreover, we have considered prediction sets constructed with a fixed user-specified threshold value A. In light of recent work by Gibbs et al. [59]. it would be interesting extend our framework allow for threshold values that depend on the data samples. Furthermore, in our definition of counterfactual harm we have treated all inaccurate predictions as equally harmful. Expanding our definition of harm weigh different inaccurate predictions according to their consequences would be a very interesting avenue for future work. In addition. under the interventional monotonicity assumption, we have empirically observed that the gap between our lower- and upper-boune bounds is often large. Therefore, it would be useful to identify other natural assumptions under which this gap is smaller.
Methodology. In our framework, the prediction sets are constructed using the softmax output of pre- trained classifier. However, hypothesize that, by accounting for the similarity between the mistakes made by humans and those made by the pre-trained classifier, we may be able to construct prediction sets that cause counterfactual harm less frequently. Moreover, we have focused controlling the average counterfactual harm. However, whenever the expert's predictions are consequential to individuals, this may lead significant disparities across demographic groups. Therefore, it would be important to extend our framework to account for fairness considerations.
Evaluation. Our experimental evaluation comprises single benchmark dataset of noisy natural images and thus one may question the generalizability of the conclusions we draw from our results. To overcome this limitation, it would be important to further investigate the trade-off between accuracy and counterfactual harm in decision support systems based on prediction sets in real-world application domains (e.g., medical diagnosis).
8 Conclusions
In this paper, we have initiated the study of counterfactual harm decision support systems based on prediction sets. We have introduced a computational framework that, under natural monotonicity assumptions on the predictions made by experts using these systems, can control how frequently these systems cause harm. Moreover, have validated our framework using data from two different human subject studies and shown that, in decision support systems based on prediction sets, there is a trade-off between accuracy and counterfactual harm.
Acknowledgments and Disclosure of Funding
Rodrigue acknowledges support from the Furopean Research Council (ERC) under the European Union's 's Horizon 2020 research and innovation programme (grant agreement No. 945719).
10

---

References
[1] Jonathan Richens, Rory Beard, and Daniel H Thompson. Counterfactual harm. In Advances in Neural Information Processing Systems, volume 35, pages 36350-36365, 2022.
[2] Sander Beckers, Hana Chockler, and Joseph Halpern A causal analysis of harm. In Advances Neural Information Processing Systems, volume 35, pages
2022.
[3] Haoxuan Li, Chunyuan Zheng, Yixiao Cao, Zhi Geng, Yue Liu, and Peng Wu. Trustworthy policy learning under the counterfactual no-harm criterion. International Conference on Machine Learning, pages 20575-20598. PMLR, 2023.
[4] Sander Beckers, Hana Chockler, and Joseph Y Halpern. Quantifying harm. In Proceedings of the International Joint Conference Artificial Intelligence. IJCAL, 2023.
[5] Joel Feinberg. Wrongful life and the counterfactual element harming. Social Philosophy and Policy, (1):114-178, 1986.
[6] Matthew Hanser. The metaphysics of harm. Philosophy and Phenomenological Research, 77(2):421-450, 2008.
[7] Justin Klocksiem. A defense of the counterfactual comparative account of harm. American Philosophical Quarterly, 49(4):285-300 2012.
[8] Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S. Lasecki, Daniel S. Weld, and Eric Horvitz. Beyond accuracy: The role of mental models human-ai team performance. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. AAAI press, 2019.
[9] Brian Lubars and Chenhao Tan. Ask not what can do, but what should do: Towards framework of task delegability. In Advances in Neural Information Processing Systems, volume 32, pages 57-67, 2019.
[10] Sebastian Bordt and Ulrike von Luxburg. When humans and machines make joint decisions: A non-symmetric bandit model. arXiv preprint arXiv:2007.04800, 2020.
[11] Kailas Vodrahalli, Tobias Gerstenberg, and James Y Zou. Uncalibrated models improve human-ai collaboration. In Advances in Neural Information Processing Systems, volume 35, pages 4004-4016. 2022.
[12] Nina Corvelo Benz and Manuel Rodriguez. Human- calibration for ai ai-assisted decision making. In Advances in Neural Information Processing Systems, volume 36, 2024.
[13] Ming Yin, Jennifer Wortman Vaughan, and Hanna Wallach. Understanding the effect of accuracy on trust in machine learning models. In Proceedings of the CHI conference on human factors in computing systems, pages 1-12. ACM, 2019.
[14] Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 295-305. ACM, 2020.
[15] Harini Suresh, Natalie Lao, and Ilaria Liccardi. Misplaced trust: Measuring the interference of machine learning in human decision Docision-making. In Proceedings of the ACM Conference on Web Science, pages 315-324. ACM, 2020.
[16] Vivian Lai, Chacha Chen, Alison Smith-Remnn Q Vera Liao, and Chenhao Tan. Towards a science of human-ai decision making: An overview of design space in empirical human- subject studies. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 1369-1385. 2023.
[17] Eleni Straitouri, Lequn Wang, Nastaran Okati, and Manuel Gomez Rodriguez. Improving expert predictions with conformal prediction. Proceedings of the International Conference on Machine Learning, pages 32633-32652 PMLR, 2023.

---

[18] Eleni Straitouri and Manuel Gooee-Rooriiuue Designing decision support systems using counterfactual prediction sets. In Proceedings of the International Conference on Machine Learning. PMLR, 2024.
[19] Google Blogpost. Improving access to skin disease information. https://health..gom google/ healthh-researcchhmaagii and -ag---aaddddagiissss... 2020.
[20] Ayush Jain, David Way, Vishakha Gupta, Yi Gao, Guilherme de Oliveira Marinho, Jay Hartford, Rory Sayres, Kimberly Kanada, Clara Eng, Kunal Nagpal, et al. Development and assessment of an artificial intelligence-bee tool for skin condition diagnosis by primary care physicians and nurse practitioners in teledermatology practices. JAMA network open, 444)::211244- 2021.
[21] Judea Pearl. Causal inference in statistics: An overview. Statistics Surveys, 3:96-146, 01 2009.
[22] Anastasios Nikolas Angelopoulos. Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schus- ter. Conformal risk control. In Proceedings of the International Conference Learning Representations. ICLR, 2024.
[23] Evgenii Chzhen, Christophe Denis, Mohamed Hebiri, and Titouan Loricul. Set-valued classifictionn-vervice via a unified framework. arXiv preprint arXiv:2102. 2021.
[24] Gen Yang, Sébastien Destercke, and Márie-Hélène Masson. Cautious classification with nested dichotomies and imprecise probabilities. Seft Computing, 21(24):447-7462, 2017.
[25] Thomas Mortier, Marek Wydmuch, Krzysztof Dembczyíikii, Eyke Hüllermeier, and Willem Waegeman. Efficient set-valued prediction in multi-class classification. Data Mining and Knowledge Discovery, 35(4):1435-1465 2021.
[26] Liyao Ma and Thierry Denoeux. Partial classification in the belief function framework. Knowledge-Based Systems, 214:106742, 2021.
[27] Vu-Linh Nguyen and Eyke Hüllermeier. Multilabel classification with partial abstention: Bayes-optimal prediction under label independence. Journal of Artificial Intelligence Research, 72:613-665, 2021.
(28) Anastasios N Angelopoulos, Stephen Bates, Emmanuel J Candès, Michael I Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. arXiv preprint arXiv:2110.01055 2021.
[29] Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael Jordan. Distribution-free, risk-controlling prediction sets. Journal of the ACM, 68(6):1-34, 2021.
[30] Varun Babbar, Umang Bhatt, and Adrian Weller. On the utility of prediction sets in human-ai teams. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 2457-2463. IJCAL, 7 2022.
[31] Jesse C. Cresswell, Yi Sui, Bhargava Kumar, and Noël Vouitsis. Conformal prediction sets improve human decision making. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27. 2024. OpenReview.net, 2024.
[32] Dongping Zhang, Angelos Chatzimparmpas, Negar Kamali, and Jessica Hullman. Evaluating the utility of conformal prediction sets for ai-advised image labeling. In Proceedings of the CHI conference on Human Factors in Computing Systems. ACM, 2024.
[33] Giovanni De Toni, Nastaran Okati, Suhas Thejaswi, Eleni Straitouri, and Manuel Gomez- Rodriguez. Towards human-ai complementarity with predictions sets. In Avances on Neural Information Processing Systems, 2024.
[34] Wouter AC van Amsterdam, Nan van Geloven, Jesse H Krijthe, Rajesh Ranganath, and Giovanni Ciná. When accurate prediction models yield harmful self-fulfiling prophecies. arXiv preprint arXiv:2312.01210. 2023.
12

---

[35] Angelina Wang, Sayash Kapoor, Solon Barocas, and Arvind Narayanan. Against predictive optimization: On the legitimacy of reciisin-mmaiing algorithms that optimize predictive accuracy. ACM Journal of Responsible Computing, 1(1), March 2024.
[36] Lydia Liu, Solon Barocas, Jon Kleinberg, and Karen Levy. On the actionability of outcome prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 22240-22249, 2024.
[37] Maithra Raghu, Katy Blumer, Greg Corrado, Jon Kleinberg, Ziad Obermeyer, and Sendhil Mullainathan. The algorithmic automation problem: Prediction, triage, and human effort. arXiv preprint arXiv: 1903. 12220, 2019.
[38] Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning, pages 7076-7087. PMLR, 2020.
[39] Abir De, Nastaran Okati, Ali Zarezade, and Manuel Gomez Rodriguez. Classification under human assistance. Proceedings of the AAAI Conference Artificial Intelligence, 35(7):5905- 5913, 2021.
[40] Nastaran Okati, Abir De, and Manuel Rodriguez. Differentiable learning under triage. In Advances in Neural Information Processing Systems, volume 34, pages 9140-9151, 2021.
[41] Mohammad-Amin Charusaie, Hussein Mozannar, David Sontag, and Samira Samadi. Sample efficient learning of predictors that complement humans. In International Conference on Machine Learning, pages 2972-3005. PMLR, 2022.
[42] Hussein Mozannar, Hunter Lang. Dennis Wei, Prasanna Sattigeri, Subhro Das, and David Sontag. Who should predict? exact algorithms for learning to defer humans. In International conference on artificial intelligence and statistics, pages 10520-10545. PMLR, 2023.
[43] Eleni Straitouri, Adish Singla, Vahid Balazadeh Meresht, and Manuel Gomez- Rodriguez. Reinforcement learning under algorithmic triage. arXiv preprint arXiv:2109.1 / 1328, 2021.
[44] Vahid Balazadeh, Abir De, Adish Singla, and Manuel Gomez Rodriguez. Learning to switch among agents in a team via 2-layer markov decision processes. Transactions on Machine Learning Research, 2022.
[45] Andrew Fuchs, Andrea Passarella, and Marco Conti. Optimizing delegation between human and ai collaborative agents. arXiv preprini arXiv:2309.1477 2023.
[46] Stratis Tsirtsis, Manuel Gomez Rodriguez, and Tobias Gerstenberg, Responsibility judgments in sequential human-Al collaboration. In Proceedings of the Annual Conference of the Cognitive Science Society. cognitvesciennessoietyyoo 2024.
[47] Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adaptive coverage. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 3581-3591 Curran Associates. Inc., 2020.
[48] Anastasios Nikolas Angelopoulos, Stephen Bates, Michael I. Jordan, and Jitendra Malik. Uncer- tainty sets for image classifiers using conformal prediction. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.nee, 2021.
[49] Jianguo Huang, Huajun Xi, Linjun Zhang, Huaxiu Yao, Yue Qiu, and Hongxin Wei. Conformal prediction for deep classifier via label ranking. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
[50] J Pearl. Causality. Cambridge university press, 2009.
[51] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: founda- tions and learning algorithms. The MIT Press, 2017.
13

---

[52] Mark Steyvers, Heliodoro Tejeda, Gavin Kerrigan, and Padhraic Smyth. Bayesian mod- eling of human ai complementarity. Proceedings of the National Academy of Sciences, 119(11)=2211 11547119,, 2022.
[53] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115:211-2555 2015.
[54] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Truck Proceedings, 2015.
[55] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4700 IEEE, 2017.
[56] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1-9. IEEE, 2015.
[57] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778. IEEE, 2016.
[58] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. In Advances in neural information processing systems, volume 25, 2012.
[59] Isaac Gibbs, John J Cherian, and Emmanuel J Candès. Conformal prediction with conditional guarantees. arXiv preprint arXiv:2305. /2616, 2023.
[60] Florian Heiss. Discrete choice methods with simulation. Econometric Reviews, 35:1-5, 02 2016.
[61] Amos Tversky and Itamar Simonson. Context-dependent preferences. Management science, 1191))1179-11888 1993.
14

---

A Comparison to Pearl's notation
Using Pearl's notation [50], our definition of counterfactual harm reads as follows:
(2, y. 0) = [E[max (0, { - u) 16s = -}} = x, Y = y. 91 = 0].
where the subindices in Y and Y1 denote (hard) interventions on the exogenous variable A, the random variables x, y: 1 ~ P(X, Y, Vi), and the expectation is over the prediction Y, made by the expert using the system Cx. Using the above notation, the definition does not explicitly specify the (counterfactual) distribution used in the expectation, in contrast to the definition using the notation by Peters et al. [51], which explicitly specifies the counterfactual distribution used in the expectation, as restated below for clarity.
hx (z. 9) By (M)) Xex, (o0))) [max(0, 1 = y} - -118 = -ylll.
15

---

B Proofs
B.1 Proof of Proposition
Given an observation (r, y, 9) under no intervention, we have
hy (z. y, 0) Ev PAMAAAAA K |max (0, 10 yt 1( y}}
= Ey www.P.AAAE///.......... [0, 1(y -
By PPMINCAAA) 8-8.8-x.Y--
[0,000,0~ 1( -)]1-1((f
= Ey ~ PACAAA -.
[max{0, - 1( = -()=))+
Eg P.M.KAAAA B 8-8.8-EY---
mmax(0, 10-1-11]] 1-0 - y) (x)=
y PM: 8-8.8-2.Y--
[max (0, 0}] ( = y) Cx))) (12)
0- 110 y} 1 1 E (x²)
+ =.=pptaas=M))) --p.x=...--- [max - 0}] 1(0 = .) 1 $ Cx)))
1 1- y} Ly & (x
= 1 y) - -iy & Cx(x)}
1 [Ø y Ay Cx)))
where, in Eq. 12, we used the counterfactual monotonicity property by setting 11` =y} if E Cx()..
B.2 Proof of Proposition
Under interventional monotonicity, by Lemma 1, we have
hx v. 0) = PM-MAN-) P- u)l 1 = v} 1(y
+ {ύ f} 1 (x())
2 15 PMMMMNAA - u (
+ 1 = y) 1 Cx(x))
where we used that {0 = y) 1.
By taking the expectation on both sides over á under no intervention we have Ep [hx v. 81)
S Bp [E Par 118 + #/
L/y Cx(2))
(13)
+ Evr
1(8'= = y) 11 Cx(2))
(14)
[1 + {u] 11
By
(15)
+ ExrapM
[118" y} 1 $ (x(=))
(16)
since 1iy E (x)) is constant with respect to the expectation over ÿ ~ P.M.
By Lemma 2, Eq. 13 becomes
Eyrap 82)) S Ep>
[1 xY|I= = = = w] ()
+ By PM [18'= y} 1(v $ C2(2))
(17)
By Assumption 2 for X' 1 and any À such that y Cx(x), have that
MM: (r- = Y X = x, = 2 pM (p = Y x = = y v).
which we can rewrite as
By [1(2 = 1 = Y, 2) 2 Ey YY X = Y= v).
1
16

---

and is equivalent to
PA [L(Y * x = =x, = yl Bp *YJIX = 2, Y = y]-
Using the above result, Eq. 17 becomes
E ppi [ 80] s Bynpu 11 xY x, =
Byrnpm
118 /t 1 1y Cx(2))
If we now take the expectation in both sides over X,Y Y. PMM and that Y' is equal in distribution
with Y, we have
E Ex,rP=pm
|xXX, & Ex [1 + v) 1 (((V)))))))
+ [1 = yj 1( $ (x(())
that is equivalent
E Pxyrpppm [
[1tr *Y) 11 (Cx() ry 11 $ cxx))))
S E BxyYSnPM
which we can write as
E Ex.rispm [,](x,,,)]
[1( [1 #YYYYECAMM) +1) Y AYY & (xx())
S S ExxxppM
For the lower-bound, by Lemma 1, we have
hx v. By (M)))))))))) P-o.X (1f8 + vill 100 v) -)- 1 1 Cx(x))
+ 1 /} 1 Cx(x) 2 (0 = y) 1 $ Cx()),
where we used that By PM. f=9. X ruy [1f * - v\ 1f E C2(x)} 2 0. By taking the expectation in both sides over X, Y.Y Y p.M we have
Ex.YY-PM
[[ 11 & cx(x))] < (xx.rxxnmmMaaxa... (Ma(x.y)))
that is equivalent to
E ExyyE>m 11 = YAY & cx(x))] S Exxxvrr [
thus we conclude the proof.
B.3 Proof of Theorem 1
We will show that 1 = YAY, & Cx(X,)) are exchangeable and that they satisfy the requirements of Theorem in [22] for each X,, Y,, Y, Since X,, Y,, Y, are i..d., 1(ý; = Y, Y, & Cx(X))) are also i.i.d thus exchangeable. Considering each 1(Y; = Y, A Y, & Cx())) for each X,, Y,, Y, E D as a function of X, it holds by Lemma 3 that for each X,, Y,, Y, 1(?;= = A, & (x.)) is non- increasing in À and right-continuous. In addition, it holds that 1(Ý; = -Y & ((,)) = 0 5 and that sup, 1(Y; Y, A Y, & (x,)) < 1 surely by definition.
17

---

B.4 Proof of Theorem 2
For the proof will follow similar procedure in [22]. Since 1( AYAY Cx(x)) s 1,
VX,Y, Y, X, it will hold that
(A+1 nGn(A) 118
o Yn+1 A Yx+1 CA(X++1)3
G,(()
n + 1 (18)
n n
n1 n+1
for every X E C. Now, let
X = sup(A E f (A) S a)-
(19)
If such X does exist, by Eq. 19 Å will also not exist. If X' exists,
we will show that s X. If A O, then it will hold that X' > Å =0 -0= minxec A. If X > 0, then by Eq. 19, Ā must satisfy
G Gnti(A) Gn( n n+
1 a,
(20)
which means that < X. Therefore, in any case we have that Å A X.
From Lemma 4, the random functions 16; * YAY, E Cx(X:)) are non-decreasing for each i E [n + 1). As result, since < X, we have that
E P...AKnpM # Yu+1 A AY41 E C2(Xn+1)]
< B P. [1 & Yot1 A Y241 E C2 (21)
where i = + 1}. Given that the data samples (X,, Y, 9i) for E 1,n + 1] are ex- changeable, the values 1(1; + Y, A Y; E (x,) for E [n + 1] are also exchangeable. Therefore, due to exchangeability and given that X is fixed given (X,, Y,, Y.) for i E [L, + 1], 1[ * Y A Yn+1 E (Xn+)))) has the same probability of taking any value in Dx (1 * YAY E (X.)) ie., the value of 1(2 * Y; A Y; E (X,)) for the 1-th sample in {(X,, Y, Yo follows a uniform distribution over all the possible val- ues in Dx-- As a result, from the law of total expectation it holds for i (1....,n + 1} that E Ex.. [1 [1 * Yo41 YA E (X,+)))) Edy, [1 & Yn+1 Ya+1
(X,+)) Dy1 DSS (X Gn+1(8)) Finally, from Eq. 20 and the above n+1
we have
Ex, PM [f + Ynt1 A Yut1 E C2
S <Bx,,Y. (1 & Yuti A Yotı E Cx, = Gnti(()) sa, (22)
where (l,...,
B.5 Additional Lemmas
Lemma Under the interventional monotonicity assumption, for any x, y. ~ PMM the counterfac- tual harm that decision support system Cy would have caused, if deployed, is given by
hx(x, y. 0) = -Ey PAM.stsAmmmM)) /-p.
11 + vll 110 = 1fy E Cx(x)) +1 = v 1(v & Cx()). (23)
18

---

Proof. Given an observation v. under no intervention, we have
hx v.
By (mpiiimA Pen.N
|max 1 y -
By Paraida 1-
ma 1 1(8 -11l-
+ By (M) Xxp.
mmax(0,0
00,0- - 1("" -1
By - P=B.
m0, 1 1(
-1)=)) +0
By By~PMan P....
|max(0, 1( = =]H- 1(0 = y) 1y E (x(x))
By ~ (. Fmm.
[max(0,1 - 1(8 - = -(ğ 11--1111100 - 3) 1(y (x(x))
Ey- P PMM.M
ma 11-1 *
+ PM... mmx -1y
Ey By
max (0, -v)ll-11
1( E Cx(x)
1 C
Ey- /= 11 # vHl 1(0 = y) 11 e cx(x)) + 1(0 y} 1( $ Cx(r)}
Lemma 2 Under the structural causal model M satisfying Eq. 3, given an observation (z, v) it holds
BprapM ( [118 {Y]]]
- [1 = = = (24)
Proof. Given an observation (I,, v. under no intervention, for the expectation of the prediction of the human Y AMM -. V= it holds
Ka 1(8 # mY}
Σ P(U V x v. =9) =(x(=,, # y),
where P(U u, =
1) is the posterior distribution of the exogenous variables U, conditional on the observation (x, y, 9).
By taking the expectation of the above with respect to the human prediction ý under no intervention have
[14 +Y]l]
ExIapM By
Σ PM (Ý' X X=,,
Σ P(U u, v = x, -. = 11ff u, Cx(x)) Fy}
4,31
P(U = u, -u Y, y. . 9) (8 x Yx Y
0
5
U, (,,,, 3
Since Y is equal in distribution y we can rewrite the above
[148 + xY]]
Eqinpm
=ΣΣ = PVU = y, y = 1) pM = =
0200
((ff((,,,Cxxx))) Fv)
19

---

and since P(U = u, V = u = -, = y) - Eg P(U u, V. e = x, Y = =v.
û) pMM( = -0|X= = x, == y). we have
E Exr [ey.pMiantsss) [10 # rJ]]
Σ u u, u, V = 8 X = 1/ 1158 Cx(xr) +
Σ P(U = u x = x, = y) Σ P(V = x = x, Y = y) (ff +
u
[] + eYJ z, = y]
where we used that the exogenous variables U and V are independent.
Lemma Under counterfactual monotonicity, for a given æ, y. 9 the counterfactual harm hx y: 9) non-increasing as a function of X and right-contimuous.
Proof. If y E Cx(x) AA E & then hx hx y. M) - O, AA A given Eq, 8. YÄ A such that A= inf E A y E (x(x)) then AA 2 X, it holds that E Cx (ar). and as result (x v. 9) For VX' < X, it holds that & Cx (xr and as result Eq. 8 becomes
(x, y. 9) {0 = y) (0- = y) 2 (x,,0,00)
(25)
where 1 y is constant A < X.
Lemma For given data sample x with label y, let G L + 10, be the function G(X) = 1( AyA E Cx(x)). Under interventional monotonicity, G is non non-decreasing in X.
Proof.
Let X2 L such that < X. Then it will always hold that Cx S (x)xx. We distinguish the following cases for Cx(x), (z):
y & (x) Then G(A) G(X) 0.
y É Cx(x) and E (x)) Then, G(A) = 0 and G(X) = 1( 20
y e Cx(x) (x). Then, G(A) G(X') 114 + y).
20

---

C Additional experimental details
In this section, describe the experimental details that are omitted from the main paper due space
constraints.
Mixture of multinomial logit models. Multinomial logit models (MNLs) [60] are among the most popular discrete choice models used to predict the probability that a human chooses an alternative, given of alternatives within a particular context [61]. Straitouri al. [17] used MNLs predict the probability that a human expert predicts label (e.g., the ground truth label), given a prediction set, while considering context the ground truth label and the level of difficulty of the image. To this end, for images with the same level of difficulty, they estimate the confusion matrix of the expert predictions their own and use it parameterize an MNL model. Therefore, their mixture of MNLs comprises of a different MNL model for each level of difficulty. To distinguish images based on their level of difficulty, they consider different quantiles of the experts' average accuracy values of all images. In our experiments, we consider two levels of difficulty. where in the first level we consider images for which the average accuracy of experts smaller than the 0.5 quantile, and consider the rest of the images in the second level.
Labels of the ImageNet16H dataset. The ImageNet16H [52] was created using 1,200 unique images labeled into 207 different fine-grained categories from the ILSRVR 2012 dataset [53]. Steyvers et al. [52] mapped each of these fine-grained categories into out of 16 coarse warse-grained categories namely airplane, bear, bicycle, bird, boat, bottle, car, cat, chair, clock, dog. elephant, keyboard, knife, oven, and truck- that serve the ground truth labels. This mapping essentially eliminated any label disagreement that could potentially occur between annotators the ILSRVR 2012 dataset.
Implementation details and licenses. We implement our methods and execute our experiments using Python 3.10.9, along with the open-source libraries NumPy 1.26. 4 (BSD License), and Pandas 2. 2. 1 (BSD 3-Clause License). For reproducibility. we used different fixed random seed for each random sampling the test and calibration sets. Both datasets used our experiments are distributed under the Creative Commons Attribution License 4.0 (CC BY 4.0).
21

---

D Additional experimental results on counterfactual monotonicity
In this section, we evaluate our methodology on additional strata of images and on larger calibration sets, and study the relationship between average counterfactual harm, average prediction set size and empirical coverage.
Other strata of images. Figures 4 and show the average accuracy A()) achieved by a human expert using Cx. as predicted by the mixture of MNLs, against the average counterfactual harm H (A) caused by Cx for a E {0.01, 0.05} on the strata of images with w = 80 and w = 95, respectively. The meaning and coloring of each point are the same in Figure 2. The results both strata of images are consistent with the results on the stratum of images with w = 110-the decision support systems C2 always harm-controlling sets and there is trade-off between accuracy and counterfactual harm. However, for the strata with {80, 95). where the classification task has lower difficulty compared the stratum with w 110, we observe that the trade-off between accuracy and counterfactual is minimal.
some amount of counterfactual harm, our framework successfully identifies
0.01
0.05
1.0
VGG19
VGG19
0.9
0.9
0.8
Average Acctracy
1e6seNet161
Average Accuracy
16nseNet161
0.9
0.9
0.6
GoogleNet
GoogleNet
0.85
0.85
0.4
15s
1esNet112
0.85
0.85
0.2
AlexNet
AlexNet
0.8
0.0
0.00
0.05
0.10
0.15
0.00
0.05
0.10
0.15
Average Counterfactual Harm
Average Counterfactual Harm
(a)
0.01
(b) a = 0.05
Figure 4: Average accuracy estimated by the mixture of MNLs against the average counterfactual harm for images with w = 80. Each point corresponds to À value from 0 to 1 with step 0.001 and the coloring indicates the relative firequency with which each À value is in A(a) across random samplings of the calibration set. Each row corresponds to decision support systems CΛ with a different pre- trained classifier with average accuracies 0.891 (VGG19), 0.892 (DenseNet161), 0.802 (GoogleNet), 0.804 (ResNet152), and 0.784 (AlexNet). The average accuracy achieved by human experts on their own is 0.9. The results are averaged across 50 random samplings of the test and calibration set. In both panels, 95% confidence intervals have width always below 0.02 and are represented using shaded areas.
22

---

0.01
0.05
10
VGG19
VGG19
0.9
0.9
0.8
Average Accuracy
0.9
Dens:Net161
Average Accuracy
0.9
DenseNet161
0.6
GoogleNet
GoogleNet
0.85
0.85
0.4
0.85
ResNet152
0.85
ResNet132
0.2
AlexNet
AlexNet
0.8
0.8
0.00
0.05
0.10
0.15
0.20
0.00
0.55
0.10
0.15
0.20
0.0
Average Counterfactual Harm
Average Counterfactual Harm
(a) a 0.01
(b) a 0.05
Figure 5: Average accuracy estimated by the mixture of MNLs against the average counterfactual harm for images with w = 95. Each point corresponds to a value from 0 to with step 0.001 and the coloring indicates the relative frequency with which each À value is in A(a) across random samplings of the calibration set. Each row corresponds to decision support systems Cx with a different pre- trained classifier with average accuracies 0.88 (VGG19), 0.868 (DenseNet161). 0.775 (GoogleNet), 0.773 (ResNet152), and 0.745 (AlexNet). The average accuracy achieved by human experts on their own is 0.86. The results averaged across 50 random samplings of the test and calibration set. In both panels, 95% confidence intervals have width always below 0.02 and are represented using shaded areas.
Larger calibration sets. Figure 6 shows the average accuracy A()) achieved by human expert using Cx, as predicted by the mixture of MNLs, against the average counterfactual harm (A) caused 110. The results show that the larger
by Cx for the calibration set, the more often values with average counterfactual harm H(A) close to a 12 are included in the harm-controlling set A(a). This because, the larger the number of samples in the calibration set (n), the lower the estimation error in the empirical estimate of the average counterfactual harm Hn. We find qualitatively similar results for the other strata of images.
0.01 the stratum of images with
Average prediction set-size and empirical coverage. Figure 7(a) shows the average prediction set-size each decision support system CX against the average counterfactual harm (A) caused by Cx the stratum of images with w 110. Figure 7(b) shows the empirical coverage, i.e., the fraction of samples x in the test set, for which ground truth label y E Cx(x), for each decision support system CΛ against the average counterfactual harm (A) caused by Cx the same stratum of images. The results show that systems with higher coverage construct larger prediction sets in average and cause less counterfactual harm. We find similar results for the other strata of images.
Here, note that for 0.01, these À values happen achieve the highest average accuracy A()).
23

---

0.01
0.01
1.0
0.9
0.9
VGG19
VGG19
0.8
0.8
0.8
0.9
0.9
161
DenseNet161
Awergge Accuracy
Awerage Avcurncy
0.8
0.8
0.6
0.85
0.85
0.80
GoogleNet.
GoogleNet
0.75
0.75
0.4
0.85
0.85
0.80
ResNet152
0.80
ResNet152
0.75
0.75
0.8
AlexNet
AlexNet
0.7
0.00
0.05
0.10
0.15
0.20
0.7
0.00
0.05
0.10
0.15
0.20
0.0
Average Counterfactual Harm
Average Counterfactual Harm
(a) Calibration samples n = 360 (30%)
(b) Calibration samples n = 600 (50%)
Figure 6: Average accuracy estimated by the mixture of MNLs against the average counterfactual harm for images with w = 110 and different sizes of the calibration set. Each point corresponds to value from to with step 0.001 and the coloring indicates the relative frequency with which each value is in A(a) across random samplings of the calibration set. Each row corresponds to decision support systems CA with a different pre-trained classifier with average accuracies 0.846 (VGG19), 0.830 (DenseNet), 0.722 (GoogleNet), 0.727 (55s, and 0.691 (AlexNet). The average accuracy achieved by the simulated human experts on their own 0.771. The results are averaged across 50 random samplings of the test and calibration set. In both panels, 95% confidence intervals are represented using shaded areas and always have width below 0.02.
15
1.0
10
VGG19
VGG19
0.9
15
1.0
10:
161
Empirical Coverage
DenseNet161
Average Set Size
0.9
15
1.00
10
GoogleNet
GoogleNet
0.75
10
155
ResNe:152
0.75
15
1.00
10-
AlexNet
AlexNet
0.75
0.00
0.05
0.10
0.15
0.20
0.00
0.06
0.10
0.15
0.20
Average Counterfactual Harm
Average Counterfactual Harm
(a) Average Set-Size
(b) Empirical Coverage
Figure Average set size and empirical coverage against the average counterfactual harm for images with = 110. Each point corresponds to À value from to 1 with step 0.001. Each row corresponds to decision support systems CA with a different pre-trained classifier with average accuracies as in Figure 6 above. The average accuracy achieved by the simulated human experts on their own is 0.771. The results are averaged across 50 random samplings of the test and calibration set. In both panels, 95% confidence intervals represented using shaded areas and always have width below 0.025.
24

---

E Experimental results on counterfactual monotonicity with a more complex set-valued predictor
In this section, we experiment with a system that constructs the prediction set C(x) using a more com- plex set-valued predictor [49]. This set-valued predictor first computes the following nonconformity score for each label value y E y using the softmax output of pre-trained classifier my(x) E [0, 1]:
u M1 (I)) Mayc + (i 2+ +u) We
if y V(1)
s s z. w;
(26)
if y D(1)
where M() denotes the label value with the i-th largest classifier output, random variable and w is a hyperparameter representing the weight of the ranking the label value. Then, given a user-specified threshold À IR, it constructs the prediction sets C(x) Cx as follows:
(O, 1) is uniform
L-1 Cx(x) (u(--)) with = Σ (1 S S} J=1
(27)
where, with a slight abuse of notation, V( is the label value with the i-th largest s(y, I, u; w) value. Here, we should note that may exist data samples with s(y, æ, u; w) > 1. Therefore, considering À E j0, 1] may not include every set-valued predictor given by Eq. 27. For this reason, in what follows, we consider E [0, max s(y, z, u, w)]. where the maximum is essentially over the classifier output my(2), (z), the uniform random variable and the hyperparameter w.
Experimental setup. We use the ImageNet16f dataset and follow the same setup described in Section 6, with only difference that we split the images into a calibration set (10%), which we use to find A(a). a validation set (10%). which we use to select the value of the hyperparameter following a similar procedure as in Huang et al. [49], and test set (80%). In each iteration of the experiment, given value, select the value of w {0.02, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35) for which the set-valued predictor achieves the smallest average prediction set-size over the data samples in the validation set.
The meaning and coloring each point are the same as Figure 2. The results on all strata of images consistent with the results using the set-valued predictor given by Eq. 1 the decision support systems Cx always cause some amount of counterfactual harm, our framework successfully identifies harm-controlling sets and there is trade-off between accuracy and counterfactual harm.
Results.
Figure I (a) shows the average prediction set-size of each decision support system C2 against the average counterfactual harm H (A) caused by CA on the stratum of images with w = 110. Figure 1 (b) shows the empirical coverage for each decision support system Cx against the average counterfactual harm (X) caused by CΛ on the stratum of images with = 110. Our results are consistent with the results for the set-valued predictor given by Eq. I in Figure 7-systems CA that achieve higher coverage, construct larger prediction sets on average and cause less harm.
25

---

0.01
0.05
1.0
0.95
0.95
VGG19
VGG19
0.90
0.90
0.8
0.95
0.95
Average Accuracy
161
Average Accuracy
DenseNet161
0.90
0.30
0.6
0.9
GoogleNet
0.9
GoogleNet
0.8
0.8
0.4
0.9
ResNet1122
Re
0.8
0.8
0.2
0.9
0.9
AlexNet
AlexNet
0.8
0.0
0.00
0.05
0.10
0.15
0.00
0.05
0.10
0.15
Average Counterfactual Harm
Average Counterfactual Harm
(a) 0.01
(b) = 0.05
Figure 8: Average accuracy estimated by the mixture MNLs against the average counterfactual 80. Each point corresponds to À value from to 6.25 with step
harm for images with 0.00625 and the coloring indicates the relative frequency with which each À value in A(a) across random samplings of the calibration set. Each corresponds to decision support systems Cx with different pre-trained classifier with average accuracies 0.891 (VGG19), 0.892 (DenseNet161), 0.802 (GoogleNet), 0.804 (ResNet152), and 0.784 (AlexNet). The average accuracy of human experts on their own 0.9. The results are averaged across 50 random samplings of the test and calibration set. In both panels, 95% confidence intervals have width always below 0.02 and represented using shaded areas.
0.01
0.05
1.0
0.925
0.925
0.900
VGG19
0.900
VGG19
0.875
0.875
0.8
0.925
0.925
Average Accuracy
0.900
DeuseNeti61
Average Accuracy
0.900
DenseNet161
0.875
0.875
0.6
0.9
0.9
GoogleNes
GoogleNet
0.8
0.4
0.9
ResNet152
ResNet152
0.2
0.9
0.9
AlexNes
AlexNet
0.8
0.8
0.00
0.05
0.10
0.15
0.20
0.00
0.05
0.10
0.15
0.20
0.0
Average Counterfactual Harm
Average Counterfactual Harm
(a)
0.01
(b) = 0.05
Figure 9: Average accuracy estimated by the mixture of MNLs against the average counterfactual harm for images with = 95. Each point corresponds to a À value from 0 to 6.25 with step 0.00625 and the coloring indicates the relative frequency with which each À value in A(a) across random samplings of the calibration set. Each corresponds to decision support systems C₁ with a different pre-trained classifier with average accuracies 0.88 (VGG19), 0.868 (DenscNet161), 0.775 (GoogleNet), 0.773 (Res and 0.745 (AlexNet). The average accuracy of human experts on their own is 0.86. The results are averaged across 50 random samplings of the test and calibration set. In both panels, 95% confidence intervals have width always below 0.02 and represented using shaded areas.
26

---

0.01
0.05
1.0
0.85
VGG19
0.85
VGG19
0.80
0.50
0.8
Average Accuray
0.85
DenseNet161
Average Accuracy
0.85
DeaseNet161
0.80
0.80
0.6
0.85
0.85
0.80
GoogleNet
0,50
GoogleNet
0.75
0.75
0.4
0.80
ResNet132
0,50
ResNet122
0.75
0.75
0.2
0.8
AlexNet
0.8
AlexNet
0.7
0.7
0.00
01.05
0.10
0.15
0.20
0.00
0,35
0.10
0.15
0.20
0.0
Average Counterfactual Harm
Average Counterfactual Harm
(a)
0.01
(b) a = 0.05
Figure 10: Average accuracy estimated by the mixture of MNLs against the average counterfactual harm for images with w 110. Each point corresponds to À value from 0 to 6.25 with step 0. 0.00625 and the coloring indicates the relative frequency with which each value is in A(a) across random samplings of the calibration set. Each row corresponds to decision support systems Cx with a different pre-trained classifier with average accuracies 0.846 (VGGI9), 0.830 (DenseNet), 0.722 (GoogleNet), 0.727 (ResNer152), and 0.691 (AlexNet). The average accuracy achieved by the simulated human experts on their own is 0.771. The results are averaged across 50 random samplings the test and calibration set. In both panels, 95% confidence intervals represented using shaded areas and always have width below 0.02.
15
1.0
10
VGG19
VGG19
0.9
15
1.0
10-
Empirical Coverage
DenseNet161
Average Set Size
15
1.00
10-
GoogleNet
GoogleNet
0.75
13P
1.00
10
ResNet152
ResNet152
0.75
13:
1.00
10
AlexNet
AlexNet
0.00
0.05
0.10
0.15
0.20
0.00
0.35
0.10
0.15
0.20
Average Counterfactual Harm
Average Counterfactus' Harm
(a) Average Set-Size
(b) Empirical Coverage
Figure 11: Average set size and empirical coverage against the average counterfactual harm for images with w 110. Each point corresponds À value from 6.25 with step 0.00625. Each row corresponds to decision support systems Cλ with a different pre-trained classifier with average accuracies as in Figure 10 above. The average accuracy achieved by the simulated human experts on their own is 0.771. The results are averaged across 50 random samplings of the test and calibration set. both panels, 95% confidence intervals are represented using shaded areas and always have width below 0.025.
27

---

F Average accuracy vs. prediction set size
In this section, we use the ImageNet16H-PS [18] dataset verify the interventional monotonicity assumption, ie., whether human experts achieve higher average accuracy when predicting from smaller prediction sets that include the ground truth label. To this end, we follow a procedure, similar to the one used by Straitouri et al. [18] to verify the interventional monotonicity assumption unconditionally of the value of the ground truth label.
Straitouri et al. estimate the average accuracy per prediction set size on images with similar difficulty, averaged across all experts and across experts with the same level of competence. They consider images with similar difficulty instead of single images, as the dataset ImageNet16H-PP does not include enough human predictions per image to faithfully estimate the average accuracy per image. They stratify the images into groups of similar difficulty, following the same procedure used in Straitouri al. [17] based on different quantiles of the average accuracy values of all images. In our work, we stratify images with the same ground truth label into the following four groups:
High difficulty: images with average accuracy within the 0.25 quantile of the average accuracy values of all images with the same ground truth label.
Medium to high difficulty: images with average accuracy within the 0.5 quantile and outside the 0.25 quantile of the average accuracy values of all images with the same ground truth label.
Medium to low difficulty: images with average accuracy within the 0.75 quantile and outside the 0.5 quantile the of the average accuracy values of all images with the same ground truth label.
Low difficulty: images with average accuracy outside the 0.75 quantile of the average accuracy values of all images with the same ground truth label.
We follow a similar method to stratify experts based on their level of competence into two groups for each ground truth label. To measure the level of competence of an expert for ground truth label, we use the average accuracy across all the predictions that she made on images with this ground truth label. For each ground truth label, consider the 50% of experts with the highest average accuracy as the experts with high level of competence and the rest 50% as experts with low level of competence.
Figure 12 shows the average accuracy per prediction set size-for prediction sets that include the ground truth label-across images of various levels of difficulty and experts with different levels of competence for some the ground truth labels. We find qualitatively similar results for the rest of the ground truth labels. The results monotonicity by Straitouri et al. [18]. showing that as long as the classification task not too easy, the experts achieve higher average accuracy when predicting from smaller prediction sets, i.e. the interventional monotonicity holds.
consistent with the results the unconditional interventional
28

---

11H
0.-
06
Aonag Amsuna
0.
Prodiction Sixe
Prodictio Set Size
Prsdictio Set
High difficulty
Medium to high difficulty
Medium to low difficulty
(a) All experts, ground truth label bottle
02-
Aemoo Amsuucy
0.44
005
00
0.6
High difficulty
Medium high difficulty
Medium low difficulty
(b) All experts, ground truth label oven
0.6
Avogee Arrurry
088
0.4-
0.3-
BuG-
Spesuge
02
15
Prodictio
High difficulty
Medium to high difficulty (c) Experts with low level of competence, ground truth label bottle
Medium low difficulty
1.0
9.6
000
10
14
0U5
15
High difficulty
Medium to high difficulty
Medium to low difficulty
(d) Experts with high level of competence, ground truth label bottle
Figure 12: Empirical success probability per prediction set size averaged across (a) all experts for
images with ground truth label bottle, (b) all experts for images with ground truth label oven, (c) experts with low level of competence for images with ground truth label bottle, and (d) experts with high level of competence for images with ground truth label bottle with high, medium to high and medium to low difficulty. In all panels, we have only considered prediction sets that included the ground truth label and thus have omitted showing the empirical success probability for singletons, as is always 1. Error bars denote standard error.
29

---

G Experiments under the interventional monotonicity assumption
In this section, we show the average accuracy (()) achieved by human expert using C2, as predicted by the mixture of MNLs, against the average counterfactual harm upper- bound caused by Cx for several a values on the strata of images with w 80 in Figure 13, with w 95 Figure 14 and with = 110 in Figure 15. Here, each point corresponds to different À value and its coloring indicates the empirical probability that is included in the harm-controlling set A'(x) given by Corollary 2, for fixed a' value across the random samplings of the test and calibration sets. For each classifier and stratum of images, we select the a' value maximizing the expected size of the harm-controlling set A(a). The results on all strata of images support the findings derived from the experiments assuming that the counterfactual monotonicity assumption holds. In addition, we observe that the gap between the lower-bound of the average counterfactual harm (that is equal to the average counterfactual harm under the counterfactual monotonicity assumption) and the upper-boure bound often large.
0.12
0.14
1.0
VGG19
VGG19
0.9
0.9
0.8
Average Aoctrracy
DeuseNet161
Average Accuracy
DenseNet161
0.9
0.9
0.6
GoogleNet
GoogleNet
0.85
0.85
0.4
ResNet152
ResNet152
0.85
0.85
0.2
AlexNet
AlexNet
0.8
0.8
0.10
0.15
0.20
0.10
0.15
0.20
0.0
Average Counterfactual Harm Bound
Average Counterfactual Harm Bound
(a) a 0.12
(b) a = 0.14
Figure 13: Average accuracy estimated by the mixture of MNLs against the average counterfactual
harm upper bound for images with w = 80. Each point corresponds to X value from 0 to with step 0.001 and the coloring indicates the relative frequency with which each À value is in A' N((a) across random samplings of the calibration set for a fixed a'. Each row corresponds to decision support systems with a different pre-traines classifiers as in Figure 4. The results are averaged across 50 random samplings of the test and calibration set. In both panels, 95% confidence intervals have width always below 0.02 and are represented using shaded areas.
30

---

0.15
0.17
1.0
VGG19
VGG19
0.9
0.8
1esseNes161
DenaeNet161
Average Accuracy
0.9
Average Accuracy
0.9
4.6
GoogleNet
GoogleNet
0.85
0.85
0.4
0.85
ResNet152
0.85
ResNes152
0.2
AlexNet
AlexNet
0.8
0.8-
0.150
0.175
0.200
0.225
0.250
0.275
0.150
0.175
0.200
0.225
0.250
0.275
0.0
Average Counterfactual Harm Bound
Average Counterfactual Harm Bound
0.15
(b) a 0.17
Average accuracy estimated by the mixture of MNL. against the average counterfactual harm upper bound for images with w = 95. Each point corresponds to À value from 0.001 and the coloring indicates the relative frequency with which each À value is in A' A'(a) across random samplings of the calibration set for a fixed a'. Each row corresponds to decision support systems with different pre-trained classifiers in Figure 5, The results averaged across 50 random samplings of the test and calibration set. In both panels, 95% confidence intervals have width always below 0.02 and represented using shaded areas.
Figure 14:
with step
0.23
0.24
1.0
0.9
DenseNet161
DenseNet161
0.8
0.8
0.8
Average Accuracy
0.85
Average Accuracy
0.85
GoogleNet
GoogleNet.
0.80
0.80
0.75
0.75
0.85
0.85-
ResNet152
ResNet132
0.4
0.80
0.80
0.75
0.75-
0.2
AlexNet
0.8
AlexNet
0.7
0.0
0.22
0.24
0.26
0.28
0.30
0.22
0.24
0.26
0.28
0.30
Average Counterfactual Harm Bound
Average Counterfactual Harm Bound
(a) a = 0.23
(b) = 0.24
Figure 15:
Average accuracy estimated by the mixture of MNLS against the average counterfactual harm upper bound for images with = 110. Each point corresponds to a À value from 0 to 1 with step 0.001 and the coloring indicates the relative frequency with which each value is in A'(a) across random samplings of the calibration set for a fixed a'. Each row corresponds to decision support systems with a different pre-trainse mreitrineed classifiers as in Figure 2. For the classifier VGG19, the average counterfactual harm bound is ~ 0.23 for each À value. The results are averaged across 50 random samplings of the test and calibration set. In both panels, 95% confidence intervals have width always below 0.02 and are represented using shaded areas.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work discussed in Section 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
32

---

Justification: The proofs of all the theoretical results are provided in Appendix B.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The information to reproduce the experimental results is provided in Section 6 and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer:
[Yes]
Justification: The datasets as well as the source code we use are publicly available (see Footnotes 7, 8, and 10).
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.ce cc/ public/guie ctteees/cCoSSuuiissiiinne
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
33

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. laapsscccpubblic ppppic/gguiddsssssppeeppppppaaee httgg/udde//ooSSubbnin Codeeeumisssiiiee ssiiisolccyyyyycoocc for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the details in Section 6 and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In Section 6, and Appendix D, Appendix E and Appendix G we report 95% confidence intervals, and in Appendix F we report standard emrors.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
34

---

Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We report computer resources in Footnote 7.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e-g- preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurrps:
sttppe///herrpp..c//bbbtt/tttttt http.////bbccctttcccssseeellllll innes
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
The paper discusses both potential positive and negative societal impacts of the work in Section 7.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
35

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper does not introduce any data nor models with high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators of the datasets we use are credited in Section and Appendix C, the URLs of the datasets are provided in Footnotes 8 and 10 and the licenses of the datasets and of the libraries used in the code are provided in Appendix C.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thwode.cco...... com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The open-source implementation our methodology that we publicly release contains documentation.
Guidelines:
36

---

The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
37


Reviews:
[{"review_id": "oKJCfqZ2ff", "reviewer": "Reviewer_RCNh", "summary": "This paper relates to decision support systems based on prediction sets. It introduces a framework using conformal risk control to design prediction sets that balance accuracy with minimizing potential harm (understood as a human who has succeeded at predicting the ground-truth label of an instance on their own failing had they used the DS system), validated with human prediction data from empirical lab studies.", "questions": "1. Are there practical examples of decision support systems based on prediction sets being successfully implemented in healthcare or other social domains? How do these systems handle the inherent complexity and variability of real-world decision-making scenarios, such as patient care pathways or policy decisions?\n\n2. Have the authors considered recent research or critiques regarding the actionability of predictive models? For instance, how does the work align with concerns raised by Wang et al (2024) \"Against Predictive Optimization\" and Liu et al. (2024) \"On the actionability of...\"?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Originality. The paper establishes formal definitions/characterizations and a theoretical foundation. The paper formally characterizes the notion of \"counterfactual harm\" and introduces the counterfactual monotonicity assumptions and weaker interventional monotonicity assumptions to make counterfactual harm identifiable and partially identifiable, respectively. This conceptual contribution improves the understanding of the risks associated with decision support systems based on prediction sets.\n\n- Significance. The authors demonstrated that the idea of \"counterfactual harm\" can be made practically useful through conformal risk control (section 5), and they performed lab experiments to validate their theoretical bounds. The result is practical: Cor 1 and 2 ensure that these systems cause less counterfactual harm on average than a user-specified threshold.", "weaknesses": "1. The main weakness is the usability and motivation of this particular formulation. The paper proposes decision support systems based on prediction sets, which might not always be feasible or realistic, especially in complex domains like healthcare. Most current systems are based on risk prediction e.g. disease risk, drop out risk, for binary outcomes. I don't see any good examples of problems where prediction sets are the right approach, and this paper also does not do this work of illustrating or justifying this formulation using real world case studies.\n\n2. Discussing the Limitations of Predictive Accuracy. While the paper discusses improving accuracy through prediction sets, there may be fundamental limitations in predicting outcomes accurately in the first place. Have the authors considered recent research or critiques regarding the actionability of predictive models? For instance, how does the work align with concerns raised by Wang et al (2024) \"Against Predictive Optimization\" and Liu et al. (2024) \"On the actionability of...\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Practical examples]** We would like to first clarify that, in the medical domain, prediction sets are often referred to as differential diagnoses. A prominent example of a decision support system that uses patient history and skin condition images to provide differential diagnoses (prediction sets) for skin diseases has been recently developed by Google (see references [1, 2]). This system has been CE-marked as a Class I medical device in the European Union and a study by Jain et al. [3] has concluded that the assistance of this tool was associated with improved diagnoses by physicians and nurses for 1 in every 8 to 10 cases, indicating potential for improving the quality of dermatologic care. We will include this example in the introduction of the revised version of our paper.\n\n[1] Google Health blogpost. Improving access to skin disease information.\n\n[2] Liu, Y., Jain, A., Eng, C. et al. A deep learning system for differential diagnosis of skin diseases. Nature Medicine 26, 900–908 (2020). \n\n[3] Jain, A., et al. Development and assessment of an artificial intelligence--based tool for skin condition diagnosis by primary care physicians and nurse practitioners in teledermatology practices. JAMA network open, 4(4), e217249--e217249 (2021). \n\n**[Complexity and variability of real-world decision-making scenarios]** Decision systems based on prediction sets such as ours and the example highlighted in the previous reply aim to help decision makers make more accurate predictions about outcomes of interest in real-world decision making scenarios. However, more research is needed to evaluate the impact that these systems and their predictions may have on patient care pathways or policy decisions.\n\n**[Limitations of predictive accuracy]** Since Wang et al. (2024) and Liu et al. (2024) were published shortly before the NeurIPS deadline, we were unintentionally unaware of their critique. However, during the rebuttal period, we have carefully read both papers and we think our work actually aligns with the critiques against prediction optimization raised by Wang et al. (2024) and Liu et al. (2024). More specifically, in lines 115-117 of our paper, we argue that improving accuracy may come at the cost of causing harm. Motivated by this observation, the goal of our work is to identify systems that are guaranteed to cause less harm, on average, than a user-specified bound. In Section 7, we discuss other critiques raised by Wang et al. (2024), such as addressing fairness considerations and the challenge of distribution shift. In the revised version of the paper, we will cite and discuss Wang et al. (2024) and Liu et al. (2024) in the introduction and related work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GGGOJekCxe", "reviewer": "Reviewer_DuUW", "summary": "This paper focuses on decision support designed to help humans in multi-class classification tasks via prediction sets. The paper defines the notion of counterfactual harm, namely when usage the system would lead the agent to a wrong prediction which would have been correct without the system’s ‘help’. The authors show that under some monotonicity assumptions such counterfactual harm is identifiable or partially identifiable. Against this theoretical backdrop, the paper proposes a method to find sets of interventions (namely thresholds for the prediction sets) which are guaranteed to have bounded harm. This approach is validated using experiments on human subject studies. These studies reveal a trade off between post-intervention accuracy and counterfactual harm.", "questions": "Beside remarks on the weaknesses mentioned above, I would be interested in answers to the following questions:\n1.   In Figure 1 you show the average accuracy of humans estimated by MNLs when w=110, but in the text you report that the second dataset contains the predictions of humans with prediction sets at all thresholds for that value of w, which is then shown in Figure 2. Why not start with the actual accuracy then, instead of the estimates? It seems to me that Figure 2 has the core findings, while Figure 1 is more of a sensitivity analysis on the fact that this phenomenon is consistent across model architectures (under the assumptions that the MNL estimates are qualitatively reliable).\n2. Three of the models tested have worse average performance than the human experts. What is the point of testing decision support with such models? I suppose we would never implement a decision support that is worse than humans alone (unless perhaps we have strong evidence it is better than humans on a subgroup of data points, but this is not the case here), since it can stir the humans with poor predictions. I can see this could be interesting since you are studying counterfactual harm, but still it is an unrealistic scenario so I would not present results for three such models.", "limitations": "The paper seems to mention and discuss all the relevant assumptions and limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "I am not an expert in prediction sets so I cannot fully evaluate how significant this contribution is, and was not able to fully check the proofs. I saw the following strengths:\n1. The trade off between accuracy and counterfactual harm is very interesting, and the possibility to control the desired amount of counterfactual harm seems very important.\n2. The theoretical results seem solid and are well rounded out by the corroborating experiments.\n3. Despite the technicality of the paper, I could follow the argument and appreciate the message. The text is well written.", "weaknesses": "1.      The notation little y-hat and capital Y-hat in section 3 is somewhat confusing. Line 123 implicitly suggests the latter is the random variable generated by the user predicting with the decision support. Line 142 however states that little y-hat is the prediction the experts would make on their own. I would have expected this to be just   Y-hat but with lambda = 1, as stated elsewhere in the text?\n2. In the first experiment, when w is different from 110, the data contains predictions made by humans on their own, and the accuracy of the humans using prediction sets is only estimated. How good are these estimates? We only get to see a comparison with the real value in Figure 3 for w=110, and it is clear that these estimates decisively overshoot.\n3. The results show that the sets of lambdas do not contain all the harm-controlling values - as the authors remark - and especially seem to contain less of the ‘good’ values, meaning those that (at the cost of some counterfactual harm within the accepted bound) achieve higher accuracy. This seems to be a serious drawback.\n4. Lambda is used in line 105 but only introduced in line 109.\n5. Theorem 1 and 2 could use a bit more intuition and/or explanation.\n\nTypos:\nline 137: use -> used", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[y-hat & Y-hat]** The random variable $\\hat{Y}$ always denotes the expert prediction from a prediction set $\\mathcal{C}\\_{\\Lambda}(X)$ following the SCM specified in Eq.3. Under no intervention, $\\Lambda=1$ and thus $\\mathcal{C}_{\\Lambda}(X) = \\mathcal{Y}$, as noted in Line 131-132, i.e., the human expert predicts on their own. The variable $\\hat{y}$ is always a realization of the random variable $\\hat{Y}$, as clarified in Page 3, Footnote 2. Consequently, in Line 142, the variable $\\hat{y}$ is the realization of the random variable $\\hat{Y}$ under no intervention, i.e., $\\Lambda = 1$.\n\n\n**[Estimated accuracy]** To compare the average accuracy estimated using the mixture of MNLs to the average accuracy estimated using predictions made by human participants, we resort to the dataset made publicly available by Straitouri et al. [18]. Unfortunately, this dataset only comprises human predictions for $\\omega = 110$ and a single classifier. However, since the reported cost for collecting such data was 7,150 British pounds [18, Section 6], we could not afford gathering additional data to assess the quality of the average accuracy estimated using the mixture of MNLs for other classifiers and $\\omega$ values.\n\nWhile it is clear that the average accuracy estimated using the mixture of MNLs is higher than the average accuracy estimated using predictions made by human participants, as acknowledged in Line 303-306, the estimated accuracy follows the same trend and supports the same qualitative conclusions.\n\n\n**[‘Good’ values in $\\Lambda(\\alpha)$]** We would like to highlight that, in Figures 2 and 3, the average counterfactual harm $H(\\lambda)$ is estimated using data from $50$ random realizations of **a test set**. Here, for each realization of the test set, we apply Theorem 1 and Corollary 1, and include a $\\lambda$ value in the set $\\Lambda(\\alpha)$ if the empirical estimate $\\hat{H}_n(\\lambda)$ of the average counterfactual harm using data from **a calibration set**, disjoint from the test set, is smaller than the user-specified bound $\\alpha$.\n\nConsequently, the closer $H(\\lambda)$ is to $\\alpha$ from below, the higher the probability that the empirical estimate $\\hat{H}_n(\\lambda) > \\alpha$ due to estimation errors and thus $\\lambda \\notin \\Lambda(\\alpha)$. In Figures 2(a) and 3(a), for the particular choice of $\\alpha = 0.01$, it turns out the harm-controlling $\\lambda$ values with the highest accuracy $A(\\lambda)$ (the 'good' values) are those for which $H(\\lambda)$ is closer to $\\alpha$ and thus are less often included in $\\Lambda(\\alpha)$. However, this is an inherent limitation of using conformal risk control and, more broadly, distribution-free uncertainty quantification, not a specific drawback of our framework.\n\nThat said, during the rebuttal period, we have conducted additional experiments with larger calibration sets. The results are summarized in Figure 1 in the one page pdf attached with the author rebuttal and show that, for larger calibration sets, the ‘good’ values of $\\lambda$ are more often included in the set $\\Lambda(\\alpha)$ since the estimation error is lower.\n\n**[Line 105]** There is a typo in line 105; $\\mathcal{C}_{\\lambda}(x)$ should have been $\\mathcal{C}(x)$. We will fix the typo in the revised version of the paper.\n\n**[Theorem 1 and 2]** We will present an intuitive explanation of Theorem 1 and 2 in the revised version of our paper.\n\n**[Figure 1 and 2]** We agree with the reviewer and will first present the results currently shown in Figure 2 and then the results in Figure 1 in the revised version of the paper. \n\n**[Classifiers worse than human experts]** We would like to point out that, in contrast with decision support systems that provide single label predictions, decision support systems that provide prediction sets such as ours can help humans improve their predictions even if the classifier used to construct the prediction sets has lower average accuracy than the humans, as (empirically) shown by Straitouri et al., ICML 2023 [17; Tables 1 and 2, Figure 3]. In fact, note that, for the three classifiers with worse average performance than the human experts, there always exist values of $\\lambda$ for which the average accuracy of the predictions made by humans using our system, as estimated using the mixture of MNLs, is greater than the average accuracy of the predictions made by humans on their own, as estimated using real human data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RIyFrxnIia", "reviewer": "Reviewer_8Whm", "summary": "This paper addresses the issue with prediction methods that generate a set of potential labels, from which a human makes the final decision. The authors identify a harmful scenario where the model's intervention leads to incorrect predictions that a human would have otherwise made correctly. To model this, they use the Structural Causal Model (SCM) framework.\n\nThe prediction set size is controlled by a parameter $\\lambda$. For a given $\\lambda$, they define counterfactual harm as the average number of instances where a human's unaided prediction would be correct, but becomes incorrect with the model's assistance. The main objective of the paper is to identify the range of $\\lambda$ values that keep the counterfactual harm within a user-specified constant $\\alpha$.\n\nTo achieve this, the authors use conformal prediction and propose two training algorithms under two conditions:\n\n- Counterfactual monotonicity: At the instance level $(x, y, \\hat{y})$, smaller and correct prediction sets lead to better human predictions.\n- Interventional monotonicity: At the sub-population level characterized by $(x, y)$, smaller and correct prediction sets lead to better human predictions *on average*.\n\nThey conduct experiments on two image datasets where human predictions from prediction sets are available, demonstrating that their methods effectively control counterfactual harm.", "questions": "**Equation 4:**\n\nIn the definition of CF harm, why is the true label $Y$ a function of the predicted label $\\hat{Y}$? Could you please elaborate on the notation?\n\n**Line 164:**\n\nShould $v$ not be abducted from the joint distribution of $(X, Y, \\hat{Y})$ rather than just $X$?\n\n**Lines 127 to 130:**\n\nI do not understand the text in these lines. Are you stating that one can always find some distribution over the exogenous variables that aligns with the observational data, or are you claiming that the original SCM is identifiable?\n\n**Typo at Line 159:**\n\nhas --> had\n\n**Proof for Proposition 1:**\n\nIn the proof for Proposition 1, where have you utilized the CF monotonicity condition? The proof seems to hold for any arbitrary SCM $\\mathcal{M}$. Am I missing something here?\n\n\n**On CF monotonicity:**\n\nI believe that under no intervention, a human predicts the label with $C_\\lambda(x) = \\mathcal{Y}$, i.e., the entire label set. If CF monotonicity holds, why wouldn't any conformal prediction algorithm that guarantees $1-\\alpha$ coverage satisfy the CF harm requirement?\n\n**Examples for CF monotonicity violations**\n\nCan you provide examples from your considered datasets where CF monotonicity is violated. In practice, why would such violations happen?\n\n**Tradeoff between CF harm and conformal accuracy:**\n\nI believe accounting for CF harm may unnecessarily increase the prediction set size, beyond what vanilla conformal prediction would offer, for the same amount of coverage guarantees provided by both methods. Can the authors comment on this, at least empirically?\n\nIf such a tradeoff exists, this approach would be interesting if one could find the subset of instances where harm occurs and explicitly find the prediction set for such instances. However, I am disappointed that the paper does not propose such approaches.\n\n**Conformal baselines**\n\nThe authors should compare their algorithms with the state-of-the-art conformal prediction algorithms and evaluate how their methods perform in terms of (a) conformal accuracy, (b) prediction set size, and (c) CF harm. The current experimental section appears too weak.\n\n\n**Conformalizing $\\hat{Y}$:**\n\nWhy shouldn't this approach work? Suppose we apply conformal prediction to both the ground truth $Y$ and the expert prediction $\\hat{Y}$, and then emit the prediction set as a union of the two prediction sets obtained from these conformalized models. How does your approach compare to such a proposal in terms of the accuracy $A(\\lambda)$ and the prediction set size?", "limitations": "Yes, limitations are addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The problem formulation is interesting. The authors establish an objective that essentially tunes the threshold parameters used in Conformal Prediction, aiming to control the number of instances where the prediction sets impede the human from making the correct prediction. Solving the optimization problem requires access to counterfactual labels for human predictions across all possible prediction sets, which are naturally unavailable in observational datasets. To address this, the authors identify two weaker conditions, which are highly likely to hold in practice, to solve the problem.\n\nFinally, they apply the conformal prediction algorithm on a hold-out calibration dataset to identify the set of $\\lambda$ values that bring the CF harm under control.", "weaknesses": "Please refer the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Eq. 4]** In Eq. 4, the true label $Y$ is not a function of the predicted label $\\hat{Y}$. The expectation is over the predicted label $\\hat{Y}$ and, following standard notation used elsewhere, the distribution $P^{\\mathcal{M} ; do(\\Lambda = \\lambda) | \\hat{Y} = \\hat{y}, X = x, Y=y}(\\hat{Y})$ denotes the counterfactual distribution entailed by the structural causal model $\\mathcal{M}$ under the intervention $do(\\Lambda = \\lambda)$ given the observation $\\hat{Y} = \\hat{y}, X = x, Y=y$.\n\n**[Line 164]** The distribution of the noise $v$ is conditioned (only) on $X=x$ because, in Eq. 6, the variable $X=x$ is fixed but the variables $Y$ and $\\hat{Y}$ are not fixed. \n\n**[Lines 127-130]** We state the former.\n\n**[Proof of Proposition 1]** We used the CF monotonicity assumption in the 5th line of the proof where we used that $\\Bbb{1}\\\\{\\hat{Y} = y\\\\} = 1$ if $y \\in \\mathcal{C}\\_{\\lambda}(x)$. We will clarify this step in the revised version of the paper. \n\n**[On CF monotonicity]** Under the CF monotonicity, any conformal predictor that guarantees $1-\\alpha$ coverage is guaranteed to cause average counterfactual harm smaller or equal than $\\alpha$. However, there may be conformal predictors that guarantee $1-\\alpha$ coverage and cause average counterfactual harm smaller than a strictly smaller bound $\\alpha' < \\alpha$. This is because the average counterfactual harm does not only depend on the coverage probability but also the probability that a human predicts the ground-truth label successfully on their own. \n\n**[Examples for CF monotonicity violations]** In general, since counterfactuals lie within level three in the “ladder of causation” [19], it is not possible to identify CF monotonicity violations directly from observational data. A violation of counterfactual monotonicity may happen in the following situation. Let $\\mathcal{S} \\subseteq \\mathcal{T}$ be two prediction sets and assume both sets contain the true label $y$. If there is a label value $y' \\in \\mathcal{T} \\backslash \\mathcal{S}$ that help the human expert realize that $y$ is the true label, the expert may succeed at predicting $y$ in $\\mathcal{T}$ but may fail at prediction $y$ in $\\mathcal{S}$.\n\n**[Tradeoff between CF harm and conformal accuracy]** We would like to clarify that, if one sets the value of the threshold $\\lambda$ to be roughly the $1-\\alpha$ quantile of the empirical distribution of the scores $1-m_{y}(x)$ in the calibration set, then, the set valued predictor defined by Eq. 1 is equivalent to a vanilla conformal predictor with nonconformity scores $1-m_{y}(x)$ and coverage $1-\\alpha$. This type of conformal predictors are the state of the art for decision support systems based on prediction sets [17, 18]. Under this view, it becomes apparent that, given the same amount of coverage guarantee, there is not a trade-off between prediction set size and average counterfactual harm and, by searching for $\\lambda$ values in $[0,1]$ that are harm controlling, we are essentially searching for vanilla conformal predictors that are harm controlling.\n\n**[Conformal baselines]** As discussed in our previous answer, the class of set valued predictors defined by Eq. 1 are equivalent to the class of vanilla conformal predictors with nonconformity scores $1-m_{y}(x)$.  Moreover, since our focus is on (counterfactual harm of) decision support, we have used the average accuracy of the predictions made by humans using prediction sets constructed by the above set-valued predictors as evaluation metric, similarly as the state of the art [17,18]. However, following the reviewer's suggestion, we have additionally evaluated all the set-valued predictors considered in our experiments in terms of empirical coverage (conformal accuracy) and prediction set size against average counterfactual harm. In the pdf attached with the rebuttal, we summarize the results for the set-valued predictors used in Figure 2.\n\n**[Conformalizing $\\hat{Y}$]** Given a set of set-valued predictors and a user-specified bound $\\alpha$, the goal of our work is to identify the subset of them that cause average counterfactual harm smaller than $\\alpha$. To this end, we have considered the set of set-valued predictors defined by Eq. 1, which as discussed above, are equivalent to the set of vanilla conformal predictors with nonconformity scores $1-m_{y}(x)$ and are the state of the art for decision support systems based on prediction sets [17, 18]. \n\nConsequently, given the set of set-valued predictors suggested by the reviewer, or other conformal predictors with different nonconformity scores, one could think of extending our framework to identify the subset of these predictors that cause average counterfactual harm smaller than $\\alpha$, however, we leave this as future work. We will clarify this in Section 7 under Methodology.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5NUdc5690U", "reviewer": "Reviewer_KmhA", "summary": "The paper introduces a notion of counterfactual harm for a classification setting in which a human is choosing among a subset of possible labels determined by an AI system, i.e., the automated system is used to \"narrow down\" the set of most likely labels for a sample. Under assumptions of counterfactual and interventional monotonicity, which imply that , the notion of harm can be identified pointwise or up to an interval. A procedure for controlling harm is introduced, based on conformal risk control.", "questions": "Please see (W1), (W2).\n\n(Q1) Lines 147-148 claim that the $\\Lambda (\\alpha)$ set cannot be found because counterfactual harm is a counterfactual quantity. This statement seems imprecise. For instance, the effect of treatment on the treated (ETT), is a counterfactual quantity that can be estimated with lack-of-confounding assumptions. Furthermore, any counterfactual quantity could be, in principle, bounded without additional assumptions. \n\nIt seems more likely that the joint distribution of $\\hat Y, \\hat Y _{\\lambda}$ is required to evaluate the expression (and this seems to be a consequence of the max() operator within the expectation), which is known hard problem, often solved with monotonicity assumptions.\n\n(Q2) When taking a step back, the question that comes to mind is: what is the end goal of the proposal? When minimizing counterfactual harm, it seems that \"counterfactual benefit\" is almost entirely ignored. Is this a reasonable notion to ignore? Furthermore, if the definition were to remove the max(), would this be equivalent to optimizing overall accuracy or something similar (note that removing the max solves the identifiability issue)? Finally, should there be an extension in which $\\lambda = \\lambda(x)$ is a function of $X = x$, so that harm is minimized in areas where it is largest?\n\n(Q3) Line 142: \"Otherwise, the expert's prediction could not have become worse\". The claim of the paper is about decision-making support systems, whereas this comment seems quite closely focused on accuracy. For instance, in the very example given in Figure 1: if someone has a common cold, mistaking this for a flu seems rather different than mistaking it for lung cancer (the consequences of these two classifications may be rather different)? Please comment on this. \n\n(Q4) Lines 127-130 are saying that an SCM can match the observational distribution. Can you comment on the significance of this observtion? Is this remark made to claim that the usage of SCMs does not limit the generality of the approach, due to their expressiveness? If so, perhaps it is worth adding such a sentence.", "limitations": "Please check (W1), (W2), and (Q1), (Q3).", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "(S1) The described problem is important. Furthermore, not many works in this space exist, and the definition of harm is not universally agreed upon, as noted by the authors. Thus, this paper explores a valuable direction of research.\n\n(S2) Many of the concepts are introduced in a clean way, and the writing is good overall.\n\n(S3) The paper seems to combine many state-of-the-art ideas, including notions of counterfactual harm, identification under monotonicity assumptions from previous works, and conformal risk control.", "weaknesses": "Major:\n\n(W1) The causal inference notation is not up to the expected standard. In Definition 1, which is key to the paper, the notation does not work. The paper is trying to describe the notion of counterfactual harm, while using only the $do()$ operator and conditioning. The do() operator and conditioning can only give quantities in the first two layers of the causal hierarchy. Therefore, the current definition, as written, is incorrect. The appearance of $\\hat Y$ in multiple places with different meanings is concerning. Instead of this, the paper probably wants to use the subscript notation $\\hat Y _{\\lambda}$. \n\nMy understanding is that the paper is trying to say\n\n$$ h_\\lambda (x, y, \\hat y) = E [ max (0, \\mathbb{1} ( \\hat Y = Y ) - \\mathbb{1} (\\hat Y _{\\lambda} = Y) ) \\mid X = x, \\hat Y = \\hat y, Y = y]. $$\n\n(W2) The connection to Richens et. al [1] is mentioned in the introduction. It is clarified that the difference here is that \"their definition requires one to deploy the system\", while this is not the case for the definition proposed in this work. It is not clear from the text whether  the current definition does not require deployment because of the definition itself, or the monotonicity assumptions that are invoked.\nThe two definitions seem closely related, and given the comment in (W1), it would be very valuable to have these two definitions compared side-by-side, with a unified choice of corrected notation. Opening a small appendix for this would be very desirable.\n\n---\n\nMinor comments / corrections: \n\n(W3) Relating to point (W1): in Pearl's notation, causal mechanisms are usually denoted by the assignment operator $\\gets$, i.e., $X \\gets f_X (U_X)$. This is useful to indicate that the mechanisms are not \"equations that can be manipulated\".\n\n(C1) Line 159: \"if they has failed\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1]** In our paper, we follow the same notation used in Elements of Causal Inference by Peters, Janzing and Schölkopf, a standard book in causality research with over 2,000 citations in Google Scholar, also used in [18] (ICML 24). In particular, under the notation used by Peters et al. to denote interventions in a counterfactual distribution in Chapter 3.3, we think our current definition is correct. In Definition 1, the distribution $P^{\\mathcal{M} ; do(\\Lambda = \\lambda) | \\hat{Y} = \\hat{y}, X = x, Y=y}(\\hat{Y})$ denotes the counterfactual distribution entailed by the structural causal model $\\mathcal{M}$ under the intervention $do(\\Lambda = \\lambda)$ given the observation $\\hat{Y} = \\hat{y}, X = x, Y=y$. We would prefer to keep the notation as is because it explicitly notes the (counterfactual) distribution used in the expectation. However, to avoid misunderstandings, we will describe our notation in the revised paper.\n\n**[W2]** In the notation we use, following Peters et al. as noted above, the definition by Richens et al. is as follows: $$h_{\\lambda}(x, y, \\hat{y}) = E\\_{\\hat{Y} \\sim P^{\\mathcal{M} | \\hat{Y} = \\hat{y}, X = x, Y=y}(\\hat{Y})}[ \\max\\\\{0, \\Bbb{1}\\\\{\\hat{Y} = y\\\\} - \\Bbb{1}\\\\{\\hat{y} = y\\\\}\\\\} ],$$ where $x, y, \\hat{y} \\sim P^{\\mathcal{M} ; do(\\Lambda=\\lambda)}$ and we consider $\\lambda=1$ to be the default action (in Richen's language). In contrast, our definition is as follows: $$h_{\\lambda}(x, y, \\hat{y}) = E_{\\hat{Y} \\sim P^{\\mathcal{M} ; do(\\Lambda = \\lambda) | \\hat{Y} = \\hat{y}, X = x, Y=y}(\\hat{Y})}[ \\max\\\\{0, \\Bbb{1}\\\\{\\hat{y} = y\\\\}  - \\Bbb{1}\\\\{\\hat{Y} = y\\\\} \\\\} ],$$ where $x, y, \\hat{y} \\sim P^{\\mathcal{M}}$. Under Richen's definition, counterfactual harm depends on an observation $x, y, \\hat{y} \\sim P^{\\mathcal{M} ; do(\\Lambda=\\lambda)}$, which comprises a setting in which the human expert makes a prediction from the prediction set $\\mathcal{C}\\_{\\lambda}(x)$ (i.e, the decision support system $\\mathcal{C}_{\\lambda}$ is deployed). In contrast, under our definition, counterfactual harm depends on an observation $x, y, \\hat{y} \\sim P^{\\mathcal{M}}$, which comprises a setting in which the human expert makes a prediction on their own.\n\nIn the notation suggested by the reviewer, the definition by Richens et al. is as follows: $$h_{\\lambda}(x, y, \\hat{y}) = \\mathbb{E}[\\max\\\\{0, \\Bbb{1}\\\\{\\hat{Y}\\_{1} = y \\\\} - \\Bbb{1}\\\\{\\hat{y} = y \\\\} \\\\} | X=x, Y=y, \\hat{Y}\\_{\\lambda}=\\hat{y}],$$ where $x, y, \\hat{y} \\sim P(X, Y, \\hat{Y}\\_{\\lambda})$ and the expectation is over the prediction $\\hat{Y}\\_{1}$ made by the expert on their own and we consider $\\lambda=1$ to be the default action. In contrast, our definition is as follows: $$h_{\\lambda}(x, y, \\hat{y}) = \\mathbb{E}[\\max\\\\{0, \\Bbb{1}\\\\{\\hat{y} = y \\\\} - \\Bbb{1}\\\\{\\hat{Y}\\_{\\lambda} = y\\\\} \\\\} | X=x, Y=y, \\hat{Y}\\_{1}=\\hat{y}],$$ where $x, y, \\hat{y} \\sim P(X, Y, \\hat{Y}\\_{1})$ and the expectation is over the prediction $\\hat{Y}\\_{\\lambda}$ made by the expert using the system $\\mathcal{C}\\_{\\lambda}$. \n\nFollowing the reviewer's advice, we will clarify this in a separate Appendix.\n\n**[W3]** As noted above, we follow standard notation in causality research, which we would respectfully prefer to keep, though, we will explicitly clarify that the mechanisms are not equations that can be manipulated in the revised paper.\n\n**[Q1]** To avoid misunderstandings, we will rephrase lines 147-148 to clarify that we cannot expect to find  $\\Lambda (\\alpha)$ from observational data as counterfactual harm is not identifiable from observational data without any further assumptions. However, we would like to clarify that, since $\\hat{Y}$ and $\\hat{Y}_{\\lambda}$ are the same variable under no intervention $(\\Lambda=1)$ and under the intervention $do(\\Lambda=\\lambda)$, it is not possible to simultaneously measure them, therefore it is not sensible to consider their joint distribution as argued in Pearl, J. (2010). An introduction to causal inference. The international journal of biostatistics, 6(2) (page 25).\n\n**[Q2]** We would like to clarify that our goal is not to minimize counterfactual harm but rather to find the largest harm-controlling set $\\Lambda(\\alpha)$ for a user-specified harm bound $\\alpha$, as described in lines 144-150. Among the $\\lambda$ values in  $\\Lambda(\\alpha)$, one could later on find the $\\lambda$ value that maximizes some utility, e.g., accuracy by using the methods in [17,18], or the counterfactual benefit, though this remains outside the scope of our work.\n\nRemoving the max from our definition of counterfactual harm would result in measuring the accuracy of the human expert under no interventions minus the accuracy of their counterfactual predictions had they used the system. That said, removing the max would not solve the identifiability issue, as this is due to the counterfactual predictions in the definition.\n\nTo the best of our knowledge, we have found no prior work on conformal risk control using $\\lambda = \\lambda(x)$ to construct the prediction sets, however, a recent work by Gibbs et al. (Conformal Prediction With Conditional Guarantees, Arxiv 2023) uses threshold values that depend on x in the context of conformal prediction. We will add this in our discussion in Section 7.\n\n**[Q3]** We agree with the reviewer that different incorrect classifications may have different consequences—a cold is much more similar to a flu rather than lung cancer in terms of symptoms and treatment. In that context, we acknowledge that, for simplicity, our definition of counterfactual harm treats all incorrect classifications as equally harmful. Expanding our definition of harm to weigh different incorrect classifications according to their consequences would be a very interesting avenue for future work, however, it may be challenging to quantify/measure the consequences. We will discuss this in Section 7 under Assumptions.\n\n**[Q4]** We will expand the discussion in lines 127-130 to include the suggested sentence.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their careful and insightful comments, which will help improve our paper. Please, find a point-by-point response below and a one page pdf with additional results attached. In the revised version of the paper, we will fix all typos pointed out by the reviewers.

Meta-review: All four reviewers were on the side of acceptance before the author rebuttal and discussion. However, one reviewer became more negative over the course of a long discussion with the authors and myself, although their score remains at borderline accept. I think that the reason for this more negative sentiment, while valid, does not outweigh the strengths of the submission, as I discuss below.

Reviewers found strengths in all aspects of the work. At the conceptual level, reviewers appreciated the formal definition of counterfactual harm caused by a prediction set. The identification of monotonicity assumptions (both a stronger and a weaker one) needed to estimate counterfactual harm is a strength in terms of theory. A method of controlling counterfactual harm is proposed, and the experiments with human subjects were also seen as a strength. Lastly, reviewers found the submission to be clearly written despite the technicality.

The rebuttal satisfactorily addressed several issues in my view, including practical applications of prediction sets, the "quality" of $\lambda$'s determined to be harm-controlling (a limitation of conformal risk control alleviated by a larger calibration set), models having worse performance than humans, confusion over causal inference notation, and an unclear relationship with the work of Richens et al. Clarifications and improvements to the paper are within the scope of a camera-ready revision.

Reviewer 8Whm engaged in a long discussion with the authors and later with myself. While part of this discussion was on more technical points, the most serious concern for this reviewer in the end was the question of comparing with additional state-of-the-art conformal predictors, in terms of counterfactual harm, coverage (conformal accuracy), and prediction set size. I think the discussion clarified somewhat that conformal predictors and the proposed method play complementary roles, where the latter provides more refined estimates of counterfactual harm for the former. Nonetheless, the reviewer wanted to know how the proposed method of choosing $\lambda$ affects conformal accuracy and prediction set size, for additional conformal predictors. I think this is a valid request. However, I do not see it as a core concern for the human-AI setting considered in the paper, where the prediction set is provided as an intermediate quantity for the human to select from, not as a final output, and human accuracy and counterfactual harm appear to be more relevant metrics. Thus, I do not think this concern should hold the submission back.  I do encourage the authors to follow through on their promise of experimenting with additional conformal predictors and to report all metrics, including conformal accuracy and prediction set size.
Decision: Accept (poster)