Title: Controlling Counterfactual Harm in Decision Support Systems Based on Prediction Sets
Venue: NeurIPS 2024
Accepted: yes
Keywords: Counterfactual Harm

Prediction Sets

Decision Support Systems
Abstract: Decision support systems based on prediction sets help humans solve multiclass classification tasks by narrowing down the set of potential label values to a subset of them, namely a prediction set, and asking them to always predict label values from the prediction sets. While this type of systems have been proven to be effective at improving the average accuracy of the predictions made by humans, by restricting human agency, they may cause harm---a human who has succeeded at predicting the ground-truth label of an instance on their own may have failed had they used these systems. In this paper, our goal is to control how frequently a decision support system based on prediction sets may cause harm, by design. To this end, we start by characterizing the above notion of harm using the theoretical framework of structural causal models. Then, we show that, under a natural, albeit unverifiable, monotonicity assumption, we can estimate how frequently a system may cause harm using only predictions made by humans on their own. Further, we also show that, under a weaker monotonicity assumption, which can be verified experimentally, we can bound how frequently a system may cause harm again using only predictions made by humans on their own. Building upon these assumptions, we introduce a computational framework to design decision support systems based on prediction sets that are guaranteed to cause harm less frequently than a user-specified value 
using conformal risk control. We validate our framework using real human predictions from two different human subject studies and show that, in decision support systems based on prediction sets, there is a trade-off between accuracy and counterfactual harm.

Controlling Counterfactual Harm in Decision Support Systems Based on Prediction Sets

Abstract

Decision support systems based on prediction sets help humans solve multiclass classification tasks by narrowing down the set of potential label values to a subset of them, namely a prediction set, and asking them to always predict label values from the prediction sets. While this type of systems have been proven to be effective at improving the average accuracy of the predictions made by humans, by restricting human agency, they may cause harm-a human who has succeeded at predicting the ground-truth label of an instance on their own may have failed had they used these systems. In this paper, our goal is to control how frequently a decision support system based on prediction sets may cause harm, by design. To this end, we start by characterizing the above notion of harm using the theoretical framework of structural causal models. Then, we show that, under a natural, albeit unverifiable, monotonicity assumption, we can estimate how frequently a system may cause harm using only predictions made by humans on their own. Further, we also show that, under a weaker monotonicity assumption, which can be verified experimentally, we can bound how frequently a system may cause harm again using only predictions made by humans on their own. Building upon these assumptions, we introduce a computational framework to design decision support systems based on prediction sets that are guaranteed to cause harm less frequently than a userspecified value using conformal risk control. We validate our framework using real human predictions from two different human subject studies and show that, in decision support systems based on prediction sets, there is a trade-off between accuracy and counterfactual harm.

1 Introduction

The principle of 'first, do no harm' holds profound significance in a variety of professions across multiple high-stakes domains. For example, in the field of medicine, doctors swear an oath to prioritize their patient's well-being or, in the legal justice system, preserving the innocence of individuals is paramount. As a result, in all of these domains, rules and guidelines have been established to prevent decision makers-doctors or judges-from making decisions that harm individuals-patients or suspects. In recent years, it has been increasingly argued that a similar principle should apply to decision support systems using machine learning algorithms in high-stakes domains [1-4]. 1

The definition of harm is not unequivocally agreed upon, however, the most widely accepted definition is the counterfactual comparative account of harm (in short, counterfactual harm) [5-7], which we adopt in our work. Under this definition, an action causes harm to an individual if they would have

Manuel Gomez Rodriguez

been in a worse state had the action been taken. Building upon this definition, we say that a decision support system causes harm to an individual if a decision maker would have made a worse decision about the individual had they used the system.

In machine learning for decision support, one of the main focus has been classification tasks. Here, the most studied setting assumes the decision support system uses a classifier to predict the value of a (ground-truth) label of interest and a human expert uses the predicted value to update their own prediction [8-12]. Unfortunately, in this setting, it is yet unclear how to guarantee that the (average) accuracy of the predictions made by an expert who uses the system is higher than the accuracy of the predictions made by the expert and the classifier on their own, what is often referred to as human-AI complementarity [13-16]. In this context, a recent line of work [17, 18] have argued, both theoretically and empirically, that an alternative setting may enable human-AI complementarity. In this alternative setting, the decision support system helps a human expert by providing a set of label predictions, namely a prediction set, and asking them to always predict a label value from the set. The key principle is that, by restricting human agency, good performance does not depend on the expert developing a good sense of when to predict a label from the prediction set. In this context, it is also worth noting that Google has recently developed a tool that uses patient history and skin condition images to provide decision support using prediction sets [19], and a study by Jain et al. [20] has found that physicians and nurses using this tool improved diagnoses for 1 in every 8 to 10 cases.

In this work, we argue that the same principle that enables human-AI complementarity on decision support systems based on prediction sets may also cause counterfactual harm-a human expert who has succeeded at predicting the label of an instance on their own may have failed had they used these systems. Consequently, our goal is to design decision support systems based on prediction sets that are guaranteed to cause, in average, less counterfactual harm than a user-specified value.

Our contributions. We start by formally characterizing the predictions made by a decision maker using a decision support system based on prediction sets using a structural causal model (SCM) and, based on this characterization, formalize our notion of counterfactual harm. In general, since counterfactual harm lies within level three in the 'ladder of causation' [21], it is not (partially) identifiable-it cannot be estimated (bounded) from data. However, we show that, under a natural counterfactual monotonicity assumption on the predictions made by decision makers using decision support systems based on prediction sets, counterfactual harm is identifiable. Further, we show that, under a weaker interventional monotonicity assumption, which can be verified experimentally, the average counterfactual harm is partially identifiable. Then, building upon these assumptions, we develop a computational framework to design decision support systems based on prediction sets that are guaranteed to cause, in average, less counterfactual harm than a user-specified value using conformal risk control [22]. Finally, we validate our framework using real human predictions from two different human subject studies and show that, in decision support systems based on prediction sets, there is a trade-off between accuracy and counterfactual harm.

Further related work. Our work builds upon further related work on set-valued predictors, critiques of prediction optimization, counterfactual harm and algorithmic triage.

Set-valued predictors output a set of label values, namely a prediction set, rather than single labels [23]. However, set-valued predictors have not been typically designed nor evaluated by their ability to help human experts make more accurate predictions [24-29]. Only very recently, an emerging line of work has shown that conformal predictors, a specific type of set-valued predictors, may help human experts make more accurate predictions [17, 18, 30-33]. Within this line of work, the work most closely related to ours is by Straitouri et al. [17, 18], which has introduced the setting, and counterfactual and interventional monotonicity properties we build upon.

Prediction optimization has been recently put into question in the context of decision support [34-36]. More specifically, it has been argued that optimizing decision support systems to improve prediction accuracy does not always translate to better decision-making. Our work aligns with this critique since we argue that improving prediction accuracy may come at the cost of counterfactual harm.

The literature on counterfactual harm in machine learning is still quite small and has focused on traditional machine learning settings in which machine learning models replace human decision makers and make automated decisions [1-4]. Within this literature, the work most closely related to ours is by Richens et al. [1], which also uses a structural causal model to define counterfactual harm. However, their definition of counterfactual harm differs in a subtle, but important, way from ours. In

our setting, their definition of counterfactual harm would compare the accuracy of a factual prediction made by an expert using the decision support system against the counterfactual prediction that the same expert would have made on their own. That means, under their definition, one would need to deploy the system to estimate the harm it may cause, potentially causing harm. In contrast, under our definition, one does not need to deploy the system to estimate (or bound) the harm it may cause, as it will become clear in Section 4, and thus we argue that our definition may be more practical.

Learning under algorithmic triage seeks to develop classifiers that make predictions for a given fraction of the samples and leave the remaining ones to human experts, as instructed by a triage policy [37-42]. In contrast, in our work, for each sample, a classifier is used to construct a prediction set and a human expert needs to predict a label value from the set. In this context, it is also worth noting that learning under algorithmic triage has been extended to reinforcement learning settings [43-46].

2 Decision support systems based on prediction sets

We consider a multiclass classification task in which, for each task instance, a human expert has to predict the value of a ground-truth label y ∈ Y = { 1 , . . . , L } . Then, our goal is to design a decision support system C : X → 2 Y that, given a set of features x ∈ X , helps the expert by narrowing down the label values to a subset of them C ( x ) ⊆ Y , namely a prediction set. Here, we focus on a setting in which the system asks the expert to always predict a label value ˆ y from the prediction set C ( x ) . Note that, by restricting the expert's agency, good performance does not depend on the human expert developing a good sense of when to predict a label from the prediction set. Moreover, we assume that the set of features, the ground-truth label and the expert's prediction are sampled from an unknown fixed distribution 2 , i.e. , x, y ∼ P ( X,Y ) and ˆ y ∼ P ( ˆ Y | X,Y, C ( X )) .

Further, similarly as in Straitouri et al. [17, 18], we consider that, given a set of features x ∈ X , the system constructs the prediction set C ( x ) using the following set-valued predictor [23]. First, the set-valued predictor ranks each potential label value y ∈ Y using the softmax output of a pre-trained classifier m y ( x ) ∈ [0 , 1] . Then, given a user-specified threshold λ ∈ [0 , 1] , it uses the resulting ranking to construct the prediction set C ( x ) = C λ ( x ) as follows:

where · ( i ) denotes the i -th label value in the ranking. Here, note that, for λ = 0 , the prediction set contains just the top ranked label value, for λ = 1 , it contains all label values. 3

Given the above parameterization, one may just focus on finding the optimal threshold λ ∗ under which the human expert achieves the highest average accuracy as in Straitouri et al. [17, 18], i.e. ,

However, this focus does not prevent the resulting system C λ from causing harm-an expert may succeed to predict the value of the ground-truth label on their own on instances in which they would have failed had they used C λ . In this work, our goal is to design a computational framework that, given a user-specified bound α ∈ [0 , 1] , finds the set of λ values which are all guaranteed to cause less harm, in average, than the bound α .

Remark. We would like to clarify that, if one sets the value of the threshold λ to be roughly the 1 -α quantile of the empirical distribution of the scores 1 -m y ( x ) in a calibration set, then, the set valued predictor defined by Eq. 1 is equivalent to a vanilla conformal predictor with nonconformity scores 1 -m y ( x ) and coverage 1 -α . Under this view, it becomes apparent that, by searching for λ values in [0 , 1] that are harm controlling, we are essentially searching for vanilla conformal predictors that are harm controlling. In this context, we would like to further clarify that our framework is agnostic to the choice of nonconformity score or more generally, the set-valued predictor, used to construct the prediction sets [47-49]. Motivated by this observation, in Appendix E, we include additional experiments where we evaluate our framework using a more complex set-valued predictor [49].

X

Set of

Features

Y

Ground

Truth Label

V

Λ

U

Exogenous

Variables

𝒞

!

(X)

Prediction

Set

Y

*

Expert

Prediction

Edema

Fibrosis

Edema

Effusion

Edema

3 Counterfactual harm of decision support systems

To formalize our notion of harm, we characterize how human experts make predictions using a decision support system C via a structural causal model (SCM) [21], which we denote as M . More specifically, similarly as in Straitouri et al. [18], we define M by the following set of assignments:

where Λ , U and V are exogenous random variables and f X , f Y , f C and f ˆ Y are given functions. 4 The exogenous variables Λ , U and V characterize the user-specified threshold, the expert's individual characteristics and the data generating process, respectively. The function f C is directly defined by Eq. 1, i.e. , f C (Λ , X ) = C Λ ( X ) . Further, as argued elsewhere [21], we can always find a distribution for the exogenous variables Λ , U and V as well as a functional form for the functions f X , f Y and f ˆ Y such that the distributions of the features, the ground-truth label and the expert's prediction introduced in Section 2 are given by the observational distribution entailed by the SCM M . For ease of exposition, we assume that, under no interventions, the distribution of the exogenous variable Λ is P (Λ) = ✶ { Λ = 1 } and thus human experts make predictions on their own. Figure 1 shows a visual representation of our SCM M .

Building upon the above characterization, we are now ready to formalize the following notion of counterfactual harm, which essentially compares the accuracy of a factual prediction made by an expert on their own against the counterfactual prediction that the same expert would have made had they used a decision support system C λ :

Definition 1 (Counterfactual Harm) For any x, y, ˆ y ∼ P M , the counterfactual harm that a decision support system C λ would have caused, if deployed, is given by 5

h λ ( x, y, ˆ y ) = E ˆ Y ∼ P M ; do (Λ= λ ) | ˆ Y =ˆ y,X = x,Y = y ( ˆ Y ) [max { 0 , ✶ { ˆ y = y } -✶ { ˆ Y = y }} ] , (4) where do (Λ = λ ) denotes a (hard) intervention on the exogenous variable Λ .

Here, note that counterfactual harm can only be nonzero if the expert has made a successful prediction on their own, i.e. , ˆ y = y . Otherwise, the expert's prediction ˆ y could not have become worse had they used the decision support system C λ .

Given the above definition of counterfactual harm and a user-specified bound α ∈ [0 , 1] , our goal is to find the largest harm-controlling set of values Λ( α ) ⊆ [0 , 1] such that, for each λ ∈ Λ( α ) , it holds that the counterfactual harm is, in expectation across all possible instances, smaller than α , i.e. ,

At this point, we cannot expect to find the set Λ( α ) because counterfactual harm lies within level three in the 'ladder of causation' [21] and thus it is not identifiable from observational data without further assumptions. However, in what follows, we will show that, under certain assumptions, the average counterfactual harm H ( λ ) is (partially) identifiable, i.e. , it can be estimated (bounded) using observational data.

Comparison to Richens's definition of counterfactual harm. Richens et al. [1] define counterfactual harm as follows:

h λ ( x, y, ˆ y ) = E ˆ Y ∼ P M| ˆ Y =ˆ y,X = x,Y = y ( ˆ Y ) [max { 0 , ✶ { ˆ Y = y } -✶ { ˆ y = y }} ] , where x, y, ˆ y ∼ P M ; do (Λ= λ ) and λ = 1 is considered to be the default action in the language of Richens et al. [1]. This definition implicitly assumes that the system C λ is deployed and it compares the factual prediction ˆ y made by an expert using the deployed system against the counterfactual prediction ˆ Y had the expert made on their own. On the contrary, our definition does not assume that the system C λ is deployed and instead it compares the factual prediction ˆ y made by an expert on their own against the counterfactual prediction ˆ Y had the expert made using the system C λ .

4 Counterfactual harm under counterfactual and interventional monotonicity

In this section, we analyze counterfactual harm h λ ( x, y, ˆ y ) , as defined in Eq. 4, from the perspective of causal identifiability under two natural monotonicity assumptions-counterfactual monotonicity and interventional monotonicity. Both of these assumptions, which were first studied by Straitouri et al. [18], formalize the intuition that increasing the number of label values in a prediction set increases its difficulty.

Under counterfactual monotonicity, for any x ∈ X and λ, λ ′ ∈ [0 , 1] such that Y ∈ C λ ( x ) ⊆ C λ ′ ( x ) , if an expert has succeeded at predicting Y using C λ ′ , they would have also succeeded had they used C λ and, conversely, if they have failed at predicting Y using C λ , they would have also failed had they used C λ ′ , while holding 'everything else fixed'. More formally, counterfactual monotonicity is defined as follows:

Assumption 1 (Counterfactual Monotonicity) Counterfactual monotonicity holds if and only if, for any x ∈ X and any λ, λ ′ ∈ [0 , 1] such that Y ∈ C λ ( x ) ⊆ C λ ′ ( x ) , we have that

Under interventional monotonicity, for any x ∈ X and λ, λ ′ ∈ [0 , 1] such that Y ∈ C λ ( x ) ⊆ C λ ′ ( x ) , the probability that experts succeed at predicting Y using C λ is equal or greater than using C λ ′ . More formally, interventional monotonicity is defined as follows 6 :

Assumption 2 (Interventional Monotonicity) Interventional monotonicity holds if and only if, for any x ∈ X , y ∈ Y , and λ, λ ′ ∈ [0 , 1] such that y ∈ C λ ( x ) ⊆ C λ ′ ( x ) , we have that

where the probability is over the exogenous random variables U and V characterizing the expert's individual characteristics and the data generating process, respectively.

In what follows, we first show that, under the counterfactual monotonicity assumption, counterfactual harm is identifiable (we provide all proofs in Appendix B):

Proposition 1 Under the counterfactual monotonicity assumption, for any x, y, ˆ y ∼ P M , the counterfactual harm that a decision support system C λ would have caused, if deployed, is given by h λ ( x, y, ˆ y ) = ✶ { ˆ y = y ∧ y / ∈ C λ ( x ) } . (8) As an immediate consequence, we can conclude that, under counterfactual monotonicity, the average counterfactual harm H ( λ ) , as defined in Eq. 5, is identifiable and can be estimated using observational data sampled from P M . However, since the inequality condition of the counterfactual monotonicity assumption compares counterfactual predictions and thus cannot be experimentally verified, one should be cautious about using the above proposition to estimate counterfactual harm in high-stakes applications.

Next, we show that, under the interventional monotonicity assumption, the average counterfactual harm is partially identifiable:

Proposition 2 Under the interventional monotonicity assumption, the average counterfactual harm H ( λ ) that a decision support system C λ would have caused, if deployed, satisfies that

̸

✶ ✶ Importantly, note that, in the above proposition, the lower bound on the left hand side of Eq. 9 matches the average counterfactual harm under the counterfactual monotonicity assumption and thus, holding 'everything else fixed', the average counterfactual harm under interventional monotonicity is always greater or equal than the average counterfactual harm under counterfactual monotonicity. Moreover, further note that the inequality condition of the interventional monotonicity assumption compares interventional probabilities and thus we can experimentally verify it (see Appendix F), lending support to using the above proposition to bound average counterfactual harm in high-stakes applications.

5 Controlling counterfactual harm using conformal risk control

In this section, we develop a computational framework that, given a decision support system C λ and a user-specified bound α , aims to find the largest harm-controlling set Λ( α ) , as defined in Eq. 5. In the development of our framework, we will first assume that counterfactual monotonicity holds and, later on, we will relax this assumption, and assume instead that interventional monotonicity holds.

Our framework builds upon the the idea of conformal risk control, which has been introduced very recently by Angelopoulos et al. [22]. Given any monotone loss function ℓ ( C λ ( X ) , Y ) with respect to λ and a calibration set { ( X i , Y i ) } n i =1 , with ( X i , Y i ) ∼ P ( X,Y ) , conformal risk control finds a value of λ under which the expected loss of a test sample ( X n +1 , Y n +1 ) ∼ P ( X,Y ) does not exceed a user-specified bound α , i.e. , E [ ℓ ( C λ ( X n +1 ) , Y n +1 )] ≤ α . However, in our framework, we re-define the loss ℓ so that it does not only depend on the prediction set and the label value but also on the expert's prediction on their own.

Under the counterfactual monotonicity assumption, we set the value of the loss ℓ using the expression of counterfactual harm in Eq. 8 and, using a similar proof technique as in Angelopoulous et al., first prove the following theorem:

Theorem 1 Let D = { ( X i , Y i , ˆ Y i ) } n i =1 be a calibration set, with ( X i , Y i , ˆ Y i ) ∼ P M ( X,Y, ˆ Y ) , α ∈ [0 , 1] be a user-specified bound, and

If counterfactual monotonicity holds, a test sample ( X n +1 , Y n +1 , ˆ Y n +1 ) ∼ P M ( X,Y, ˆ Y ) satisfies that

✶ where the expectation is over the randomness in the calibration set used to compute the threshold ˆ λ ( α ) and the test sample.

Then, we leverage the above theorem and the fact that, under counterfactual monotonicity, the counterfactual harm is nonincreasing with respect to λ , as shown in Lemma 3 in Appendix B.5, to recover the largest harm-controlling set Λ( α ) :

Corollary 1 Let α ∈ [0 , 1] be a user-specified bound. Then, under the counterfactual monotonicity assumption, it holds that Λ( α ) = { λ ∈ [0 , 1] | λ ≥ ˆ λ ( α ) } , where ˆ λ ( α ) is given by Eq. 10.

Under the interventional monotonicity assumption, rather than directly controlling the counterfactual harm, we will control the upper bound given by Eq. 9. Consequently, rather than recovering the largest harm-controlling set Λ( α ) , we will recover a harm-controlling set Λ ′ ( α ) ⊆ Λ( α ) . However, since the expression inside the expectation of the upper bound is nonmonotone with respect to λ , we cannot directly use it to set the value of the loss ℓ in conformal risk control. That said, since the first term of the expression is nonincreasing and the second term is nondecreasing, as shown in Lemmas 3 and 4 in Appendix B.5, we can apply conformal risk control separately for each term. For the first term, we use Theorem 1 because the term matches the counterfactual harm under the counterfactual monotonicity assumption. For the second term, we prove the following theorem:

Theorem 2 Let D = { ( X i , Y i , ˆ Y i ) } n i =1 be a calibration set, with ( X i , Y i , ˆ Y i ) ∼ P M , α ∈ [ 1 n +1 , 1 ] be a user-specified bound, and

̸

If interventional monotonicity holds and ˇ λ exists, a test sample ( X n +1 , Y n +1 , ˆ Y n +1 ) ∼ P M ( X,Y, ˆ Y ) satisfies that

̸

✶ where the expectation is over the randomness in the calibration set used to compute the threshold ˇ λ ( α ) and the test sample ( X n +1 , Y n +1 , ˆ Y n +1 ) .

Finally, we leverage the above theorems and the fact that the first term inside the expectation of the upper bound of counterfactual harm in Eq. 9 is nonincreasing and the second term is nondecreasing with respect to λ to recover a harm-controlling set Λ ′ ( α ) ⊆ Λ( α ) :

Corollary 2 Let α ∈ [0 , 1] be a user-specified bound. Then, under the interventional monotonicity assumption, for any choice of α ′ ≤ α , the set

where ˆ λ ( α ′ ) is given by Eq. 10 and ˇ λ ( α -α ′ ) is given by Eq. 11 satisfies that Λ ′ ( α ) ⊆ Λ( α ) .

Note that, in practice, the above corollaries can be implemented efficiently since, by definition, the functions ˆ H n ( λ ) and ˆ G n ( λ ) are piecewise constant functions of λ with n different pieces.

6 Experiments

In this section, we use data from two different human subject studies to: a) evaluate the average counterfactual harm caused by decision support systems based on prediction sets; b) validate the theoretical guarantees offered by our computational framework ( i.e. , Corollaries 1 and 2); c) investigate the trade-off between the average counterfactual harm caused by decision support systems based on prediction sets and the average accuracy achieved by human experts using these systems. In what follows, we assume that counterfactual monotonicity holds. In Appendix G, we conduct experiments where we relax this assumption and assume instead that interventional monotonicity holds. 7

Experimental setup. We first experiment with the ImageNet16H dataset by Steyvers et al. [52], which comprises 32 , 431 predictions made by 145 human participants on their own about noisy

0

.

8

0

.

9

0.01

VGG19

0

.

8

0

.

9

DenseNet161

0

.

75

0

.

85

GoogleNet

0

.

75

0

.

85

ResNet152

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

7

0

.

8

AlexNet

Average Accuracy

Average Counterfactual Harm

(a)

α

= 0

.

01

0

.

8

0

.

9

0.05

VGG19

0

.

8

0

.

9

DenseNet161

0

.

75

0

.

85

GoogleNet

0

.

75

0

.

85

ResNet152

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

7

0

.

8

AlexNet

Average Accuracy

Average Counterfactual Harm

(b)

α

= 0

.

05

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

images created using 1 , 200 unique natural images from the ImageNet Large Scale Visual Recognition Challenge (ILSRVR) 2012 dataset [53]. More specifically, each of the natural images was used to generate four noisy images with different amount of phase noise distortion ω ∈ { 80 , 95 , 110 , 125 } and with the same ground-truth label y from a label set Y of size L = 16 . Here, the amount of phase noise controls the difficulty of the classification task-the higher the noise, the more difficult the classification task. In our experiments, we use (all) the noisy images with noise value ω ∈ { 80 , 95 , 110 } because, for the noisy images with ω = 125 , humans perform poorly; moreover, we stratify these images (and human predictions) with respect to their amount of phase noise.

For each stratum of images, we apply our framework to decision support systems C λ with different pre-trained classifiers, namely VGG19 [54], DenseNet161 [55], GoogleNet [56], ResNet152 [57] and AlexNet [58] after 10 epochs of fine-tuning, as provided by Steyvers et al. [52]. 8 To this end, we randomly split the images (and human predictions) into a calibration set ( 10 %), which we use to find the harm-controlling sets Λ( α ) by applying Corollary 1, and a test set ( 90 %), which we use to estimate the average counterfactual harm H ( λ ) caused by the decision support systems C λ as well as the average accuracy A ( λ ) of the predictions made by a human expert using C λ . Here, since the dataset only contains predictions made by human participants on their own, we use the mixture of multinomial logit models (MNLs) introduced by Straitouri et al. [17] to estimate the average accuracy A ( λ ) . Refer to Appendix C for more details regarding the mixture of MNLs and to Appendix D for additional experiments studying the relationship between average counterfactual harm H ( λ ) , average prediction set size and empirical coverage 9 .

Then, we experiment with the ImageNet16H-PS dataset 10 by Straitouri et al. [18], which comprises 194 , 407 predictions made by 2 , 751 human participants using decision support systems C λ about the

Real

Predicted

0

.

00

0

.

02

0

.

04

0

.

06

0

.

08

Average Counterfactual Harm

0

.

80

0

.

85

0

.

90

Average Accuracy

0.01

(a)

α

= 0

.

01

0

.

00

0

.

02

0

.

04

0

.

06

0

.

08

Average Counterfactual Harm

0

.

80

0

.

85

0

.

90

Average Accuracy

0.05

(b)

α

= 0

.

05

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

set of noisy images with ω = 110 described above. More specifically, for each noisy image, the dataset contains human predictions made under any possible prediction set that can be constructed using Eq. 1 with (the softmax output of) VGG19 after 10 epochs of fine-tuning, as provided by Steyvers et al. [52]. Here, similarly as in the ImageNet16H dataset, we randomly split the images (and human predictions) into a calibration set ( 10 %), which we use to find Λ( α ) , and a test set ( 90 %), which we use to estimate H ( λ ) and A ( λ ) . However, in this case, we can estimate A ( λ ) using the predictions made by human participants using C λ from the dataset, and we can compare this empirical estimate to the one using the mixture of MNLs.

In both datasets, we calculate confidence intervals and validate the theoretical guarantees offered by Corollary 1 by repeating each experiment 50 times and, each time, sampling different calibration and test sets.

Results. Figure 2 shows the average accuracy A ( λ ) achieved by a human participant using C λ , as predicted by the mixture of MNLs, against the average counterfactual harm H ( λ ) caused by C λ for α ∈ { 0 . 01 , 0 . 05 } on the stratum of images with ω = 110 from the ImageNet16H dataset. Refer to Appendix D for results on other strata. Here, each point corresponds to a different λ value and its coloring indicates the empirical probability that λ is included in the harm-controlling set Λ( α ) . The results show several interesting insights. First, we find that, as long as λ < 1 , the decision support systems C λ always cause some amount of counterfactual harm 11 . This suggests that, while restricting human agency enables human-AI complementarity, it inevitably causes (some amount of) counterfactual harm. Second, we find that the sets of λ values provided by our framework are typically harm-controlling, i.e. , they do not include λ values such that H ( λ ) > α . However, the sets are often conservative and do not include all the harm-controlling λ values due to estimation error in the the empirical estimate ˆ H n ( λ ) of the average counterfactual harm using data from the calibration set. In Appendix D, we show that using larger calibration sets reduces the above mentioned estimation error and results in sets of λ values that are less conservative. Third, we find that there is a trade-off between accuracy and counterfactual harm and this trade-off is qualitatively consistent across decision support systems using different pre-trained classifiers. In fact, for α = 0 . 01 , the decision support systems C λ ∗ offering the greatest average accuracy A ( λ ∗ ) are not harm-controlling.

Figure 3 shows the average accuracy A ( λ ) achieved by a human participant using C λ , as estimated using predictions made by human participants using C λ , against the average counterfactual harm

H ( λ ) caused by C λ for α ∈ { 0 . 01 , 0 . 05 } on the ImageNet16H-PS dataset. Here, the meaning and coloring of each point is the same as in Figure 2. The results also support the findings derived from the experiments with the ImageNet16H dataset-the decision support systems C λ always cause some amount of harm, our framework succeeds at identifying harm-controlling sets, and there is a trade-off between accuracy and counterfactual harm. Further, they also show that, while there is a gap between the average accuracy estimated using the mixture of MNLs and the average accuracy estimated using predictions made by human participants using C λ , they follow the same trend and support the same qualitative conclusions.

7 Discussion and limitations

In this section, we highlight several limitations of our work, discuss its broader impact, and propose avenues for future work.

Assumptions. We have assumed that the data samples and the expert predictions are drawn i.i.d. from a fixed distribution and the calibration set contains samples with noiseless ground truth labels and expert label predictions. It would be very interesting to extend our framework to allow for distribution shifts and label noise. Moreover, we have considered prediction sets constructed with a fixed user-specified threshold value λ . In light of recent work by Gibbs et al. [59], it would be interesting to extend our framework to allow for threshold values that depend on the data samples. Furthermore, in our definition of counterfactual harm we have treated all inaccurate predictions as equally harmful. Expanding our definition of harm to weigh different inaccurate predictions according to their consequences would be a very interesting avenue for future work. In addition, under the interventional monotonicity assumption, we have empirically observed that the gap between our lower- and upper-bounds is often large. Therefore, it would be useful to identify other natural assumptions under which this gap is smaller.

Methodology. In our framework, the prediction sets are constructed using the softmax output of a pretrained classifier. However, we hypothesize that, by accounting for the similarity between the mistakes made by humans and those made by the pre-trained classifier, we may be able to construct prediction sets that cause counterfactual harm less frequently. Moreover, we have focused on controlling the average counterfactual harm. However, whenever the expert's predictions are consequential to individuals, this may lead significant disparities across demographic groups. Therefore, it would be important to extend our framework to account for fairness considerations.

Evaluation. Our experimental evaluation comprises a single benchmark dataset of noisy natural images and thus one may question the generalizability of the conclusions we draw from our results. To overcome this limitation, it would be important to further investigate the trade-off between accuracy and counterfactual harm in decision support systems based on prediction sets in real-world application domains ( e.g. , medical diagnosis).

8 Conclusions

In this paper, we have initiated the study of counterfactual harm in decision support systems based on prediction sets. We have introduced a computational framework that, under natural monotonicity assumptions on the predictions made by experts using these systems, can control how frequently these systems cause harm. Moreover, we have validated our framework using data from two different human subject studies and shown that, in decision support systems based on prediction sets, there is a trade-off between accuracy and counterfactual harm.

Acknowledgments and Disclosure of Funding

Gomez-Rodriguez acknowledges support from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No. 945719).

References

Jonathan Richens, Rory Beard, and Daniel H Thompson. Counterfactual harm. In Advances in Neural Information Processing Systems , volume 35, pages 36350-36365, 2022.

Sander Beckers, Hana Chockler, and Joseph Halpern. A causal analysis of harm. In Advances in Neural Information Processing Systems , volume 35, pages 2365-2376, 2022.

Haoxuan Li, Chunyuan Zheng, Yixiao Cao, Zhi Geng, Yue Liu, and Peng Wu. Trustworthy policy learning under the counterfactual no-harm criterion. In International Conference on Machine Learning , pages 20575-20598. PMLR, 2023.

Sander Beckers, Hana Chockler, and Joseph Y Halpern. Quantifying harm. In Proceedings of the International Joint Conference on Artificial Intelligence . IJCAI, 2023.

Joel Feinberg. Wrongful life and the counterfactual element in harming. Social Philosophy and Policy , 4(1):145-178, 1986.

Matthew Hanser. The metaphysics of harm. Philosophy and Phenomenological Research , 77(2):421-450, 2008.

Justin Klocksiem. A defense of the counterfactual comparative account of harm. American Philosophical Quarterly , 49(4):285-300, 2012.

Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S. Lasecki, Daniel S. Weld, and Eric Horvitz. Beyond accuracy: The role of mental models in human-ai team performance. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing . AAAI press, 2019.

Brian Lubars and Chenhao Tan. Ask not what ai can do, but what ai should do: Towards a framework of task delegability. In Advances in Neural Information Processing Systems , volume 32, pages 57-67, 2019.

Sebastian Bordt and Ulrike von Luxburg. When humans and machines make joint decisions: A non-symmetric bandit model. arXiv preprint arXiv:2007.04800 , 2020.

Kailas Vodrahalli, Tobias Gerstenberg, and James Y Zou. Uncalibrated models can improve human-ai collaboration. In Advances in Neural Information Processing Systems , volume 35, pages 4004-4016, 2022.

Nina Corvelo Benz and Manuel Rodriguez. Human-aligned calibration for ai-assisted decision making. In Advances in Neural Information Processing Systems , volume 36, 2024.

Ming Yin, Jennifer Wortman Vaughan, and Hanna Wallach. Understanding the effect of accuracy on trust in machine learning models. In Proceedings of the CHI conference on human factors in computing systems , pages 1-12. ACM, 2019.

Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making. In Proceedings of the Conference on Fairness, Accountability, and Transparency , pages 295-305. ACM, 2020.

Harini Suresh, Natalie Lao, and Ilaria Liccardi. Misplaced trust: Measuring the interference of machine learning in human decision-making. In Proceedings of the ACM Conference on Web Science , pages 315-324. ACM, 2020.

Vivian Lai, Chacha Chen, Alison Smith-Renner, Q Vera Liao, and Chenhao Tan. Towards a science of human-ai decision making: An overview of design space in empirical humansubject studies. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency , pages 1369-1385, 2023.

Eleni Straitouri, Lequn Wang, Nastaran Okati, and Manuel Gomez Rodriguez. Improving expert predictions with conformal prediction. In Proceedings of the International Conference on Machine Learning , pages 32633-32653. PMLR, 2023.

Eleni Straitouri and Manuel Gomez-Rodriguez. Designing decision support systems using counterfactual prediction sets. In Proceedings of the International Conference on Machine Learning . PMLR, 2024.

Google Blogpost. Improving access to skin disease information. https://health.google/ health-research/imaging-and-diagnostics/ , 2020.

Ayush Jain, David Way, Vishakha Gupta, Yi Gao, Guilherme de Oliveira Marinho, Jay Hartford, Rory Sayres, Kimberly Kanada, Clara Eng, Kunal Nagpal, et al. Development and assessment of an artificial intelligence-based tool for skin condition diagnosis by primary care physicians and nurse practitioners in teledermatology practices. JAMA network open , 4(4):e217249-e217249, 2021.

Judea Pearl. Causal inference in statistics: An overview. Statistics Surveys , 3:96-146, 01 2009.

Anastasios Nikolas Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. In Proceedings of the International Conference on Learning Representations . ICLR, 2024.

Evgenii Chzhen, Christophe Denis, Mohamed Hebiri, and Titouan Lorieul. Set-valued classification-overview via a unified framework. arXiv preprint arXiv:2102.12318 , 2021.

Gen Yang, Sébastien Destercke, and Marie-Hélène Masson. Cautious classification with nested dichotomies and imprecise probabilities. Soft Computing , 21(24):7447-7462, 2017.

Thomas Mortier, Marek Wydmuch, Krzysztof Dembczy´ nski, Eyke Hüllermeier, and Willem Waegeman. Efficient set-valued prediction in multi-class classification. Data Mining and Knowledge Discovery , 35(4):1435-1469, 2021.

Liyao Ma and Thierry Denoeux. Partial classification in the belief function framework. Knowledge-Based Systems , 214:106742, 2021.

Vu-Linh Nguyen and Eyke Hüllermeier. Multilabel classification with partial abstention: Bayes-optimal prediction under label independence. Journal of Artificial Intelligence Research , 72:613-665, 2021.

Anastasios N Angelopoulos, Stephen Bates, Emmanuel J Candès, Michael I Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. arXiv preprint arXiv:2110.01052 , 2021.

Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael Jordan. Distribution-free, risk-controlling prediction sets. Journal of the ACM , 68(6):1-34, 2021.

Varun Babbar, Umang Bhatt, and Adrian Weller. On the utility of prediction sets in human-ai teams. In Proceedings of the International Joint Conference on Artificial Intelligence , pages 2457-2463. IJCAI, 7 2022.

Jesse C. Cresswell, Yi Sui, Bhargava Kumar, and Noël Vouitsis. Conformal prediction sets improve human decision making. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024.

Dongping Zhang, Angelos Chatzimparmpas, Negar Kamali, and Jessica Hullman. Evaluating the utility of conformal prediction sets for ai-advised image labeling. In Proceedings of the CHI conference on Human Factors in Computing Systems . ACM, 2024.

Giovanni De Toni, Nastaran Okati, Suhas Thejaswi, Eleni Straitouri, and Manuel GomezRodriguez. Towards human-ai complementarity with predictions sets. In Avances on Neural Information Processing Systems , 2024.

Wouter AC van Amsterdam, Nan van Geloven, Jesse H Krijthe, Rajesh Ranganath, and Giovanni Ciná. When accurate prediction models yield harmful self-fulfilling prophecies. arXiv preprint arXiv:2312.01210 , 2023.

Angelina Wang, Sayash Kapoor, Solon Barocas, and Arvind Narayanan. Against predictive optimization: On the legitimacy of decision-making algorithms that optimize predictive accuracy. ACM Journal of Responsible Computing , 1(1), March 2024.

Lydia T Liu, Solon Barocas, Jon Kleinberg, and Karen Levy. On the actionability of outcome prediction. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 22240-22249, 2024.

Maithra Raghu, Katy Blumer, Greg Corrado, Jon Kleinberg, Ziad Obermeyer, and Sendhil Mullainathan. The algorithmic automation problem: Prediction, triage, and human effort. arXiv preprint arXiv:1903.12220 , 2019.

Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning , pages 7076-7087. PMLR, 2020.

Abir De, Nastaran Okati, Ali Zarezade, and Manuel Gomez Rodriguez. Classification under human assistance. Proceedings of the AAAI Conference on Artificial Intelligence , 35(7):59055913, 2021.

Nastaran Okati, Abir De, and Manuel Rodriguez. Differentiable learning under triage. In Advances in Neural Information Processing Systems , volume 34, pages 9140-9151, 2021.

Mohammad-Amin Charusaie, Hussein Mozannar, David Sontag, and Samira Samadi. Sample efficient learning of predictors that complement humans. In International Conference on Machine Learning , pages 2972-3005. PMLR, 2022.

Hussein Mozannar, Hunter Lang, Dennis Wei, Prasanna Sattigeri, Subhro Das, and David Sontag. Who should predict? exact algorithms for learning to defer to humans. In International conference on artificial intelligence and statistics , pages 10520-10545. PMLR, 2023.

Eleni Straitouri, Adish Singla, Vahid Balazadeh Meresht, and Manuel Gomez-Rodriguez. Reinforcement learning under algorithmic triage. arXiv preprint arXiv:2109.11328 , 2021.

Vahid Balazadeh, Abir De, Adish Singla, and Manuel Gomez Rodriguez. Learning to switch among agents in a team via 2-layer markov decision processes. Transactions on Machine Learning Research , 2022.

Andrew Fuchs, Andrea Passarella, and Marco Conti. Optimizing delegation between human and ai collaborative agents. arXiv preprint arXiv:2309.14718 , 2023.

Stratis Tsirtsis, Manuel Gomez Rodriguez, and Tobias Gerstenberg. Responsibility judgments in sequential human-AI collaboration. In Proceedings of the Annual Conference of the Cognitive Science Society . cognitivesciencesociety.org, 2024.

Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adaptive coverage. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 3581-3591. Curran Associates, Inc., 2020.

Anastasios Nikolas Angelopoulos, Stephen Bates, Michael I. Jordan, and Jitendra Malik. Uncertainty sets for image classifiers using conformal prediction. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

Jianguo Huang, Huajun Xi, Linjun Zhang, Huaxiu Yao, Yue Qiu, and Hongxin Wei. Conformal prediction for deep classifier via label ranking. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024.

J Pearl. Causality . Cambridge university press, 2009.

Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms . The MIT Press, 2017.

Mark Steyvers, Heliodoro Tejeda, Gavin Kerrigan, and Padhraic Smyth. Bayesian modeling of human-ai complementarity. Proceedings of the National Academy of Sciences , 119(11):e2111547119, 2022.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision , 115:211-252, 2015.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4700-4708. IEEE, 2017.

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1-9. IEEE, 2015.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778. IEEE, 2016.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , volume 25, 2012.

Isaac Gibbs, John J Cherian, and Emmanuel J Candès. Conformal prediction with conditional guarantees. arXiv preprint arXiv:2305.12616 , 2023.

Florian Heiss. Discrete choice methods with simulation. Econometric Reviews , 35:1-5, 02 2016.

Amos Tversky and Itamar Simonson. Context-dependent preferences. Management science , 39(10):1179-1189, 1993.

A Comparison to Pearl's notation

Using Pearl's notation [50], our definition of counterfactual harm reads as follows:

✶ ✶ where the subindices in ˆ Y λ and ˆ Y 1 denote (hard) interventions on the exogenous variable Λ , the random variables x, y, ˆ y ∼ P ( X,Y, ˆ Y 1 ) , and the expectation is over the prediction ˆ Y λ made by the expert using the system C λ . Using the above notation, the definition does not explicitly specify the (counterfactual) distribution used in the expectation, in contrast to the definition using the notation by Peters et al. [51], which explicitly specifies the counterfactual distribution used in the expectation, as restated below for clarity.

B Proofs

B.1 Proof of Proposition 1

Given an observation ( x, y, ˆ y ) under no intervention, we have

B.2 Proof of Proposition 2

̸

✶ where, in Eq. 12, we used the counterfactual monotonicity property by setting ✶ { ˆ Y = y } = 1 if y ∈ C λ ( x ) .

Under interventional monotonicity, by Lemma 1, we have

̸

̸

✶ where we used that { ˆ y = y } ≤ 1 .

̸

✶ By taking the expectation on both sides over ˆ y under no intervention we have

̸

✶ ✶ since ✶ { y ∈ C λ ( x ) } is a constant with respect to the expectation over ˆ Y ′ ∼ P M . By Lemma 2, Eq. 13 becomes

̸

By Assumption 2 for λ ′ = 1 and any λ such that y ∈ C λ ( x ) , we have that

which we can rewrite as

and is equivalent to

E ˆ Y ∼ P M ; do (Λ= λ ) [ ✶ { ˆ Y = Y } | X = x, Y = y ] ≤ E ˆ Y ∼ P M [ ✶ { ˆ Y = Y } | X = x, Y = y ] . Using the above result, Eq. 17 becomes

̸

If we now take the expectation in both sides over X,Y ∼ P M and use that ˆ Y ′ is equal in distribution with ˆ Y , we have

̸

̸

̸

̸

̸

̸

✶ ✶ where we used that E ˆ Y ∼ P M ; do (Λ= λ ) | ˆ Y =ˆ y,X = x,Y = y [ ✶ { ˆ Y = y } ] · ✶ { ˆ y = y } · ✶ { y ∈ C λ ( x ) } ≥ 0 . By taking the expectation in both sides over X,Y, ˆ Y ∼ P M , we have

that is equivalent to

̸

thus we conclude the proof.

B.3 Proof of Theorem 1

We will show that ✶ { ˆ Y i = Y i ∧ Y i / ∈ C λ ( X i ) } are exchangeable and that they satisfy the requirements of Theorem 1 in [22] for each X i , Y i , ˆ Y i . Since X i , Y i , ˆ Y i are i.i.d., ✶ { ˆ Y i = Y i ∧ Y i / ∈ C λ ( X i ) } are also i.i.d thus exchangeable. Considering each ✶ { ˆ Y i = Y i ∧ Y i / ∈ C λ ( X i ) } for each X i , Y i , ˆ Y i ∈ D as a function of λ , it holds by Lemma 3 that for each X i , Y i , ˆ Y i , ✶ { ˆ Y i = Y i ∧ Y i / ∈ C λ ( X i ) } is nonincreasing in λ and right-continuous. In addition, it holds that ✶ { ˆ Y i = Y i ∧ Y i / ∈ C 1 ( X i ) } = 0 ≤ α and that sup λ ✶ { ˆ Y i = Y i ∧ Y i / ∈ C λ ( X i ) } ≤ 1 surely by definition.

B.4 Proof of Theorem 2

For the proof we will follow a similar procedure as in [22]. Since ✶ { ˆ Y = Y ∧ Y ∈ C λ ( X ) } ≤ 1 , ∀ X,Y, ˆ Y , λ , it will hold that

̸

̸

If such ˆ λ ′ does not exist, by Eq. 19 ˆ λ will also not exist. If ˆ λ ′ exists, we will show that ˆ λ ≤ ˆ λ ′ . If ˆ λ = 0 , then it will hold that ˆ λ ′ ≥ ˆ λ = 0 = min λ ∈L λ . If ˆ λ > 0 , then by Eq. 19, ˆ λ must satisfy

which means that ˆ λ ≤ ˆ λ ′ . Therefore, in any case we have that ˆ λ ≤ ˆ λ ′ .

̸

From Lemma 4, the random functions ✶ { ˆ Y i = Y i ∧ Y i ∈ C λ ( X i ) } are non-decreasing for each i ∈ [ n +1] . As a result, since ˆ λ ≤ ˆ λ ′ , we have that

̸

✶ where i = { 1 , . . . , n + 1 } . Given that the data samples ( X i , Y i , ˆ Y i ) for i ∈ [1 , n + 1] are exchangeable, the values ✶ { ˆ Y i = Y i ∧ Y i ∈ C ˆ λ ′ ( X i ) } for i ∈ [ n + 1] are also exchangeable. Therefore, due to exchangeability and given that ˆ λ ′ is fixed given ( X i , Y i , ˆ Y i ) for i ∈ [1 , n + 1] , ✶ { ˆ Y n +1 = Y n +1 ∧ Y n +1 ∈ C ˆ λ ′ ( X n +1 ) } has the same probability of taking any value in D ˆ λ ′ = { ✶ { ˆ Y i = Y i ∧ Y i ∈ C ˆ λ ′ ( X i ) }} n +1 i =1 , i.e. , the value of ✶ { ˆ Y i = Y i ∧ Y i ∈ C ˆ λ ′ ( X i ) } for the n +1 -th sample in { ( X i , Y i , ˆ Y i ) } n +1 i =1 follows a uniform distribution over all the possible values in D ˆ λ ′ . As a result, from the law of total expectation it holds for i = { 1 , . . . , n + 1 } that E X i ,Y i , ˆ Y i ∼ P M [ ✶ { ˆ Y n +1 = Y n +1 ∧ Y n +1 ∈ C ˆ λ ′ ( X n +1 ) } ] = E D ˆ λ ′ [ ✶ { ˆ Y n +1 = Y n +1 ∧ Y n +1 ∈ C ˆ λ ′ ( X n +1 ) } | D ˆ λ ′ ] = ∑ n +1 i =1 ✶ { ˆ Y i = Y i ∧ Y i ∈C ˆ λ ′ ( X i ) } n +1 = ˆ G n +1 ( ˆ λ ′ ) . Finally, from Eq. 20 and the above we have

̸

̸

B.5 Additional Lemmas

Lemma 1 Under the interventional monotonicity assumption, for any x, y, ˆ y ∼ P M , the counterfactual harm that a decision support system C λ would have caused, if deployed, is given by

̸

̸

for every λ ∈ L . Now, let

̸

̸

̸

̸

̸

̸

̸

Proof. Given an observation ( x, y, ˆ y ) under no intervention, we have

̸

̸

Lemma 2 Under the structural causal model M satisfying Eq. 3, given an observation ( x, y ) it holds

̸

✶ Proof. Given an observation ( x, y, ˆ y ) under no intervention, for the expectation of the prediction of the human ˆ Y ∼ P M ; do (Λ= λ ) | X = x,Y = y, ˆ Y =ˆ y it holds

̸

̸

̸

where P ( U = u, V = v | X = x, Y = y, ˆ Y = ˆ y ) is the posterior distribution of the exogenous variables U, V conditional on the observation ( x, y, ˆ y ) .

By taking the expectation of the above with respect to the human prediction ˆ y under no intervention we have

̸

✶ Since ˆ Y ′ is equal in distribution to ˆ Y we can rewrite the above as

̸

̸

̸

̸

̸

̸

̸

and since P ( U = u, V = v | X = x, Y = y ) = ∑ ˆ y P ( U = u, V = v | X = x, Y = y, ˆ Y = ˆ y ) · P M ( ˆ Y = ˆ y | X = x, Y = y ) , we have E ˆ Y ′ ∼ P M [ E ˆ Y ∼ P M ; do (Λ= λ ) | X = x,Y = y, ˆ Y =ˆ Y ′ [ ✶ { ˆ Y = Y } ]] = ∑ u,v P ( U = u, V = v | X = x, Y = y ) · ✶ { f ˆ Y ( u, v, C λ ( x )) = y } = ∑ u P ( U = u | X = x, Y = y ) · ∑ v P ( V = v | X = x, Y = y ) · ✶ { f ˆ Y ( u, v, C λ ( x )) = y } = E ˆ Y ∼ P M ; do (Λ= λ ) [ ✶ { ˆ Y = Y } | X = x, Y = y ] , where we used that the exogenous variables U and V are independent.

Lemma 3 Under counterfactual monotonicity, for a given x, y, ˆ y the counterfactual harm h λ ( x, y, ˆ y ) is non-increasing as a function of λ and right-continuous.

Proof. If y ∈ C λ ( x ) ∀ λ ∈ L then h λ ( x, y, ˆ y ) = 0 , ∀ λ ∈ Λ given Eq. 8. If ∃ ˜ λ ∈ Λ such that ˜ λ = inf { λ ∈ Λ : y ∈ C λ ( x ) } , then ∀ λ ≥ ˜ λ , it holds that y ∈ C λ ( x ) , and as a result h λ ( x, y, ˆ y ) = 0 . For ∀ λ ′ < ˜ λ , it holds that y / ∈ C λ ′ ( x ) and as a result Eq. 8 becomes

✶ where { ˆ y = y } is constant ∀ λ ′ < ˜ λ .

̸

✶ Lemma 4 For a given data sample x with label y , let G : L → { 0 , 1 } be the function G ( λ ) = { ˆ Y = y ∧ y ∈ C λ ( x ) } . Under interventional monotonicity, G is non-decreasing in λ .

Let λ, λ ′ ∈ L such that λ ≤ λ ′ . Then it will always hold that C λ ( x ) ⊆ C λ ′ ( x ) . We distinguish the following cases for C λ ( x ) , C λ ′ ( x ) :

y / ∈ C λ ′ ( x ) . Then G ( λ ) = G ( λ ′ ) = 0 .

y / ∈ C λ ( x ) and y ∈ C λ ′ ( x ) . Then, G ( λ ) = 0 and G ( λ ′ ) = ✶ { ˆ Y = y } ≥ 0 = G ( λ ) · y ∈ C λ ( x ) ⊆ C λ ′ ( x ) . Then, G ( λ ) = G ( λ ′ ) = ✶ { ˆ Y = y } .

✶ Proof.

̸

̸

̸

̸

̸

C Additional experimental details

In this section, we describe the experimental details that are omitted from the main paper due to space constraints.

Mixture of multinomial logit models. Multinomial logit models (MNLs) [60] are among the most popular discrete choice models used to predict the probability that a human chooses an alternative, given a set of alternatives within a particular context [61]. Straitouri et al. [17] used MNLs to predict the probability that a human expert predicts a label ( e.g. , the ground truth label), given a prediction set, while considering as context the ground truth label and the level of difficulty of the image. To this end, for images with the same level of difficulty, they estimate the confusion matrix of the expert predictions on their own and use it to parameterize an MNL model. Therefore, their mixture of MNLs comprises of a different MNL model for each level of difficulty. To distinguish images based on their level of difficulty, they consider different quantiles of the experts' average accuracy values of all images. In our experiments, we consider two levels of difficulty, where in the first level we consider images for which the average accuracy of experts is smaller than the 0 . 5 quantile, and consider the rest of the images in the second level.

Labels of the ImageNet16H dataset. The ImageNet16H [52] was created using 1 , 200 unique images labeled into 207 different fine-grained categories from the ILSRVR 2012 dataset [53]. Steyvers et al. [52] mapped each of these fine-grained categories into one out of 16 coarse-grained categoriesnamely airplane, bear, bicycle, bird, boat, bottle, car, cat, chair, clock, dog, elephant, keyboard, knife, oven, and truck-that serve as the ground truth labels. This mapping essentially eliminated any label disagreement that could potentially occur between annotators in the ILSRVR 2012 dataset.

Implementation details and licenses. We implement our methods and execute our experiments using Python 3 . 10 . 9 , along with the open-source libraries NumPy 1.26.4 (BSD License), and Pandas 2.2.1 (BSD 3-Clause License). For reproducibility, we used a different fixed random seed for each random sampling of the test and calibration sets. Both datasets used in our experiments are distributed under the Creative Commons Attribution License 4.0 (CC BY 4.0).

D Additional experimental results on counterfactual monotonicity

In this section, we evaluate our methodology on additional strata of images and on larger calibration sets, and study the relationship between average counterfactual harm, average prediction set size and empirical coverage.

Other strata of images. Figures 4 and 5 show the average accuracy A ( λ ) achieved by a human expert using C λ , as predicted by the mixture of MNLs, against the average counterfactual harm H ( λ ) caused by C λ for α ∈ { 0 . 01 , 0 . 05 } on the strata of images with ω = 80 and ω = 95 , respectively. The meaning and coloring of each point are the same as in Figure 2. The results on both strata of images are consistent with the results on the stratum of images with ω = 110 -the decision support systems C λ always cause some amount of counterfactual harm, our framework successfully identifies harm-controlling sets and there is a trade-off between accuracy and counterfactual harm. However, for the strata with ω ∈ { 80 , 95 } , where the classification task has lower difficulty compared to the stratum with ω = 110 , we observe that the trade-off between accuracy and counterfactual is minimal.

0

.

9

0.01

VGG19

0

.

9

DenseNet161

0

.

85

GoogleNet

0

.

85

ResNet152

0

.

00

0

.

05

0

.

10

0

.

15

0

.

8

AlexNet

Average Accuracy

Average Counterfactual Harm

(a)

α

= 0

.

01

0

.

9

0.05

VGG19

0

.

9

DenseNet161

0

.

85

GoogleNet

0

.

85

ResNet152

0

.

00

0

.

05

0

.

10

0

.

15

0

.

8

AlexNet

Average Accuracy

Average Counterfactual Harm

(b)

α

= 0

.

05

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

9

0.01

VGG19

0

.

9

DenseNet161

0

.

85

GoogleNet

0

.

85

ResNet152

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

8

AlexNet

Average Accuracy

Average Counterfactual Harm

(a)

α

= 0

.

01

0

.

9

0.05

VGG19

0

.

9

DenseNet161

0

.

85

GoogleNet

0

.

85

ResNet152

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

8

AlexNet

Average Accuracy

Average Counterfactual Harm

(b)

α

= 0

.

05

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Larger calibration sets. Figure 6 shows the average accuracy A ( λ ) achieved by a human expert using C λ , as predicted by the mixture of MNLs, against the average counterfactual harm H ( λ ) caused by C λ for α = 0 . 01 on the stratum of images with ω = 110 . The results show that the larger the calibration set, the more often λ values with average counterfactual harm H ( λ ) close to α 12 are included in the harm-controlling set Λ( α ) . This is because, the larger the number of samples in the calibration set ( n ), the lower the estimation error in the empirical estimate of the average counterfactual harm ˆ H n . We find qualitatively similar results for the other strata of images.

Average prediction set-size and empirical coverage. Figure 7(a) shows the average prediction set-size of each decision support system C λ against the average counterfactual harm H ( λ ) caused by C λ on the stratum of images with ω = 110 . Figure 7(b) shows the empirical coverage, i.e. , the fraction of samples x in the test set, for which ground truth label y ∈ C λ ( x ) , for each decision support system C λ against the average counterfactual harm H ( λ ) caused by C λ on the same stratum of images. The results show that systems with higher coverage construct larger prediction sets in average and cause less counterfactual harm. We find similar results for the other strata of images.

0

.

8

0

.

9

0.01

VGG19

0

.

8

0

.

9

DenseNet161

0

.

75

0

.

80

0

.

85

GoogleNet

0

.

75

0

.

80

0

.

85

ResNet152

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

7

0

.

8

AlexNet

Average Accuracy

Average Counterfactual Harm

(a) Calibration samples

n

= 360 (30%)

0

.

8

0

.

9

0.01

VGG19

0

.

8

0

.

9

DenseNet161

0

.

75

0

.

80

0

.

85

GoogleNet

0

.

75

0

.

80

0

.

85

ResNet152

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

7

0

.

8

AlexNet

Average Accuracy

Average Counterfactual Harm

(b) Calibration samples

n

= 600 (50%)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

5

10

15

VGG19

5

10

15

DenseNet161

5

10

15

GoogleNet

5

10

15

ResNet152

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

5

10

15

AlexNet

Average Set Size

Average Counterfactual Harm

(a) Average Set-Size

0

.

9

1

.

0

VGG19

0

.

9

1

.

0

DenseNet161

0

.

75

1

.

00

GoogleNet

0

.

75

1

.

00

ResNet152

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

75

1

.

00

AlexNet

Empirical Coverage

Average Counterfactual Harm

(b) Empirical Coverage

E Experimental results on counterfactual monotonicity with a more complex set-valued predictor

In this section, we experiment with a system that constructs the prediction set C ( x ) using a more complex set-valued predictor [49]. This set-valued predictor first computes the following nonconformity score for each label value y ∈ Y using the softmax output of a pre-trained classifier m y ( x ) ∈ [0 , 1] :

where y ( i ) denotes the label value with the i -th largest classifier output, u ∼ U (0 , 1) is a uniform random variable and w is a hyperparameter representing the weight of the ranking of the label value. Then, given a user-specified threshold λ ∈ R , it constructs the prediction sets C ( x ) = C λ ( x ) as follows:

where, with a slight abuse of notation, y ( i ) is the label value with the i -th largest s ( y, x, u ; w ) value. Here, we should note that may exist data samples with s ( y, x, u ; w ) > 1 . Therefore, considering λ ∈ [0 , 1] may not include every set-valued predictor given by Eq. 27. For this reason, in what follows, we consider λ ∈ [0 , max s ( y, x, u, w )] , where the maximum is essentially over the classifier output m y ( x ) , the uniform random variable u and the hyperparameter w .

Experimental setup. We use the ImageNet16H dataset and follow the same setup as described in Section 6, with only difference that we split the images into a calibration set (10%), which we use to find Λ( α ) , a validation set (10%) , which we use to select the value of the hyperparameter w following a similar procedure as in Huang et al. [49], and a test set (80%). In each iteration of the experiment, given a λ value, we select the value of w ∈ { 0 . 02 , 0 . 05 , 0 . 1 , 0 . 15 , 0 . 2 , 0 . 25 , 0 . 3 , 0 . 35 } for which the set-valued predictor achieves the smallest average prediction set-size over the data samples in the validation set.

Results. The meaning and coloring of each point are the same as in Figure 2. The results on all strata of images are consistent with the results using the set-valued predictor given by Eq. 1-the decision support systems C λ always cause some amount of counterfactual harm, our framework successfully identifies harm-controlling sets and there is a trade-off between accuracy and counterfactual harm.

Figure 11(a) shows the average prediction set-size of each decision support system C λ against the average counterfactual harm H ( λ ) caused by C λ on the stratum of images with ω = 110 . Figure 11(b) shows the empirical coverage for each decision support system C λ against the average counterfactual harm H ( λ ) caused by C λ on the stratum of images with ω = 110 . Our results are consistent with the results for the set-valued predictor given by Eq. 1 in Figure 7-systems C λ that achieve higher coverage, construct larger prediction sets on average and cause less harm.

0

.

90

0

.

95

0.01

VGG19

0

.

90

0

.

95

DenseNet161

0

.

8

0

.

9

GoogleNet

0

.

8

0

.

9

ResNet152

0

.

00

0

.

05

0

.

10

0

.

15

0

.

8

0

.

9

AlexNet

Average Accuracy

Average Counterfactual Harm

(a)

α

= 0

.

01

0

.

90

0

.

95

0.05

VGG19

0

.

90

0

.

95

DenseNet161

0

.

8

0

.

9

GoogleNet

0

.

8

0

.

9

ResNet152

0

.

00

0

.

05

0

.

10

0

.

15

0

.

8

0

.

9

AlexNet

Average Accuracy

Average Counterfactual Harm

(b)

α

= 0

.

05

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

875

0

.

900

0

.

925

0.01

VGG19

0

.

875

0

.

900

0

.

925

DenseNet161

0

.

8

0

.

9

GoogleNet

0

.

8

0

.

9

ResNet152

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

8

0

.

9

AlexNet

Average Accuracy

Average Counterfactual Harm

(a)

α

= 0

.

01

0

.

875

0

.

900

0

.

925

0.05

VGG19

0

.

875

0

.

900

0

.

925

DenseNet161

0

.

8

0

.

9

GoogleNet

0

.

8

0

.

9

ResNet152

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

8

0

.

9

AlexNet

Average Accuracy

Average Counterfactual Harm

(b)

α

= 0

.

05

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

80

0

.

85

0.01

VGG19

0

.

80

0

.

85

DenseNet161

0

.

75

0

.

80

0

.

85

GoogleNet

0

.

75

0

.

80

ResNet152

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

7

0

.

8

AlexNet

Average Accuracy

Average Counterfactual Harm

(a)

α

= 0

.

01

0

.

80

0

.

85

0.05

VGG19

0

.

80

0

.

85

DenseNet161

0

.

75

0

.

80

0

.

85

GoogleNet

0

.

75

0

.

80

ResNet152

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

7

0

.

8

AlexNet

Average Accuracy

Average Counterfactual Harm

(b)

α

= 0

.

05

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

5

10

15

VGG19

5

10

15

DenseNet161

5

10

15

GoogleNet

5

10

15

ResNet152

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

5

10

15

AlexNet

Average Set Size

Average Counterfactual Harm

(a) Average Set-Size

0

.

9

1

.

0

VGG19

0

.

9

1

.

0

DenseNet161

0

.

75

1

.

00

GoogleNet

0

.

75

1

.

00

ResNet152

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

75

1

.

00

AlexNet

Empirical Coverage

Average Counterfactual Harm

(b) Empirical Coverage

F Average accuracy vs. prediction set size

In this section, we use the ImageNet16H-PS [18] dataset to verify the interventional monotonicity assumption, i.e. , whether human experts achieve higher average accuracy when predicting from smaller prediction sets that include the ground truth label. To this end, we follow a procedure, similar to the one used by Straitouri et al. [18] to verify the interventional monotonicity assumption unconditionally of the value of the ground truth label.

Straitouri et al. estimate the average accuracy per prediction set size on images with similar difficulty, averaged across all experts and across experts with the same level of competence. They consider images with similar difficulty instead of single images, as the dataset ImageNet16H-PS does not include enough human predictions per image to faithfully estimate the average accuracy per image. They stratify the images into groups of similar difficulty, following the same procedure used in Straitouri et al. [17] based on different quantiles of the average accuracy values of all images. In our work, we stratify images with the same ground truth label into the following four groups:

High difficulty: images with average accuracy within the 0 . 25 quantile of the average accuracy values of all images with the same ground truth label.

Medium to high difficulty: images with average accuracy within the 0 . 5 quantile and outside the 0 . 25 quantile of the average accuracy values of all images with the same ground truth label.

Medium to low difficulty: images with average accuracy within the 0 . 75 quantile and outside the 0 . 5 quantile the of the average accuracy values of all images with the same ground truth label.

Low difficulty: images with average accuracy outside the 0 . 75 quantile of the average accuracy values of all images with the same ground truth label.

We follow a similar method to stratify experts based on their level of competence into two groups for each ground truth label. To measure the level of competence of an expert for a ground truth label, we use the average accuracy across all the predictions that she made on images with this ground truth label. For each ground truth label, we consider the 50% of experts with the highest average accuracy as the experts with high level of competence and the rest 50% as experts with low level of competence.

Figure 12 shows the average accuracy per prediction set size-for prediction sets that include the ground truth label-across images of various levels of difficulty and experts with different levels of competence for some of the ground truth labels. We find qualitatively similar results for the rest of the ground truth labels. The results are consistent with the results on the unconditional interventional monotonicity by Straitouri et al. [18], showing that as long as the classification task is not too easy, the experts achieve higher average accuracy when predicting from smaller prediction sets, i.e. , the interventional monotonicity holds.

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16

0

.

2

0

.

4

0

.

6

Average Accuracy

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16

0

.

6

0

.

7

0

.

8

0

.

9

Average Accuracy

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16

0

.

8

0

.

9

1

.

0

Average Accuracy

High difficulty

Medium to high difficulty

Medium to low difficulty

(a) All experts, ground truth label

bottle

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16

0

.

2

0

.

4

0

.

6

Average Accuracy

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

Average Accuracy

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16

0

.

6

0

.

7

0

.

8

0

.

9

Average Accuracy

High difficulty

Medium to high difficulty

Medium to low difficulty

(b) All experts, ground truth label

oven

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16

0

.

0

0

.

2

0

.

4

0

.

6

Average Accuracy

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16

0

.

4

0

.

6

0

.

8

Average Accuracy

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16

0

.

6

0

.

8

1

.

0

Average Accuracy

High difficulty

Medium to high difficulty

Medium to low difficulty

(c) Experts with low level of competence, ground truth label

bottle

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16

0

.

4

0

.

6

0

.

8

Average Accuracy

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

Average Accuracy

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

Average Accuracy

High difficulty

Medium to high difficulty

Medium to low difficulty

(d) Experts with high level of competence, ground truth label bottle

G Experiments under the interventional monotonicity assumption

In this section, we show the average accuracy A ( λ ) achieved by a human expert using C λ , as predicted by the mixture of MNLs, against the average counterfactual harm upper-bound caused by C λ for several α values on the strata of images with ω = 80 in Figure 13, with ω = 95 in Figure 14 and with ω = 110 in Figure 15. Here, each point corresponds to a different λ value and its coloring indicates the empirical probability that λ is included in the harm-controlling set Λ ′ ( α ) given by Corollary 2, for a fixed α ′ value across the random samplings of the test and calibration sets. For each classifier and stratum of images, we select the α ′ value maximizing the expected size of the harm-controlling set Λ ′ ( α ) . The results on all strata of images support the findings derived from the experiments assuming that the counterfactual monotonicity assumption holds. In addition, we observe that the gap between the lower-bound of the average counterfactual harm (that is equal to the average counterfactual harm under the counterfactual monotonicity assumption) and the upper-bound is often large.

0

.

9

0.12

VGG19

0

.

9

DenseNet161

0

.

85

GoogleNet

0

.

85

ResNet152

0

.

10

0

.

15

0

.

20

0

.

8

AlexNet

Average Accuracy

Average Counterfactual Harm Bound

(a)

α

= 0

.

12

0

.

9

0.14

VGG19

0

.

9

DenseNet161

0

.

85

GoogleNet

0

.

85

ResNet152

0

.

10

0

.

15

0

.

20

0

.

8

AlexNet

Average Accuracy

Average Counterfactual Harm Bound

(b)

α

= 0

.

14

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

9

0.15

VGG19

0

.

9

DenseNet161

0

.

85

GoogleNet

0

.

85

ResNet152

0

.

150

0

.

175

0

.

200

0

.

225

0

.

250

0

.

275

0

.

8

AlexNet

Average Accuracy

Average Counterfactual Harm Bound

(a)

α

= 0

.

15

0

.

9

0.17

VGG19

0

.

9

DenseNet161

0

.

85

GoogleNet

0

.

85

ResNet152

0

.

150

0

.

175

0

.

200

0

.

225

0

.

250

0

.

275

0

.

8

AlexNet

Average Accuracy

Average Counterfactual Harm Bound

(b)

α

= 0

.

17

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

8

0

.

9

0.23

DenseNet161

0

.

75

0

.

80

0

.

85

GoogleNet

0

.

75

0

.

80

0

.

85

ResNet152

0

.

22

0

.

24

0

.

26

0

.

28

0

.

30

0

.

7

0

.

8

AlexNet

Average Accuracy

Average Counterfactual Harm Bound

(a)

α

= 0

.

23

0

.

8

0

.

9

0.24

DenseNet161

0

.

75

0

.

80

0

.

85

GoogleNet

0

.

75

0

.

80

0

.

85

ResNet152

0

.

22

0

.

24

0

.

26

0

.

28

0

.

30

0

.

7

0

.

8

AlexNet

Average Accuracy

Average Counterfactual Harm Bound

(b)

α

= 0

.

24

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of the work are discussed in Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The proofs of all the theoretical results are provided in Appendix B.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The information to reproduce the experimental results is provided in Section 6 and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The datasets as well as the source code we use are publicly available (see Footnotes 7, 8, and 10).

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify all the details in Section 6 and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In Section 6, and Appendix D, Appendix E and Appendix G we report 95% confidence intervals, and in Appendix F we report standard errors.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We report computer resources in Footnote 7.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper discusses both potential positive and negative societal impacts of the work in Section 7.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not introduce any data nor models with high risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators of the datasets we use are credited in Section 6 and Appendix C, the URLs of the datasets are provided in Footnotes 8 and 10 and the licenses of the datasets and of the libraries used in the code are provided in Appendix C.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The open-source implementation of our methodology that we publicly release contains documentation.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a notion of counterfactual harm for a classification setting in which a human is choosing among a subset of possible labels determined by an AI system, i.e., the automated system is used to \"narrow down\" the set of most likely labels for a sample. Under assumptions of counterfactual and interventional monotonicity, which imply that , the notion of harm can be identified pointwise or up to an interval. A procedure for controlling harm is introduced, based on conformal risk control.\n\nSTRENGTHS:\n(S1) The described problem is important. Furthermore, not many works in this space exist, and the definition of harm is not universally agreed upon, as noted by the authors. Thus, this paper explores a valuable direction of research.\n\n(S2) Many of the concepts are introduced in a clean way, and the writing is good overall.\n\n(S3) The paper seems to combine many state-of-the-art ideas, including notions of counterfactual harm, identification under monotonicity assumptions from previous works, and conformal risk control.\n\nWEAKNESSES:\nMajor:\n\n(W1) The causal inference notation is not up to the expected standard. In Definition 1, which is key to the paper, the notation does not work. The paper is trying to describe the notion of counterfactual harm, while using only the $do()$ operator and conditioning. The do() operator and conditioning can only give quantities in the first two layers of the causal hierarchy. Therefore, the current definition, as written, is incorrect. The appearance of $\\hat Y$ in multiple places with different meanings is concerning. Instead of this, the paper probably wants to use the subscript notation $\\hat Y _{\\lambda}$. \n\nMy understanding is that the paper is trying to say\n\n$$ h_\\lambda (x, y, \\hat y) = E [ max (0, \\mathbb{1} ( \\hat Y = Y ) - \\mathbb{1} (\\hat Y _{\\lambda} = Y) ) \\mid X = x, \\hat Y = \\hat y, Y = y]. $$\n\n(W2) The connection to Richens et. al [1] is mentioned in the introduction. It is clarified that the difference here is that \"their definition requires one to deploy the system\", while this is not the case for the definition proposed in this work. It is not clear from the text whether  the current definition does not require deployment because of the definition itself, or the monotonicity assumptions that are invoked.\nThe two definitions seem closely related, and given the comment in (W1), it would be very valuable to have these two definitions compared side-by-side, with a unified choice of corrected notation. Opening a small appendix for this would be very desirable.\n\n---\n\nMinor comments / corrections: \n\n(W3) Relating to point (W1): in Pearl's notation, causal mechanisms are usually denoted by the assignment operator $\\gets$, i.e., $X \\gets f_X (U_X)$. This is useful to indicate that the mechanisms are not \"equations that can be manipulated\".\n\n(C1) Line 159: \"if they has failed\"\n\nQUESTIONS:\nPlease see (W1), (W2).\n\n(Q1) Lines 147-148 claim that the $\\Lambda (\\alpha)$ set cannot be found because counterfactual harm is a counterfactual quantity. This statement seems imprecise. For instance, the effect of treatment on the treated (ETT), is a counterfactual quantity that can be estimated with lack-of-confounding assumptions. Furthermore, any counterfactual quantity could be, in principle, bounded without additional assumptions. \n\nIt seems more likely that the joint distribution of $\\hat Y, \\hat Y _{\\lambda}$ is required to evaluate the expression (and this seems to be a consequence of the max() operator within the expectation), which is known hard problem, often solved with monotonicity assumptions.\n\n(Q2) When taking a step back, the question that comes to mind is: what is the end goal of the proposal? When minimizing counterfactual harm, it seems that \"counterfactual benefit\" is almost entirely ignored. Is this a reasonable notion to ignore? Furthermore, if the definition were to remove the max(), would this be equivalent to optimizing overall accuracy or something similar (note that removing the max solves the identifiability issue)? Finally, should there be an extension in which $\\lambda = \\lambda(x)$ is a function of $X = x$, so that harm is minimized in areas where it is largest?\n\n(Q3) Line 142: \"Otherwise, the expert's prediction could not have become worse\". The claim of the paper is about decision-making support systems, whereas this comment seems quite closely focused on accuracy. For instance, in the very example given in Figure 1: if someone has a common cold, mistaking this for a flu seems rather different than mistaking it for lung cancer (the consequences of these two classifications may be rather different)? Please comment on this. \n\n(Q4) Lines 127-130 are saying that an SCM can match the observational distribution. Can you comment on the significance of this observtion? Is this remark made to claim that the usage of SCMs does not limit the generality of the approach, due to their expressiveness? If so, perhaps it is worth adding such a sentence.\n\nLIMITATIONS:\nPlease check (W1), (W2), and (Q1), (Q3).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the issue with prediction methods that generate a set of potential labels, from which a human makes the final decision. The authors identify a harmful scenario where the model's intervention leads to incorrect predictions that a human would have otherwise made correctly. To model this, they use the Structural Causal Model (SCM) framework.\n\nThe prediction set size is controlled by a parameter $\\lambda$. For a given $\\lambda$, they define counterfactual harm as the average number of instances where a human's unaided prediction would be correct, but becomes incorrect with the model's assistance. The main objective of the paper is to identify the range of $\\lambda$ values that keep the counterfactual harm within a user-specified constant $\\alpha$.\n\nTo achieve this, the authors use conformal prediction and propose two training algorithms under two conditions:\n\n- Counterfactual monotonicity: At the instance level $(x, y, \\hat{y})$, smaller and correct prediction sets lead to better human predictions.\n- Interventional monotonicity: At the sub-population level characterized by $(x, y)$, smaller and correct prediction sets lead to better human predictions *on average*.\n\nThey conduct experiments on two image datasets where human predictions from prediction sets are available, demonstrating that their methods effectively control counterfactual harm.\n\nSTRENGTHS:\nThe problem formulation is interesting. The authors establish an objective that essentially tunes the threshold parameters used in Conformal Prediction, aiming to control the number of instances where the prediction sets impede the human from making the correct prediction. Solving the optimization problem requires access to counterfactual labels for human predictions across all possible prediction sets, which are naturally unavailable in observational datasets. To address this, the authors identify two weaker conditions, which are highly likely to hold in practice, to solve the problem.\n\nFinally, they apply the conformal prediction algorithm on a hold-out calibration dataset to identify the set of $\\lambda$ values that bring the CF harm under control.\n\nWEAKNESSES:\nPlease refer the questions below.\n\nQUESTIONS:\n**Equation 4:**\n\nIn the definition of CF harm, why is the true label $Y$ a function of the predicted label $\\hat{Y}$? Could you please elaborate on the notation?\n\n**Line 164:**\n\nShould $v$ not be abducted from the joint distribution of $(X, Y, \\hat{Y})$ rather than just $X$?\n\n**Lines 127 to 130:**\n\nI do not understand the text in these lines. Are you stating that one can always find some distribution over the exogenous variables that aligns with the observational data, or are you claiming that the original SCM is identifiable?\n\n**Typo at Line 159:**\n\nhas --> had\n\n**Proof for Proposition 1:**\n\nIn the proof for Proposition 1, where have you utilized the CF monotonicity condition? The proof seems to hold for any arbitrary SCM $\\mathcal{M}$. Am I missing something here?\n\n\n**On CF monotonicity:**\n\nI believe that under no intervention, a human predicts the label with $C_\\lambda(x) = \\mathcal{Y}$, i.e., the entire label set. If CF monotonicity holds, why wouldn't any conformal prediction algorithm that guarantees $1-\\alpha$ coverage satisfy the CF harm requirement?\n\n**Examples for CF monotonicity violations**\n\nCan you provide examples from your considered datasets where CF monotonicity is violated. In practice, why would such violations happen?\n\n**Tradeoff between CF harm and conformal accuracy:**\n\nI believe accounting for CF harm may unnecessarily increase the prediction set size, beyond what vanilla conformal prediction would offer, for the same amount of coverage guarantees provided by both methods. Can the authors comment on this, at least empirically?\n\nIf such a tradeoff exists, this approach would be interesting if one could find the subset of instances where harm occurs and explicitly find the prediction set for such instances. However, I am disappointed that the paper does not propose such approaches.\n\n**Conformal baselines**\n\nThe authors should compare their algorithms with the state-of-the-art conformal prediction algorithms and evaluate how their methods perform in terms of (a) conformal accuracy, (b) prediction set size, and (c) CF harm. The current experimental section appears too weak.\n\n\n**Conformalizing $\\hat{Y}$:**\n\nWhy shouldn't this approach work? Suppose we apply conformal prediction to both the ground truth $Y$ and the expert prediction $\\hat{Y}$, and then emit the prediction set as a union of the two prediction sets obtained from these conformalized models. How does your approach compare to such a proposal in terms of the accuracy $A(\\lambda)$ and the prediction set size?\n\nLIMITATIONS:\nYes, limitations are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on decision support designed to help humans in multi-class classification tasks via prediction sets. The paper defines the notion of counterfactual harm, namely when usage the system would lead the agent to a wrong prediction which would have been correct without the system’s ‘help’. The authors show that under some monotonicity assumptions such counterfactual harm is identifiable or partially identifiable. Against this theoretical backdrop, the paper proposes a method to find sets of interventions (namely thresholds for the prediction sets) which are guaranteed to have bounded harm. This approach is validated using experiments on human subject studies. These studies reveal a trade off between post-intervention accuracy and counterfactual harm.\n\nSTRENGTHS:\nI am not an expert in prediction sets so I cannot fully evaluate how significant this contribution is, and was not able to fully check the proofs. I saw the following strengths:\n1. The trade off between accuracy and counterfactual harm is very interesting, and the possibility to control the desired amount of counterfactual harm seems very important.\n2. The theoretical results seem solid and are well rounded out by the corroborating experiments.\n3. Despite the technicality of the paper, I could follow the argument and appreciate the message. The text is well written.\n\nWEAKNESSES:\n1.      The notation little y-hat and capital Y-hat in section 3 is somewhat confusing. Line 123 implicitly suggests the latter is the random variable generated by the user predicting with the decision support. Line 142 however states that little y-hat is the prediction the experts would make on their own. I would have expected this to be just   Y-hat but with lambda = 1, as stated elsewhere in the text?\n2. In the first experiment, when w is different from 110, the data contains predictions made by humans on their own, and the accuracy of the humans using prediction sets is only estimated. How good are these estimates? We only get to see a comparison with the real value in Figure 3 for w=110, and it is clear that these estimates decisively overshoot.\n3. The results show that the sets of lambdas do not contain all the harm-controlling values - as the authors remark - and especially seem to contain less of the ‘good’ values, meaning those that (at the cost of some counterfactual harm within the accepted bound) achieve higher accuracy. This seems to be a serious drawback.\n4. Lambda is used in line 105 but only introduced in line 109.\n5. Theorem 1 and 2 could use a bit more intuition and/or explanation.\n\nTypos:\nline 137: use -> used\n\nQUESTIONS:\nBeside remarks on the weaknesses mentioned above, I would be interested in answers to the following questions:\n1.   In Figure 1 you show the average accuracy of humans estimated by MNLs when w=110, but in the text you report that the second dataset contains the predictions of humans with prediction sets at all thresholds for that value of w, which is then shown in Figure 2. Why not start with the actual accuracy then, instead of the estimates? It seems to me that Figure 2 has the core findings, while Figure 1 is more of a sensitivity analysis on the fact that this phenomenon is consistent across model architectures (under the assumptions that the MNL estimates are qualitatively reliable).\n2. Three of the models tested have worse average performance than the human experts. What is the point of testing decision support with such models? I suppose we would never implement a decision support that is worse than humans alone (unless perhaps we have strong evidence it is better than humans on a subgroup of data points, but this is not the case here), since it can stir the humans with poor predictions. I can see this could be interesting since you are studying counterfactual harm, but still it is an unrealistic scenario so I would not present results for three such models.\n\nLIMITATIONS:\nThe paper seems to mention and discuss all the relevant assumptions and limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper relates to decision support systems based on prediction sets. It introduces a framework using conformal risk control to design prediction sets that balance accuracy with minimizing potential harm (understood as a human who has succeeded at predicting the ground-truth label of an instance on their own failing had they used the DS system), validated with human prediction data from empirical lab studies.\n\nSTRENGTHS:\n- Originality. The paper establishes formal definitions/characterizations and a theoretical foundation. The paper formally characterizes the notion of \"counterfactual harm\" and introduces the counterfactual monotonicity assumptions and weaker interventional monotonicity assumptions to make counterfactual harm identifiable and partially identifiable, respectively. This conceptual contribution improves the understanding of the risks associated with decision support systems based on prediction sets.\n\n- Significance. The authors demonstrated that the idea of \"counterfactual harm\" can be made practically useful through conformal risk control (section 5), and they performed lab experiments to validate their theoretical bounds. The result is practical: Cor 1 and 2 ensure that these systems cause less counterfactual harm on average than a user-specified threshold.\n\nWEAKNESSES:\n1. The main weakness is the usability and motivation of this particular formulation. The paper proposes decision support systems based on prediction sets, which might not always be feasible or realistic, especially in complex domains like healthcare. Most current systems are based on risk prediction e.g. disease risk, drop out risk, for binary outcomes. I don't see any good examples of problems where prediction sets are the right approach, and this paper also does not do this work of illustrating or justifying this formulation using real world case studies.\n\n2. Discussing the Limitations of Predictive Accuracy. While the paper discusses improving accuracy through prediction sets, there may be fundamental limitations in predicting outcomes accurately in the first place. Have the authors considered recent research or critiques regarding the actionability of predictive models? For instance, how does the work align with concerns raised by Wang et al (2024) \"Against Predictive Optimization\" and Liu et al. (2024) \"On the actionability of...\"?\n\nQUESTIONS:\n1. Are there practical examples of decision support systems based on prediction sets being successfully implemented in healthcare or other social domains? How do these systems handle the inherent complexity and variability of real-world decision-making scenarios, such as patient care pathways or policy decisions?\n\n2. Have the authors considered recent research or critiques regarding the actionability of predictive models? For instance, how does the work align with concerns raised by Wang et al (2024) \"Against Predictive Optimization\" and Liu et al. (2024) \"On the actionability of...\"?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}