Controlling Counterfactual Harm in Decision Support Systems Based on Prediction Sets

Authors: eleni  straitouri
suhas  thejaswi
manuel  gomez rodriguez
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 14:51:07

Abstract: Decision support systems based on prediction sets help humans solve multiclass classification tasks by narrowing down the set of potential label values to a subset of them, namely a prediction set, and asking them to always predict label values from the prediction sets. While this type of systems have been proven to be effective at improving the average accuracy of the predictions made by humans, by restricting human agency, they may cause harm---a human who has succeeded at predicting the ground-truth label of an instance on their own may have failed had they used these systems. In this paper, our goal is to control how frequently a decision support system based on prediction sets may cause harm, by design. To this end, we start by characterizing the above notion of harm using the theoretical framework of structural causal models. Then, we show that, under a natural, albeit unverifiable, monotonicity assumption, we can estimate how frequently a system may cause harm using only predictions made by humans on their own. Further, we also show that, under a weaker monotonicity assumption, which can be verified experimentally, we can bound how frequently a system may cause harm again using only predictions made by humans on their own. Building upon these assumptions, we introduce a computational framework to design decision support systems based on prediction sets that are guaranteed to cause harm less frequently than a user-specified value 
using conformal risk control. We validate our framework using real human predictions from two different human subject studies and show that, in decision support systems based on prediction sets, there is a trade-off between accuracy and counterfactual harm.

Main paper: 
 introduction: 
 The principle of "first, do no harm" holds profound significance in a variety of professions across multiple high-stakes domains. For example, in the field of medicine, doctors swear an oath to prioritize their patient's well-being or, in the legal justice system, preserving the innocence of individuals is paramount. As a result, in all of these domains, rules and guidelines have been established to prevent decision makers-doctors or judges-from making decisions that harm individuals-patients or suspects. In recent years, it has been increasingly argued that a similar principle should apply to decision support systems using machine learning algorithms in high-stakes domains [1] [2] [3] [4] . 1The definition of harm is not unequivocally agreed upon, however, the most widely accepted definition is the counterfactual comparative account of harm (in short, counterfactual harm) [5] [6] [7] , which we adopt in our work. Under this definition, an action causes harm to an individual if they would have been in a worse state had the action been taken. Building upon this definition, we say that a decision support system causes harm to an individual if a decision maker would have made a worse decision about the individual had they used the system.In machine learning for decision support, one of the main focus has been classification tasks. Here, the most studied setting assumes the decision support system uses a classifier to predict the value of a (ground-truth) label of interest and a human expert uses the predicted value to update their own prediction [8] [9] [10] [11] [12] . Unfortunately, in this setting, it is yet unclear how to guarantee that the (average) accuracy of the predictions made by an expert who uses the system is higher than the accuracy of the predictions made by the expert and the classifier on their own, what is often referred to as human-AI complementarity [13] [14] [15] [16] . In this context, a recent line of work [17, 18] have argued, both theoretically and empirically, that an alternative setting may enable human-AI complementarity. In this alternative setting, the decision support system helps a human expert by providing a set of label predictions, namely a prediction set, and asking them to always predict a label value from the set. The key principle is that, by restricting human agency, good performance does not depend on the expert developing a good sense of when to predict a label from the prediction set. In this context, it is also worth noting that Google has recently developed a tool that uses patient history and skin condition images to provide decision support using prediction sets [19] , and a study by Jain et al. [20] has found that physicians and nurses using this tool improved diagnoses for 1 in every 8 to 10 cases.In this work, we argue that the same principle that enables human-AI complementarity on decision support systems based on prediction sets may also cause counterfactual harm-a human expert who has succeeded at predicting the label of an instance on their own may have failed had they used these systems. Consequently, our goal is to design decision support systems based on prediction sets that are guaranteed to cause, in average, less counterfactual harm than a user-specified value.Our contributions. We start by formally characterizing the predictions made by a decision maker using a decision support system based on prediction sets using a structural causal model (SCM) and, based on this characterization, formalize our notion of counterfactual harm. In general, since counterfactual harm lies within level three in the "ladder of causation" [21] , it is not (partially) identifiable-it cannot be estimated (bounded) from data. However, we show that, under a natural counterfactual monotonicity assumption on the predictions made by decision makers using decision support systems based on prediction sets, counterfactual harm is identifiable. Further, we show that, under a weaker interventional monotonicity assumption, which can be verified experimentally, the average counterfactual harm is partially identifiable. Then, building upon these assumptions, we develop a computational framework to design decision support systems based on prediction sets that are guaranteed to cause, in average, less counterfactual harm than a user-specified value using conformal risk control [22] . Finally, we validate our framework using real human predictions from two different human subject studies and show that, in decision support systems based on prediction sets, there is a trade-off between accuracy and counterfactual harm.Further related work. Our work builds upon further related work on set-valued predictors, critiques of prediction optimization, counterfactual harm and algorithmic triage.Set-valued predictors output a set of label values, namely a prediction set, rather than single labels [23] . However, set-valued predictors have not been typically designed nor evaluated by their ability to help human experts make more accurate predictions [24] [25] [26] [27] [28] [29] . Only very recently, an emerging line of work has shown that conformal predictors, a specific type of set-valued predictors, may help human experts make more accurate predictions [17, 18, [30] [31] [32] [33] . Within this line of work, the work most closely related to ours is by Straitouri et al. [17, 18] , which has introduced the setting, and counterfactual and interventional monotonicity properties we build upon.Prediction optimization has been recently put into question in the context of decision support [34] [35] [36] . More specifically, it has been argued that optimizing decision support systems to improve prediction accuracy does not always translate to better decision-making. Our work aligns with this critique since we argue that improving prediction accuracy may come at the cost of counterfactual harm.The literature on counterfactual harm in machine learning is still quite small and has focused on traditional machine learning settings in which machine learning models replace human decision makers and make automated decisions [1] [2] [3] [4] . Within this literature, the work most closely related to ours is by Richens et al. [1] , which also uses a structural causal model to define counterfactual harm. However, their definition of counterfactual harm differs in a subtle, but important, way from ours. In our setting, their definition of counterfactual harm would compare the accuracy of a factual prediction made by an expert using the decision support system against the counterfactual prediction that the same expert would have made on their own. That means, under their definition, one would need to deploy the system to estimate the harm it may cause, potentially causing harm. In contrast, under our definition, one does not need to deploy the system to estimate (or bound) the harm it may cause, as it will become clear in Section 4, and thus we argue that our definition may be more practical.Learning under algorithmic triage seeks to develop classifiers that make predictions for a given fraction of the samples and leave the remaining ones to human experts, as instructed by a triage policy [37] [38] [39] [40] [41] [42] . In contrast, in our work, for each sample, a classifier is used to construct a prediction set and a human expert needs to predict a label value from the set. In this context, it is also worth noting that learning under algorithmic triage has been extended to reinforcement learning settings [43] [44] [45] [46] . 
decision support systems based on prediction sets: 
 We consider a multiclass classification task in which, for each task instance, a human expert has to predict the value of a ground-truth label y ∈ Y = {1, . . . , L}. Then, our goal is to design a decision support system C : X → 2 Y that, given a set of features x ∈ X , helps the expert by narrowing down the label values to a subset of them C(x) ⊆ Y, namely a prediction set. Here, we focus on a setting in which the system asks the expert to always predict a label value ŷ from the prediction set C(x). Note that, by restricting the expert's agency, good performance does not depend on the human expert developing a good sense of when to predict a label from the prediction set. Moreover, we assume that the set of features, the ground-truth label and the expert's prediction are sampled from an unknown fixed distribution2 , i.e., x, y ∼ P (X, Y ) andŷ ∼ P ( Ŷ | X, Y, C(X)).Further, similarly as in Straitouri et al. [17, 18] , we consider that, given a set of features x ∈ X , the system constructs the prediction set C(x) using the following set-valued predictor [23] . First, the set-valued predictor ranks each potential label value y ∈ Y using the softmax output of a pre-trained classifier m y (x) ∈ [0, 1]. Then, given a user-specified threshold λ ∈ [0, 1], it uses the resulting ranking to construct the prediction set C(x) = C λ (x) as follows:EQUATIONwhere • (i) denotes the i-th label value in the ranking. Here, note that, for λ = 0, the prediction set contains just the top ranked label value, for λ = 1, it contains all label values. 3Given the above parameterization, one may just focus on finding the optimal threshold λ * under which the human expert achieves the highest average accuracy as in Straitouri et al. [17, 18] , i.e.,EQUATIONHowever, this focus does not prevent the resulting system C λ from causing harm-an expert may succeed to predict the value of the ground-truth label on their own on instances in which they would have failed had they used C λ . In this work, our goal is to design a computational framework that, given a user-specified bound α ∈ [0, 1], finds the set of λ values which are all guaranteed to cause less harm, in average, than the bound α.Remark. We would like to clarify that, if one sets the value of the threshold λ to be roughly the 1 -α quantile of the empirical distribution of the scores 1 -m y (x) in a calibration set, then, the set valued predictor defined by Eq. 1 is equivalent to a vanilla conformal predictor with nonconformity scores 1 -m y (x) and coverage 1 -α. Under this view, it becomes apparent that, by searching for λ values in [0, 1] that are harm controlling, we are essentially searching for vanilla conformal predictors that are harm controlling. In this context, we would like to further clarify that our framework is agnostic to the choice of nonconformity score or more generally, the set-valued predictor, used to construct the prediction sets [47] [48] [49] . Motivated by this observation, in Appendix E, we include additional experiments where we evaluate our framework using a more complex set-valued predictor [49] .Y Ground Truth Label V Λ U Exogenous Variables 𝒞 ! (X)Prediction SetEdema Figure 1 : Our structural causal model M. Circles represent endogenous random variables and boxes represent exogenous random variables. The value of each endogenous variable is given by a function of the values of its ancestors, as defined by Eq. 3. The value of each exogenous variable is sampled independently from a given distribution. 
counterfactual harm of decision support systems: 
 To formalize our notion of harm, we characterize how human experts make predictions using a decision support system C via a structural causal model (SCM) [21] , which we denote as M. More specifically, similarly as in Straitouri et al. [18] , we define M by the following set of assignments:EQUATIONwhere Λ, U and V are exogenous random variables and f X , f Y , f C and f Ŷ are given functions. 4 The exogenous variables Λ, U and V characterize the user-specified threshold, the expert's individual characteristics and the data generating process, respectively. The function f C is directly defined by Eq. 1, i.e., f C (Λ, X) = C Λ (X). Further, as argued elsewhere [21] , we can always find a distribution for the exogenous variables Λ, U and V as well as a functional form for the functions f X , f Y and f Ŷ such that the distributions of the features, the ground-truth label and the expert's prediction introduced in Section 2 are given by the observational distribution entailed by the SCM M. For ease of exposition, we assume that, under no interventions, the distribution of the exogenous variable Λ is P (Λ) = 1{Λ = 1} and thus human experts make predictions on their own. Figure 1 shows a visual representation of our SCM M.Building upon the above characterization, we are now ready to formalize the following notion of counterfactual harm, which essentially compares the accuracy of a factual prediction made by an expert on their own against the counterfactual prediction that the same expert would have made had they used a decision support system C λ :Definition 1 (Counterfactual Harm) For any x, y, ŷ ∼ P M , the counterfactual harm that a decision support system C λ would have caused, if deployed, is given by5 EQUATIONwhere do(Λ = λ) denotes a (hard) intervention on the exogenous variable Λ.Here, note that counterfactual harm can only be nonzero if the expert has made a successful prediction on their own, i.e., ŷ = y. Otherwise, the expert's prediction ŷ could not have become worse had they used the decision support system C λ .Given the above definition of counterfactual harm and a user-specified bound α ∈ [0, 1], our goal is to find the largest harm-controlling set of values Λ(α) ⊆ [0, 1] such that, for each λ ∈ Λ(α), it holds that the counterfactual harm is, in expectation across all possible instances, smaller than α, i.e.,EQUATIONAt this point, we cannot expect to find the set Λ(α) because counterfactual harm lies within level three in the "ladder of causation" [21] and thus it is not identifiable from observational data without further assumptions. However, in what follows, we will show that, under certain assumptions, the average counterfactual harm H(λ) is (partially) identifiable, i.e., it can be estimated (bounded) using observational data.Comparison to Richens's definition of counterfactual harm. Richens et al. [1] define counterfactual harm as follows:h λ (x, y, ŷ) = E Ŷ ∼P M| Ŷ = ŷ,X=x,Y =y ( Ŷ ) [max{0, 1{ Ŷ = y} -1{ŷ = y}}],where x, y, ŷ ∼ P M;do(Λ=λ) and λ = 1 is considered to be the default action in the language of Richens et al. [1] . This definition implicitly assumes that the system C λ is deployed and it compares the factual prediction ŷ made by an expert using the deployed system against the counterfactual prediction Ŷ had the expert made on their own. On the contrary, our definition does not assume that the system C λ is deployed and instead it compares the factual prediction ŷ made by an expert on their own against the counterfactual prediction Ŷ had the expert made using the system C λ . 
counterfactual harm under counterfactual and interventional monotonicity: 
 In this section, we analyze counterfactual harm h λ (x, y, ŷ), as defined in Eq. 4, from the perspective of causal identifiability under two natural monotonicity assumptions-counterfactual monotonicity and interventional monotonicity. Both of these assumptions, which were first studied by Straitouri et al. [18] , formalize the intuition that increasing the number of label values in a prediction set increases its difficulty.Under counterfactual monotonicity, for any x ∈ X and λ, λ′ ∈ [0, 1] such that Y ∈ C λ (x) ⊆ C λ ′ (x),if an expert has succeeded at predicting Y using C λ ′ , they would have also succeeded had they used C λ and, conversely, if they have failed at predicting Y using C λ , they would have also failed had they used C λ ′ , while holding "everything else fixed". More formally, counterfactual monotonicity is defined as follows:Assumption 1 (Counterfactual Monotonicity) Counterfactual monotonicity holds if and only if, for any x ∈ X and any λ,EQUATIONfor any u ∼ P M (U ) and v ∼ P M (V | X = x).Under interventional monotonicity, for any x ∈ X and λ, λ′ ∈ [0, 1] such that Y ∈ C λ (x) ⊆ C λ ′ (x),the probability that experts succeed at predicting Y using C λ is equal or greater than using C λ ′ . More formally, interventional monotonicity is defined as follows 6 :Assumption 2 (Interventional Monotonicity) Interventional monotonicity holds if and only if, for any x ∈ X , y ∈ Y, and λ, λ ′ ∈ [0, 1] such that y ∈ C λ (x) ⊆ C λ ′ (x), we have thatEQUATIONwhere the probability is over the exogenous random variables U and V characterizing the expert's individual characteristics and the data generating process, respectively.In what follows, we first show that, under the counterfactual monotonicity assumption, counterfactual harm is identifiable (we provide all proofs in Appendix B):Proposition 1 Under the counterfactual monotonicity assumption, for any x, y, ŷ ∼ P M , the counterfactual harm that a decision support system C λ would have caused, if deployed, is given byEQUATIONAs an immediate consequence, we can conclude that, under counterfactual monotonicity, the average counterfactual harm H(λ), as defined in Eq. 5, is identifiable and can be estimated using observational data sampled from P M . However, since the inequality condition of the counterfactual monotonicity assumption compares counterfactual predictions and thus cannot be experimentally verified, one should be cautious about using the above proposition to estimate counterfactual harm in high-stakes applications.Next, we show that, under the interventional monotonicity assumption, the average counterfactual harm is partially identifiable:Proposition 2 Under the interventional monotonicity assumption, the average counterfactual harm H(λ) that a decision support system C λ would have caused, if deployed, satisfies thatEQUATIONImportantly, note that, in the above proposition, the lower bound on the left hand side of Eq. 9 matches the average counterfactual harm under the counterfactual monotonicity assumption and thus, holding "everything else fixed", the average counterfactual harm under interventional monotonicity is always greater or equal than the average counterfactual harm under counterfactual monotonicity. Moreover, further note that the inequality condition of the interventional monotonicity assumption compares interventional probabilities and thus we can experimentally verify it (see Appendix F), lending support to using the above proposition to bound average counterfactual harm in high-stakes applications. 
controlling counterfactual harm using conformal risk control: 
 In this section, we develop a computational framework that, given a decision support system C λ and a user-specified bound α, aims to find the largest harm-controlling set Λ(α), as defined in Eq. 5. In the development of our framework, we will first assume that counterfactual monotonicity holds and, later on, we will relax this assumption, and assume instead that interventional monotonicity holds.Our framework builds upon the the idea of conformal risk control, which has been introduced very recently by Angelopoulos et al. [22] . Given any monotone loss function ℓ(C λ (X), Y ) with respect to λ and a calibration set {(X i , Y i )} n i=1 , with (X i , Y i ) ∼ P (X, Y ), conformal risk control finds a value of λ under which the expected loss of a test sample (X n+1 , Y n+1 ) ∼ P (X, Y ) does not exceed a user-specified bound α, i.e., E[ℓ(C λ (X n+1 ), Y n+1 )] ≤ α. However, in our framework, we re-define the loss ℓ so that it does not only depend on the prediction set and the label value but also on the expert's prediction on their own. Under the counterfactual monotonicity assumption, we set the value of the loss ℓ using the expression of counterfactual harm in Eq. 8 and, using a similar proof technique as in Angelopoulous et al., first prove the following theorem:Theorem 1 Let D = {(X i , Y i , Ŷi )} n i=1 be a calibration set, with (X i , Y i , Ŷi ) ∼ P M (X, Y, Ŷ ), α ∈ [0, 1]be a user-specified bound, andλ(α) = inf λ : n n + 1 Ĥn (λ) + 1 n + 1 ≤ α where Ĥn (λ) = n i=1 1{ Ŷi = Y i ∧ Y i / ∈ C λ (X i )} n . (10) If counterfactual monotonicity holds, a test sample (X n+1 , Y n+1 , Ŷn+1 ) ∼ P M (X, Y, Ŷ ) satisfies that E 1{ Ŷn+1 = Y n+1 ∧ Y n+1 / ∈ C λ(α) (X n+1 )} ≤ α,where the expectation is over the randomness in the calibration set used to compute the threshold λ(α) and the test sample.Then, we leverage the above theorem and the fact that, under counterfactual monotonicity, the counterfactual harm is nonincreasing with respect to λ, as shown in Lemma 3 in Appendix B.5, to recover the largest harm-controlling set Λ(α):Corollary 1 Let α ∈ [0, 1] be a user-specified bound. Then, under the counterfactual monotonicity assumption, it holds that Λ(α) = {λ ∈ [0, 1] | λ ≥ λ(α)}, where λ(α) is given by Eq. 10.Under the interventional monotonicity assumption, rather than directly controlling the counterfactual harm, we will control the upper bound given by Eq. 9. Consequently, rather than recovering the largest harm-controlling set Λ(α), we will recover a harm-controlling set Λ ′ (α) ⊆ Λ(α). However, since the expression inside the expectation of the upper bound is nonmonotone with respect to λ, we cannot directly use it to set the value of the loss ℓ in conformal risk control. That said, since the first term of the expression is nonincreasing and the second term is nondecreasing, as shown in Lemmas 3 and 4 in Appendix B.5, we can apply conformal risk control separately for each term. For the first term, we use Theorem 1 because the term matches the counterfactual harm under the counterfactual monotonicity assumption. For the second term, we prove the following theorem:Theorem 2 Let D = {(X i , Y i , Ŷi )} n i=1 be a calibration set, with (X i , Y i , Ŷi ) ∼ P M , α ∈ 1 n+1, 1 be a user-specified bound, andλ(α) = sup λ : n n + 1 Ĝn (λ) + 1 n + 1 ≤ α where Ĝn (λ) = n i=1 1{ Ŷi ̸ = Y i ∧ Y i ∈ C λ (X i )} n .(11) If interventional monotonicity holds and λ exists, a test sample(X n+1 , Y n+1 , Ŷn+1 ) ∼ P M (X, Y, Ŷ ) satisfies that E 1{ Ŷn+1 ̸ = Y n+1 ∧ Y n+1 ∈ Cλ (α) (X n+1 )} ≤ α,where the expectation is over the randomness in the calibration set used to compute the threshold λ(α) and the test sample (X n+1 , Y n+1 , Ŷn+1 ).Finally, we leverage the above theorems and the fact that the first term inside the expectation of the upper bound of counterfactual harm in Eq. 9 is nonincreasing and the second term is nondecreasing with respect to λ to recover a harm-controlling set Λ ′ (α) ⊆ Λ(α):Corollary 2 Let α ∈ [0, 1] be a user-specified bound. Then, under the interventional monotonicity assumption, for any choice of α ′ ≤ α, the setΛ ′ (α) = {λ ∈ [0, 1] | λ(α ′ ) ≤ λ ≤ λ(α -α ′ )},where λ(α ′ ) is given by Eq. 10 and λ(α -α ′ ) is given by Eq. 11 satisfies that Λ ′ (α) ⊆ Λ(α).Note that, in practice, the above corollaries can be implemented efficiently since, by definition, the functions Ĥn (λ) and Ĝn (λ) are piecewise constant functions of λ with n different pieces. 
experiments: 
 In this section, we use data from two different human subject studies to: a) evaluate the average counterfactual harm caused by decision support systems based on prediction sets; b) validate the theoretical guarantees offered by our computational framework (i.e., Corollaries 1 and 2); c) investigate the trade-off between the average counterfactual harm caused by decision support systems based on prediction sets and the average accuracy achieved by human experts using these systems. In what follows, we assume that counterfactual monotonicity holds. In Appendix G, we conduct experiments where we relax this assumption and assume instead that interventional monotonicity holds. 7Experimental setup. We first experiment with the ImageNet16H dataset by Steyvers et al. [52] , which comprises 32,431 predictions made by 145 human participants on their own about noisy Here, the amount of phase noise controls the difficulty of the classification task-the higher the noise, the more difficult the classification task. In our experiments, we use (all) the noisy images with noise value ω ∈ {80, 95, 110} because, for the noisy images with ω = 125, humans perform poorly; moreover, we stratify these images (and human predictions) with respect to their amount of phase noise.For each stratum of images, we apply our framework to decision support systems C λ with different pre-trained classifiers, namely VGG19 [54] , DenseNet161 [55] , GoogleNet [56] , ResNet152 [57] and AlexNet [58] after 10 epochs of fine-tuning, as provided by Steyvers et al. [52] . 8 To this end, we randomly split the images (and human predictions) into a calibration set (10%), which we use to find the harm-controlling sets Λ(α) by applying Corollary 1, and a test set (90%), which we use to estimate the average counterfactual harm H(λ) caused by the decision support systems C λ as well as the average accuracy A(λ) of the predictions made by a human expert using C λ . Here, since the dataset only contains predictions made by human participants on their own, we use the mixture of multinomial logit models (MNLs) introduced by Straitouri et al. [17] to estimate the average accuracy A(λ). Refer to Appendix C for more details regarding the mixture of MNLs and to Appendix D for additional experiments studying the relationship between average counterfactual harm H(λ), average prediction set size and empirical coverage 9 .Then, we experiment with the ImageNet16H-PS dataset10 by Straitouri et al. [18] , which comprises 194,407 predictions made by 2,751 human participants using decision support systems C λ about the set of noisy images with ω = 110 described above. More specifically, for each noisy image, the dataset contains human predictions made under any possible prediction set that can be constructed using Eq. 1 with (the softmax output of) VGG19 after 10 epochs of fine-tuning, as provided by Steyvers et al. [52] . Here, similarly as in the ImageNet16H dataset, we randomly split the images (and human predictions) into a calibration set (10%), which we use to find Λ(α), and a test set (90%), which we use to estimate H(λ) and A(λ). However, in this case, we can estimate A(λ) using the predictions made by human participants using C λ from the dataset, and we can compare this empirical estimate to the one using the mixture of MNLs.In both datasets, we calculate confidence intervals and validate the theoretical guarantees offered by Corollary 1 by repeating each experiment 50 times and, each time, sampling different calibration and test sets.Results. Figure 2 shows the average accuracy A(λ) achieved by a human participant using C λ , as predicted by the mixture of MNLs, against the average counterfactual harm H(λ) caused by C λ for α ∈ {0.01, 0.05} on the stratum of images with ω = 110 from the ImageNet16H dataset. Refer to Appendix D for results on other strata. Here, each point corresponds to a different λ value and its coloring indicates the empirical probability that λ is included in the harm-controlling set Λ(α). The results show several interesting insights. First, we find that, as long as λ < 1, the decision support systems C λ always cause some amount of counterfactual harm 11 . This suggests that, while restricting human agency enables human-AI complementarity, it inevitably causes (some amount of) counterfactual harm. Second, we find that the sets of λ values provided by our framework are typically harm-controlling, i.e., they do not include λ values such that H(λ) > α. However, the sets are often conservative and do not include all the harm-controlling λ values due to estimation error in the the empirical estimate Ĥn (λ) of the average counterfactual harm using data from the calibration set. In Appendix D, we show that using larger calibration sets reduces the above mentioned estimation error and results in sets of λ values that are less conservative. Third, we find that there is a trade-off between accuracy and counterfactual harm and this trade-off is qualitatively consistent across decision support systems using different pre-trained classifiers. In fact, for α = 0.01, the decision support systems C λ * offering the greatest average accuracy A(λ * ) are not harm-controlling.Figure 3 shows the average accuracy A(λ) achieved by a human participant using C λ , as estimated using predictions made by human participants using C λ , against the average counterfactual harm H(λ) caused by C λ for α ∈ {0.01, 0.05} on the ImageNet16H-PS dataset. Here, the meaning and coloring of each point is the same as in Figure 2 . The results also support the findings derived from the experiments with the ImageNet16H dataset-the decision support systems C λ always cause some amount of harm, our framework succeeds at identifying harm-controlling sets, and there is a trade-off between accuracy and counterfactual harm. Further, they also show that, while there is a gap between the average accuracy estimated using the mixture of MNLs and the average accuracy estimated using predictions made by human participants using C λ , they follow the same trend and support the same qualitative conclusions. 
discussion and limitations: 
 In this section, we highlight several limitations of our work, discuss its broader impact, and propose avenues for future work.Assumptions. We have assumed that the data samples and the expert predictions are drawn i.i.d. from a fixed distribution and the calibration set contains samples with noiseless ground truth labels and expert label predictions. It would be very interesting to extend our framework to allow for distribution shifts and label noise. Moreover, we have considered prediction sets constructed with a fixed user-specified threshold value λ. In light of recent work by Gibbs et al. [59] , it would be interesting to extend our framework to allow for threshold values that depend on the data samples. Furthermore, in our definition of counterfactual harm we have treated all inaccurate predictions as equally harmful. Expanding our definition of harm to weigh different inaccurate predictions according to their consequences would be a very interesting avenue for future work. In addition, under the interventional monotonicity assumption, we have empirically observed that the gap between our lower-and upper-bounds is often large. Therefore, it would be useful to identify other natural assumptions under which this gap is smaller.Methodology. In our framework, the prediction sets are constructed using the softmax output of a pretrained classifier. However, we hypothesize that, by accounting for the similarity between the mistakes made by humans and those made by the pre-trained classifier, we may be able to construct prediction sets that cause counterfactual harm less frequently. Moreover, we have focused on controlling the average counterfactual harm. However, whenever the expert's predictions are consequential to individuals, this may lead significant disparities across demographic groups. Therefore, it would be important to extend our framework to account for fairness considerations.Evaluation. Our experimental evaluation comprises a single benchmark dataset of noisy natural images and thus one may question the generalizability of the conclusions we draw from our results. To overcome this limitation, it would be important to further investigate the trade-off between accuracy and counterfactual harm in decision support systems based on prediction sets in real-world application domains (e.g., medical diagnosis).Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: In Section 6, and Appendix D, Appendix E and Appendix G we report 95% confidence intervals, and in Appendix F we report standard errors.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
conclusions: 
 In this paper, we have initiated the study of counterfactual harm in decision support systems based on prediction sets. We have introduced a computational framework that, under natural monotonicity assumptions on the predictions made by experts using these systems, can control how frequently these systems cause harm. Moreover, we have validated our framework using data from two different human subject studies and shown that, in decision support systems based on prediction sets, there is a trade-off between accuracy and counterfactual harm.Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We report computer resources in Footnote 7.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The paper discusses both potential positive and negative societal impacts of the work in Section 7.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Using Pearl's notation [50] , our definition of counterfactual harm reads as follows:h λ (x, y, ŷ) = E[max{0, 1{ŷ = y} -1{ Ŷλ = y}}|X = x, Y = y, Ŷ1 = ŷ],where the subindices in Ŷλ and Ŷ1 denote (hard) interventions on the exogenous variable Λ, the random variables x, y, ŷ ∼ P (X, Y, Ŷ1 ), and the expectation is over the prediction Ŷλ made by the expert using the system C λ . Using the above notation, the definition does not explicitly specify the (counterfactual) distribution used in the expectation, in contrast to the definition using the notation by Peters et al. [51] , which explicitly specifies the counterfactual distribution used in the expectation, as restated below for clarity.h λ (x, y, ŷ) = E Ŷ ∼P M;do(Λ=λ) | Ŷ = ŷ,X=x,Y =y ( Ŷ ) [max{0, 1{ŷ = y} -1{ Ŷ = y}}].Given an observation (x, y, ŷ) under no intervention, we haveh λ (x, y, ŷ) = E Ŷ ∼P M;do(Λ=λ) | Ŷ = ŷ,X=x,Y =y [max{0, 1{ŷ = y} -1{ Ŷ = y}}] = E Ŷ ∼P M;do(Λ=λ) | Ŷ = ŷ,X=x,Y =y [max{0, 1 -1{ Ŷ = y}}] • 1{ŷ = y} + E Ŷ ∼P M;do(Λ=λ) | Ŷ = ŷ,X=x,Y =y [max{0, 0 -1{ Ŷ = y}}] • 1{ŷ ̸ = y} = E Ŷ ∼P M;do(Λ=λ) | Ŷ = ŷ,X=x,Y =y [max{0, 1 -1{ Ŷ = y}}] • 1{ŷ = y} + 0 = E Ŷ ∼P M;do(Λ=λ) | Ŷ = ŷ,X=x,Y =y [max{0, 1 -1}] • 1{ŷ = y} • 1{y ∈ C λ (x)} + E Ŷ ∼P M;do(Λ=λ) | Ŷ = ŷ,X=x,Y =y [max{0, 1 -0}] • 1{ŷ = y} • 1{y / ∈ C λ (x)} (12) = 0 • 1{ŷ = y} • 1{y ∈ C λ (x)} + E Ŷ ∼P M;do(Λ=λ) | Ŷ = ŷ,X=x,Y =y [max{0, 1 -0}] • 1{ŷ = y} • 1{y / ∈ C λ (x)} = 1 • 1{ŷ = y} • 1{y / ∈ C λ (x)} = 1{ŷ = y} • 1{y / ∈ C λ (x)} = 1{ŷ = y ∧ y / ∈ C λ (x)},where, in Eq. 12, we used the counterfactual monotonicity property by setting1{ Ŷ = y} = 1 if y ∈ C λ (x).Under interventional monotonicity, by Lemma 1, we haveh λ (x, y, ŷ) = E Ŷ ∼P M;do(Λ=λ) | Ŷ = ŷ,X=x,Y =y [1{ Ŷ ̸ = y}] • 1{ŷ = y} • 1{y ∈ C λ (x)} + 1{ŷ = y} • 1{y / ∈ C λ (x)} ≤ E Ŷ ∼P M;do(Λ=λ) | Ŷ = ŷ,X=x,Y =y [1{ Ŷ ̸ = y}] • 1{y ∈ C λ (x)} + 1{ŷ = y} • 1{y / ∈ C λ (x)},where we used that 1{ŷ = y} ≤ 1.By taking the expectation on both sides over ŷ under no intervention we haveEQUATIONEQUATIONEQUATIONsince 1{y ∈ C λ (x)} is a constant with respect to the expectation over Ŷ ′ ∼ P M .By Lemma 2, Eq. 13 becomesEQUATION)By Assumption 2 for λ ′ = 1 and any λ such that y ∈ C λ (x), we have thatP M;do(Λ=λ) ( Ŷ = Y | X = x, Y = y) ≥ P M ( Ŷ = Y | X = x, Y = y),which we can rewrite asE Ŷ ∼P M;do(Λ=λ) [1{ Ŷ = Y } | X = x, Y = y] ≥ E Ŷ ∼P M [1{ Ŷ = Y } | X = x, Y = y],and is equivalent toE Ŷ ∼P M;do(Λ=λ) [1{ Ŷ ̸ = Y } | X = x, Y = y] ≤ E Ŷ ∼P M [1{ Ŷ ̸ = Y } | X = x, Y = y].Using the above result, Eq. 17 becomesE Ŷ ′ ∼P M h λ (x, y, Ŷ ′ ) ≤ E Ŷ ∼P M 1{ Ŷ ̸ = Y } | X = x, Y = y • 1{y ∈ C λ (x)} + E Ŷ ′ ∼P M 1{ Ŷ ′ = y} • 1{y / ∈ C λ (x)} .If we now take the expectation in both sides over X, Y ∼ P M and use that Ŷ ′ is equal in distribution with Ŷ , we haveE X,Y, Ŷ ∼P M h λ (X, Y, Ŷ ) ≤ E X,Y, Ŷ ∼P M 1{ Ŷ ̸ = Y } • 1{Y ∈ C λ (X)} + E X,Y, Ŷ ∼P M 1{ Ŷ = Y } • 1{Y / ∈ C λ (X)} , that is equivalent to E X,Y, Ŷ ∼P M h λ (X, Y, Ŷ ) ≤ E X,Y, Ŷ ∼P M 1{ Ŷ ̸ = Y } • 1{Y ∈ C λ (X)} + 1{ Ŷ = Y } • 1{Y / ∈ C λ (X)} ,which we can write asE X,Y, Ŷ ∼P M h λ (X, Y, Ŷ ) ≤ E X,Y, Ŷ ∼P M 1{ Ŷ ̸ = Y ∧ Y ∈ C λ (X)} + 1{ Ŷ = Y ∧ Y / ∈ C λ (X)} .For the lower-bound, by Lemma 1, we haveh λ (x, y, ŷ) = E Ŷ ∼P M;do(Λ=λ) | Ŷ = ŷ,X=x,Y =y [1{ Ŷ ̸ = y}] • 1{ŷ = y} • 1{y ∈ C λ (x)} + 1{ŷ = y} • 1{y / ∈ C λ (x)} ≥ 1{ŷ = y} • 1{y / ∈ C λ (x)},where we used thatE Ŷ ∼P M;do(Λ=λ) | Ŷ = ŷ,X=x,Y =y [1{ Ŷ ̸ = y}] • 1{ŷ = y} • 1{y ∈ C λ (x)} ≥ 0.By taking the expectation in both sides over X, Y, Ŷ ∼ P M , we haveE X,Y, Ŷ ∼P M 1{ Ŷ = Y } • 1{Y / ∈ C λ (X)} ≤ E X,Y, Ŷ ∼P M h λ (X, Y, Ŷ ) , that is equivalent to E X,Y, Ŷ ∼P M 1{ Ŷ = Y ∧ Y / ∈ C λ (X)} ≤ E X,Y, Ŷ ∼P M h λ (X, Y, Ŷ ) ,thus we conclude the proof.We will show that 1{ Ŷi = Y i ∧ Y i / ∈ C λ (X i )} are exchangeable and that they satisfy the requirements of Theorem 1 in [22] for eachX i , Y i , Ŷi . Since X i , Y i , Ŷi are i.i.d., 1{ Ŷi = Y i ∧ Y i / ∈ C λ (X i )} are also i.i.d thus exchangeable. Considering each 1{ Ŷi = Y i ∧ Y i / ∈ C λ (X i )} for each X i , Y i , Ŷi ∈ Das a function of λ, it holds by Lemma 3 that for eachX i , Y i , Ŷi , 1{ Ŷi = Y i ∧ Y i / ∈ C λ (X i )} is non-increasing in λ and right-continuous. In addition, it holds that1{ Ŷi = Y i ∧ Y i / ∈ C 1 (X i )} = 0 ≤ α and that sup λ 1{ Ŷi = Y i ∧ Y i / ∈ C λ (X i )} ≤ 1 surely by definition.For the proof we will follow a similar procedure as in [22] .EQUATIONfor every λ ∈ L. Now, letEQUATIONIf such λ′ does not exist, by Eq. 19 λ will also not exist. If λ′ exists, we will show that λ ≤ λ′ . If λ = 0, then it will hold that λ′ ≥ λ = 0 = min λ∈L λ. If λ > 0, then by Eq. 19, λ must satisfyEQUATIONwhich means that λ ≤ λ′ . Therefore, in any case we have that λ ≤ λ′ .From Lemma 4, the random functions1{ Ŷi ̸ = Y i ∧ Y i ∈ C λ (X i )} are non-decreasing for each i ∈ [n + 1].As a result, since λ ≤ λ′ , we have thatEQUATIONwhere i = {1, . . . , n + 1}. Given that the data samples(X i , Y i , Ŷi ) for i ∈ [1, n + 1] are ex- changeable, the values 1{ Ŷi ̸ = Y i ∧ Y i ∈ C λ′ (X i )} for i ∈ [n + 1] are also exchangeable.Therefore, due to exchangeability and given that λ′ is fixed given(X i , Y i , Ŷi ) for i ∈ [1, n + 1], 1{ Ŷn+1 ̸ = Y n+1 ∧ Y n+1 ∈ C λ′ (X n+1)} has the same probability of taking any value inD λ′ = {1{ Ŷi ̸ = Y i ∧ Y i ∈ C λ′ (X i )}} n+1 i=1 , i.e., the value of 1{ Ŷi ̸ = Y i ∧ Y i ∈ C λ′ (X i )} for the n + 1-th sample in {(X i , Y i , Ŷi )} n+1i=1 follows a uniform distribution over all the possible values in D λ′ . As a result, from the law of total expectation it holds for i = {1, . . . , n + 1} thatE Xi,Yi, Ŷi∼P M [1{ Ŷn+1 ̸ = Y n+1 ∧ Y n+1 ∈ C λ′ (X n+1 )}] = E D λ′ [1{ Ŷn+1 ̸ = Y n+1 ∧ Y n+1 ∈ C λ′ (X n+1 )} | D λ′ ] = n+1 i=1 1{ Ŷi̸ =Yi∧Yi∈C λ′ (Xi)} n+1 = Ĝn+1 ( λ′ ).Finally, from Eq. 20 and the above we have (22) where i = {1, . . . , n + 1}.E Xi,Yi, Ŷi∼P M [1{ Ŷn+1 ̸ = Y n+1 ∧ Y n+1 ∈ C λ(X n+1 )}] ≤ E Xi,Yi, Ŷi∼P M [1{ Ŷn+1 ̸ = Y n+1 ∧ Y n+1 ∈ C λ′ (X n+1 )}] = Ĝn+1 ( λ′ ) ≤ α,Lemma 1 Under the interventional monotonicity assumption, for any x, y, ŷ ∼ P M , the counterfactual harm that a decision support system C λ would have caused, if deployed, is given byEQUATION)Proof. Given an observation (x, y, ŷ) under no intervention, we haveh λ (x, y, ŷ) = E Ŷ ∼P M;do(Λ=λ) | Ŷ = ŷ,X=x,Y =y [max{0, 1{ŷ = y} -1{ Ŷ = y}}] = E Ŷ ∼P M;do(Λ=λ) | Ŷ = ŷ,X=x,Y =y [max{0, 1 -1{ Ŷ = y}}] • 1{ŷ = y} + E Ŷ ∼P M;do(Λ=λ) | Ŷ = ŷ,X=x,Y =y [max{0, 0 -1{ Ŷ = y}}] • 1{ŷ ̸ = y} = E Ŷ ∼P M;do(Λ=λ) | Ŷ = ŷ,X=x,Y =y [max{0, 1 -1{ Ŷ = y}}] • 1{ŷ = y} + 0 = E Ŷ ∼P M;do(Λ=λ) | Ŷ = ŷ,X=x,Y =y [max{0, 1 -1{ Ŷ = y}}] • 1{ŷ = y} • 1{y ∈ C λ (x)} + E Ŷ ∼P M;do(Λ=λ) | Ŷ = ŷ,X=x,Y =y [max{0, 1 -1{ Ŷ = y}}] • 1{ŷ = y} • 1{y / ∈ C λ (x)} = E Ŷ ∼P M;do(Λ=λ) | Ŷ = ŷ,X=x,Y =y [max{0, 1{ Ŷ ̸ = y}}] • 1{ŷ = y} • 1{y ∈ C λ (x)} + E Ŷ ∼P M;do(Λ=λ) | Ŷ = ŷ,X=x,Y =y [max{0, 1 -0}] • 1{ŷ = y} • 1{y / ∈ C λ (x)} = E Ŷ ∼P M;do(Λ=λ) | Ŷ = ŷ,X=x,Y =y [max{0, 1{ Ŷ ̸ = y}}] • 1{ŷ = y} • 1{y ∈ C λ (x)} + 1 • 1{ŷ = y} • 1{y / ∈ C λ (x)} = E Ŷ ∼P M;do(Λ=λ) | Ŷ = ŷ,X=x,Y =y [1{ Ŷ ̸ = y}] • 1{ŷ = y} • 1{y ∈ C λ (x)} + 1{ŷ = y} • 1{y / ∈ C λ (x)}Lemma 2 Under the structural causal model M satisfying Eq. 3, given an observation (x, y) it holdsEQUATIONProof. Given an observation (x, y, ŷ) under no intervention, for the expectation of the prediction of the human Ŷ ∼ P M;do(Λ=λ) | X=x,Y =y, Ŷ =ŷ it holdsE Ŷ ∼P M;do(Λ=λ) | X=x,Y =y, Ŷ = ŷ 1{ Ŷ ̸ = Y } = u,v P (U = u, V = v | X = x, Y = y, Ŷ = ŷ) • 1{f Ŷ (u, v, C λ (x)) ̸ = y},whereP (U = u, V = v | X = x, Y = y, Ŷ = ŷ)is the posterior distribution of the exogenous variables U, V conditional on the observation (x, y, ŷ).By taking the expectation of the above with respect to the human prediction ŷ under no intervention we haveE Ŷ ′ ∼P M E Ŷ ∼P M;do(Λ=λ) | X=x,Y =y, Ŷ = Ŷ ′ 1{ Ŷ ̸ = Y } = ŷ P M ( Ŷ ′ = ŷ | X = x, Y = y) • u,v P (U = u, V = v | X = x, Y = y, Ŷ = ŷ) • 1{f Ŷ (u, v, C λ (x)) ̸ = y} = u,v ŷ P (U = u, V = v | X = x, Y = y, Ŷ = ŷ) • P M ( Ŷ ′ = ŷ | X = x, Y = y) • 1{f Ŷ (u, v, C λ (x)) ̸ = y}.Since Ŷ ′ is equal in distribution to Ŷ we can rewrite the above asE Ŷ ′ ∼P M E Ŷ ∼P M;do(Λ=λ) | X=x,Y =y, Ŷ = Ŷ ′ 1{ Ŷ ̸ = Y } = u,v ŷ P (U = u, V = v | X = x, Y = y, Ŷ = ŷ) • P M ( Ŷ = ŷ | X = x, Y = y) • 1{f Ŷ (u, v, C λ (x)) ̸ = y} and since P (U = u, V = v | X = x, Y = y) = ŷ P (U = u, V = v | X = x, Y = y, Ŷ = ŷ) • P M ( Ŷ = ŷ | X = x, Y = y), we have E Ŷ ′ ∼P M E Ŷ ∼P M;do(Λ=λ) | X=x,Y =y, Ŷ = Ŷ ′ 1{ Ŷ ̸ = Y } = u,v P (U = u, V = v | X = x, Y = y) • 1{f Ŷ (u, v, C λ (x)) ̸ = y} = u P (U = u | X = x, Y = y) • v P (V = v | X = x, Y = y) • 1{f Ŷ (u, v, C λ (x)) ̸ = y} = E Ŷ ∼P M;do(Λ=λ) [1{ Ŷ ̸ = Y } | X = x, Y = y],where we used that the exogenous variables U and V are independent.Lemma 3 Under counterfactual monotonicity, for a given x, y, ŷ the counterfactual harm h λ (x, y, ŷ) is non-increasing as a function of λ and right-continuous.Proof. If y ∈ C λ (x) ∀λ ∈ L then h λ (x, y, ŷ) = 0, ∀λ ∈ Λ given Eq. 8. If ∃ λ ∈ Λ such that λ = inf{λ ∈ Λ : y ∈ C λ (x)}, then ∀λ ≥ λ, it holds that y ∈ C λ (x), and as a result h λ (x, y, ŷ) = 0.For ∀λ ′ < λ, it holds that y / ∈ C λ ′ (x) and as a result Eq. 8 becomesEQUATIONwhere 1{ŷ = y} is constant ∀λ ′ < λ.Lemma 4 For a given data sample x with label y, let G : L → {0, 1} be the functionG(λ) = 1{ Ŷ ̸ = y ∧ y ∈ C λ (x)}. Under interventional monotonicity, G is non-decreasing in λ.Proof.Let λ, λ ′ ∈ L such that λ ≤ λ ′ . Then it will always hold that C λ (x) ⊆ C λ ′ (x). We distinguish the following cases for C λ (x), C λ ′ (x):• y / ∈ C λ ′ (x). Then G(λ) = G(λ ′ ) = 0. • y / ∈ C λ (x) and y ∈ C λ ′ (x). Then, G(λ) = 0 and G(λ ′ ) = 1{ Ŷ ̸ = y} ≥ 0 = G(λ) • y ∈ C λ (x) ⊆ C λ ′ (x). Then, G(λ) = G(λ ′ ) = 1{ Ŷ ̸ = y}.In this section, we evaluate our methodology on additional strata of images and on larger calibration sets, and study the relationship between average counterfactual harm, average prediction set size and empirical coverage.Other strata of images. Figures 4 and 5 show the average accuracy A(λ) achieved by a human expert using C λ , as predicted by the mixture of MNLs, against the average counterfactual harm H(λ) caused by C λ for α ∈ {0.01, 0.05} on the strata of images with ω = 80 and ω = 95, respectively. The meaning and coloring of each point are the same as in Figure 2 . The results on both strata of images are consistent with the results on the stratum of images with ω = 110-the decision support systems C λ always cause some amount of counterfactual harm, our framework successfully identifies harm-controlling sets and there is a trade-off between accuracy and counterfactual harm. However, for the strata with ω ∈ {80, 95}, where the classification task has lower difficulty compared to the stratum with ω = 110, we observe that the trade-off between accuracy and counterfactual is minimal. In both panels, 95% confidence intervals have width always below 0.02 and are represented using shaded areas.Larger calibration sets. Figure 6 shows the average accuracy A(λ) achieved by a human expert using C λ , as predicted by the mixture of MNLs, against the average counterfactual harm H(λ) caused by C λ for α = 0.01 on the stratum of images with ω = 110. The results show that the larger the calibration set, the more often λ values with average counterfactual harm H(λ) close to α12 are included in the harm-controlling set Λ(α). This is because, the larger the number of samples in the calibration set (n), the lower the estimation error in the empirical estimate of the average counterfactual harm Ĥn . We find qualitatively similar results for the other strata of images.Average prediction set-size and empirical coverage. The results show that systems with higher coverage construct larger prediction sets in average and cause less counterfactual harm. We find similar results for the other strata of images. Each point corresponds to a λ value from 0 to 1 with step 0.001. Each row corresponds to decision support systems C λ with a different pre-trained classifier with average accuracies as in Figure 6 above. The average accuracy achieved by the simulated human experts on their own is 0.771. The results are averaged across 50 random samplings of the test and calibration set. In both panels, 95% confidence intervals are represented using shaded areas and always have width below 0.025.In this section, we experiment with a system that constructs the prediction set C(x) using a more complex set-valued predictor [49] . This set-valued predictor first computes the following nonconformity score for each label value y ∈ Y using the softmax output of a pre-trained classifier m y (x) ∈ [0, 1]:EQUATIONwhere y (i) denotes the label value with the i-th largest classifier output, u ∼ U (0, 1) is a uniform random variable and w is a hyperparameter representing the weight of the ranking of the label value.Then, given a user-specified threshold λ ∈ R, it constructs the prediction sets C(x) = C λ (x) as follows:EQUATIONwhere, with a slight abuse of notation, y (i) is the label value with the i-th largest s(y, x, u; w) value.Here, we should note that may exist data samples with s(y, x, u; w) > 1. Therefore, considering λ ∈ [0, 1] may not include every set-valued predictor given by Eq. 27. For this reason, in what follows, we consider λ ∈ [0, max s(y, x, u, w)], where the maximum is essentially over the classifier output m y (x), the uniform random variable u and the hyperparameter w.Experimental setup. We use the ImageNet16H dataset and follow the same setup as described in Section 6, with only difference that we split the images into a calibration set (10%), which we use to find Λ(α), a validation set (10%), which we use to select the value of the hyperparameter w following a similar procedure as in Huang et al. [49] , and a test set (80%). In each iteration of the experiment, given a λ value, we select the value of w ∈ {0.02, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35} for which the set-valued predictor achieves the smallest average prediction set-size over the data samples in the validation set.Results. The meaning and coloring of each point are the same as in Figure 2 . The results on all strata of images are consistent with the results using the set-valued predictor given by Eq. 1-the decision support systems C λ always cause some amount of counterfactual harm, our framework successfully identifies harm-controlling sets and there is a trade-off between accuracy and counterfactual harm. shows the empirical coverage for each decision support system C λ against the average counterfactual harm H(λ) caused by C λ on the stratum of images with ω = 110. Our results are consistent with the results for the set-valued predictor given by Eq. 1 in Figure 7 -systems C λ that achieve higher coverage, construct larger prediction sets on average and cause less harm. In this section, we use the ImageNet16H-PS [18] dataset to verify the interventional monotonicity assumption, i.e., whether human experts achieve higher average accuracy when predicting from smaller prediction sets that include the ground truth label. To this end, we follow a procedure, similar to the one used by Straitouri et al. [18] to verify the interventional monotonicity assumption unconditionally of the value of the ground truth label.Straitouri et al. estimate the average accuracy per prediction set size on images with similar difficulty, averaged across all experts and across experts with the same level of competence. They consider images with similar difficulty instead of single images, as the dataset ImageNet16H-PS does not include enough human predictions per image to faithfully estimate the average accuracy per image. They stratify the images into groups of similar difficulty, following the same procedure used in Straitouri et al. [17] based on different quantiles of the average accuracy values of all images. In our work, we stratify images with the same ground truth label into the following four groups:-High difficulty: images with average accuracy within the 0.25 quantile of the average accuracy values of all images with the same ground truth label. -Medium to high difficulty: images with average accuracy within the 0.5 quantile and outside the 0.25 quantile of the average accuracy values of all images with the same ground truth label. -Medium to low difficulty: images with average accuracy within the 0.75 quantile and outside the 0.5 quantile the of the average accuracy values of all images with the same ground truth label. -Low difficulty: images with average accuracy outside the 0.75 quantile of the average accuracy values of all images with the same ground truth label.We follow a similar method to stratify experts based on their level of competence into two groups for each ground truth label. To measure the level of competence of an expert for a ground truth label, we use the average accuracy across all the predictions that she made on images with this ground truth label. For each ground truth label, we consider the 50% of experts with the highest average accuracy as the experts with high level of competence and the rest 50% as experts with low level of competence.Figure 12 shows the average accuracy per prediction set size-for prediction sets that include the ground truth label-across images of various levels of difficulty and experts with different levels of competence for some of the ground truth labels. We find qualitatively similar results for the rest of the ground truth labels. The results are consistent with the results on the unconditional interventional monotonicity by Straitouri et al. [18] , showing that as long as the classification task is not too easy, the experts achieve higher average accuracy when predicting from smaller prediction sets, i.e., the interventional monotonicity holds.In this section, we show the average accuracy A(λ) achieved by a human expert using C λ , as predicted by the mixture of MNLs, against the average counterfactual harm upper-bound caused by C λ for several α values on the strata of images with ω = 80 in Figure 13 , with ω = 95 in Figure 14 and with ω = 110 in Figure 15 . Here, each point corresponds to a different λ value and its coloring indicates the empirical probability that λ is included in the harm-controlling set Λ ′ (α) given by Corollary 2, for a fixed α ′ value across the random samplings of the test and calibration sets. For each classifier and stratum of images, we select the α ′ value maximizing the expected size of the harm-controlling set Λ ′ (α). The results on all strata of images support the findings derived from the experiments assuming that the counterfactual monotonicity assumption holds. In addition, we observe that the gap between the lower-bound of the average counterfactual harm (that is equal to the average counterfactual harm under the counterfactual monotonicity assumption) and the upper-bound is often large. Justification: We specify all the details in Section 6 and Appendix C.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).The European Unions' AI act mentions the term "harm" more than 35 times and points out that, its crucial role in the design of algorithmic systems must be defined carefully.We denote random variables with capital letters and realizations of random variables with lowercase letters.The assumption that my(x) ∈ [0, 1] and λ ∈ [0, 1] is without loss of generality.The functions fX , fY , fC and f Ŷ are causal mechanisms and not equations that can be manipulated[50].To denote interventions in a counterfactual distribution, we follow the notation by Peters et al.[51]. Refer to Appendix A for a comparison to Pearl's notation[50].In Straitouri et al.[18], interventional monotonicity is originally defined unconditionally of the value of ground-truth label Y .All experiments ran on a Mac OS machine with an M1 processor and 16GB Memory. An open-source implementation of our methodology is publicly available at https://github.com/Networks-Learning/controllingcounterfactual-harm-prediction-sets.All classifiers and images are publicly available at https://osf.io/2ntrf.By empirical coverage, we refer to the fraction of the test samples for which the prediction sets include the ground truth label value.The dataset is publicly available at https://github.com/Networks-Learning/counterfactual-prediction-sets/.For λ = 1, the decision support system C λ does not cause harm because the prediction sets always contain all label values and thus human participants always make predictions on their own.Here, note that for α = 0.01, these λ values happen to achieve the highest average accuracy A(λ). 


Decision: All four reviewers were on the side of acceptance before the author rebuttal and discussion. However, one reviewer became more negative over the course of a long discussion with the authors and myself, although their score remains at borderline accept. I think that the reason for this more negative sentiment, while valid, does not outweigh the strengths of the submission, as I discuss below.

Reviewers found strengths in all aspects of the work. At the conceptual level, reviewers appreciated the formal definition of counterfactual harm caused by a prediction set. The identification of monotonicity assumptions (both a stronger and a weaker one) needed to estimate counterfactual harm is a strength in terms of theory. A method of controlling counterfactual harm is proposed, and the experiments with human subjects were also seen as a strength. Lastly, reviewers found the submission to be clearly written despite the technicality.

The rebuttal satisfactorily addressed several issues in my view, including practical applications of prediction sets, the "quality" of $\lambda$'s determined to be harm-controlling (a limitation of conformal risk control alleviated by a larger calibration set), models having worse performance than humans, confusion over causal inference notation, and an unclear relationship with the work of Richens et al. Clarifications and improvements to the paper are within the scope of a camera-ready revision.

Reviewer 8Whm engaged in a long discussion with the authors and later with myself. While part of this discussion was on more technical points, the most serious concern for this reviewer in the end was the question of comparing with additional state-of-the-art conformal predictors, in terms of counterfactual harm, coverage (conformal accuracy), and prediction set size. I think the discussion clarified somewhat that conformal predictors and the proposed method play complementary roles, where the latter provides more refined estimates of counterfactual harm for the former. Nonetheless, the reviewer wanted to know how the proposed method of choosing $\lambda$ affects conformal accuracy and prediction set size, for additional conformal predictors. I think this is a valid request. However, I do not see it as a core concern for the human-AI setting considered in the paper, where the prediction set is provided as an intermediate quantity for the human to select from, not as a final output, and human accuracy and counterfactual harm appear to be more relevant metrics. Thus, I do not think this concern should hold the submission back.  I do encourage the authors to follow through on their promise of experimenting with additional conformal predictors and to report all metrics, including conformal accuracy and prediction set size.

Reviews: review: paper_summary: paper_summary: This paper relates to decision support systems based on prediction sets. It introduces a framework using conformal risk control to design prediction sets that balance accuracy with minimizing potential harm (understood as a human who has succeeded at predicting the ground-truth label of an instance on their own failing had they used the DS system), validated with human prediction data from empirical lab studies.
strength_weakness: strength_weakness: 1. The main weakness is the usability and motivation of this particular formulation. The paper proposes decision support systems based on prediction sets, which might not always be feasible or realistic, especially in complex domains like healthcare. Most current systems are based on risk prediction e.g. disease risk, drop out risk, for binary outcomes. I don't see any good examples of problems where prediction sets are the right approach, and this paper also does not do this work of illustrating or justifying this formulation using real world case studies.

2. Discussing the Limitations of Predictive Accuracy. While the paper discusses improving accuracy through prediction sets, there may be fundamental limitations in predicting outcomes accurately in the first place. Have the authors considered recent research or critiques regarding the actionability of predictive models? For instance, how does the work align with concerns raised by Wang et al (2024) "Against Predictive Optimization" and Liu et al. (2024) "On the actionability of..."?
questions: questions: 1. Are there practical examples of decision support systems based on prediction sets being successfully implemented in healthcare or other social domains? How do these systems handle the inherent complexity and variability of real-world decision-making scenarios, such as patient care pathways or policy decisions?

2. Have the authors considered recent research or critiques regarding the actionability of predictive models? For instance, how does the work align with concerns raised by Wang et al (2024) "Against Predictive Optimization" and Liu et al. (2024) "On the actionability of..."?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper focuses on decision support designed to help humans in multi-class classification tasks via prediction sets. The paper defines the notion of counterfactual harm, namely when usage the system would lead the agent to a wrong prediction which would have been correct without the system’s ‘help’. The authors show that under some monotonicity assumptions such counterfactual harm is identifiable or partially identifiable. Against this theoretical backdrop, the paper proposes a method to find sets of interventions (namely thresholds for the prediction sets) which are guaranteed to have bounded harm. This approach is validated using experiments on human subject studies. These studies reveal a trade off between post-intervention accuracy and counterfactual harm.
strength_weakness: strength_weakness: 1.      The notation little y-hat and capital Y-hat in section 3 is somewhat confusing. Line 123 implicitly suggests the latter is the random variable generated by the user predicting with the decision support. Line 142 however states that little y-hat is the prediction the experts would make on their own. I would have expected this to be just   Y-hat but with lambda = 1, as stated elsewhere in the text?
2. In the first experiment, when w is different from 110, the data contains predictions made by humans on their own, and the accuracy of the humans using prediction sets is only estimated. How good are these estimates? We only get to see a comparison with the real value in Figure 3 for w=110, and it is clear that these estimates decisively overshoot.
3. The results show that the sets of lambdas do not contain all the harm-controlling values - as the authors remark - and especially seem to contain less of the ‘good’ values, meaning those that (at the cost of some counterfactual harm within the accepted bound) achieve higher accuracy. This seems to be a serious drawback.
4. Lambda is used in line 105 but only introduced in line 109.
5. Theorem 1 and 2 could use a bit more intuition and/or explanation.

Typos:
line 137: use -> used
questions: questions: Beside remarks on the weaknesses mentioned above, I would be interested in answers to the following questions:
1.   In Figure 1 you show the average accuracy of humans estimated by MNLs when w=110, but in the text you report that the second dataset contains the predictions of humans with prediction sets at all thresholds for that value of w, which is then shown in Figure 2. Why not start with the actual accuracy then, instead of the estimates? It seems to me that Figure 2 has the core findings, while Figure 1 is more of a sensitivity analysis on the fact that this phenomenon is consistent across model architectures (under the assumptions that the MNL estimates are qualitatively reliable).
2. Three of the models tested have worse average performance than the human experts. What is the point of testing decision support with such models? I suppose we would never implement a decision support that is worse than humans alone (unless perhaps we have strong evidence it is better than humans on a subgroup of data points, but this is not the case here), since it can stir the humans with poor predictions. I can see this could be interesting since you are studying counterfactual harm, but still it is an unrealistic scenario so I would not present results for three such models.
limitations: limitations: The paper seems to mention and discuss all the relevant assumptions and limitations.
paper_summary: paper_summary: This paper addresses the issue with prediction methods that generate a set of potential labels, from which a human makes the final decision. The authors identify a harmful scenario where the model's intervention leads to incorrect predictions that a human would have otherwise made correctly. To model this, they use the Structural Causal Model (SCM) framework.

The prediction set size is controlled by a parameter $\lambda$. For a given $\lambda$, they define counterfactual harm as the average number of instances where a human's unaided prediction would be correct, but becomes incorrect with the model's assistance. The main objective of the paper is to identify the range of $\lambda$ values that keep the counterfactual harm within a user-specified constant $\alpha$.

To achieve this, the authors use conformal prediction and propose two training algorithms under two conditions:

- Counterfactual monotonicity: At the instance level $(x, y, \hat{y})$, smaller and correct prediction sets lead to better human predictions.
- Interventional monotonicity: At the sub-population level characterized by $(x, y)$, smaller and correct prediction sets lead to better human predictions *on average*.

They conduct experiments on two image datasets where human predictions from prediction sets are available, demonstrating that their methods effectively control counterfactual harm.
strength_weakness: strength_weakness: Please refer the questions below.
questions: questions: **Equation 4:**

In the definition of CF harm, why is the true label $Y$ a function of the predicted label $\hat{Y}$? Could you please elaborate on the notation?

**Line 164:**

Should $v$ not be abducted from the joint distribution of $(X, Y, \hat{Y})$ rather than just $X$?

**Lines 127 to 130:**

I do not understand the text in these lines. Are you stating that one can always find some distribution over the exogenous variables that aligns with the observational data, or are you claiming that the original SCM is identifiable?

**Typo at Line 159:**

has --> had

**Proof for Proposition 1:**

In the proof for Proposition 1, where have you utilized the CF monotonicity condition? The proof seems to hold for any arbitrary SCM $\mathcal{M}$. Am I missing something here?


**On CF monotonicity:**

I believe that under no intervention, a human predicts the label with $C_\lambda(x) = \mathcal{Y}$, i.e., the entire label set. If CF monotonicity holds, why wouldn't any conformal prediction algorithm that guarantees $1-\alpha$ coverage satisfy the CF harm requirement?

**Examples for CF monotonicity violations**

Can you provide examples from your considered datasets where CF monotonicity is violated. In practice, why would such violations happen?

**Tradeoff between CF harm and conformal accuracy:**

I believe accounting for CF harm may unnecessarily increase the prediction set size, beyond what vanilla conformal prediction would offer, for the same amount of coverage guarantees provided by both methods. Can the authors comment on this, at least empirically?

If such a tradeoff exists, this approach would be interesting if one could find the subset of instances where harm occurs and explicitly find the prediction set for such instances. However, I am disappointed that the paper does not propose such approaches.

**Conformal baselines**

The authors should compare their algorithms with the state-of-the-art conformal prediction algorithms and evaluate how their methods perform in terms of (a) conformal accuracy, (b) prediction set size, and (c) CF harm. The current experimental section appears too weak.


**Conformalizing $\hat{Y}$:**

Why shouldn't this approach work? Suppose we apply conformal prediction to both the ground truth $Y$ and the expert prediction $\hat{Y}$, and then emit the prediction set as a union of the two prediction sets obtained from these conformalized models. How does your approach compare to such a proposal in terms of the accuracy $A(\lambda)$ and the prediction set size?
limitations: limitations: Yes, limitations are addressed.
paper_summary: paper_summary: The paper introduces a notion of counterfactual harm for a classification setting in which a human is choosing among a subset of possible labels determined by an AI system, i.e., the automated system is used to "narrow down" the set of most likely labels for a sample. Under assumptions of counterfactual and interventional monotonicity, which imply that , the notion of harm can be identified pointwise or up to an interval. A procedure for controlling harm is introduced, based on conformal risk control.
strength_weakness: strength_weakness: Major:

(W1) The causal inference notation is not up to the expected standard. In Definition 1, which is key to the paper, the notation does not work. The paper is trying to describe the notion of counterfactual harm, while using only the $do()$ operator and conditioning. The do() operator and conditioning can only give quantities in the first two layers of the causal hierarchy. Therefore, the current definition, as written, is incorrect. The appearance of $\hat Y$ in multiple places with different meanings is concerning. Instead of this, the paper probably wants to use the subscript notation $\hat Y _{\lambda}$. 

My understanding is that the paper is trying to say

$$ h_\lambda (x, y, \hat y) = E [ max (0, \mathbb{1} ( \hat Y = Y ) - \mathbb{1} (\hat Y _{\lambda} = Y) ) \mid X = x, \hat Y = \hat y, Y = y]. $$

(W2) The connection to Richens et. al [1] is mentioned in the introduction. It is clarified that the difference here is that "their definition requires one to deploy the system", while this is not the case for the definition proposed in this work. It is not clear from the text whether  the current definition does not require deployment because of the definition itself, or the monotonicity assumptions that are invoked.
The two definitions seem closely related, and given the comment in (W1), it would be very valuable to have these two definitions compared side-by-side, with a unified choice of corrected notation. Opening a small appendix for this would be very desirable.

---

Minor comments / corrections: 

(W3) Relating to point (W1): in Pearl's notation, causal mechanisms are usually denoted by the assignment operator $\gets$, i.e., $X \gets f_X (U_X)$. This is useful to indicate that the mechanisms are not "equations that can be manipulated".

(C1) Line 159: "if they has failed"
questions: questions: Please see (W1), (W2).

(Q1) Lines 147-148 claim that the $\Lambda (\alpha)$ set cannot be found because counterfactual harm is a counterfactual quantity. This statement seems imprecise. For instance, the effect of treatment on the treated (ETT), is a counterfactual quantity that can be estimated with lack-of-confounding assumptions. Furthermore, any counterfactual quantity could be, in principle, bounded without additional assumptions. 

It seems more likely that the joint distribution of $\hat Y, \hat Y _{\lambda}$ is required to evaluate the expression (and this seems to be a consequence of the max() operator within the expectation), which is known hard problem, often solved with monotonicity assumptions.

(Q2) When taking a step back, the question that comes to mind is: what is the end goal of the proposal? When minimizing counterfactual harm, it seems that "counterfactual benefit" is almost entirely ignored. Is this a reasonable notion to ignore? Furthermore, if the definition were to remove the max(), would this be equivalent to optimizing overall accuracy or something similar (note that removing the max solves the identifiability issue)? Finally, should there be an extension in which $\lambda = \lambda(x)$ is a function of $X = x$, so that harm is minimized in areas where it is largest?

(Q3) Line 142: "Otherwise, the expert's prediction could not have become worse". The claim of the paper is about decision-making support systems, whereas this comment seems quite closely focused on accuracy. For instance, in the very example given in Figure 1: if someone has a common cold, mistaking this for a flu seems rather different than mistaking it for lung cancer (the consequences of these two classifications may be rather different)? Please comment on this. 

(Q4) Lines 127-130 are saying that an SCM can match the observational distribution. Can you comment on the significance of this observtion? Is this remark made to claim that the usage of SCMs does not limit the generality of the approach, due to their expressiveness? If so, perhaps it is worth adding such a sentence.
limitations: limitations: Please check (W1), (W2), and (Q1), (Q3).

Comments: title: Re: Thanks for the explanation
Thanks for the explanation
Re: Follow up questions.
Follow up questions.
Re: Reaction to rebuttal
Reaction to rebuttal
comment: 2. Yes, in the definition both $v_1$ and $v_2$ are valid abductions. For each of the abductions, we need Eq. 6 to hold, once for $u, v_1, x$ and $Y = y_1 = f_y(v_1)$ and once for $u,v_2,x$ and $Y=y_2=f_y(v_2)$. In the definition, we fix X=x and we set the value of $Y$ using the mechanism $f_y(v)$ with $v \sim P(v | X=x)$.

3. We would like to once again emphasize that our methodology *does not* propose a way to construct prediction sets or, as the reviewer says, a conformal formula. *Given* a set of set-valued predictors (e.g. a type of conformal predictors), our methodology  can be used to identify which of these predictors cause counterfactual harm less frequently than a user-specified bound.  After applying our methodology, one can use the set-valued predictors off-the-shelf. In our previous reply, we committed to apply our methodology to APS and RAPS but not to compare against them since our methodology is incomparable. If the reviewer feels that we should apply to other specific types of conformal predictors, we would be happy to do so in the revised version of the paper. However, we would like to emphasize once more that such additional experiments will showcase our methodology with other conformal predictors, it *will not* compare our methodology against them.

4. Under the CF monotonicity assumption, a conformal predictor with coverage $1-\alpha$ will cause average counterfactual harm $\alpha$ **if and only if** the human expert succeeds in predicting the true label on their own for any of the $\alpha$ samples for which the true label is not in the prediction set. Note that such a condition *is not* CF monotonicity, it can be actually validated empirically, and it does not hold in practice for any of the classifiers and human predictions we have experimented with. If a human expert fails in predicting the true label on their own for any of the $\alpha$ samples for which the true label is not in the prediction set, then the average counterfactual harm of the conformal predictor will be less than $\alpha$.
2. But you said in your previous response that you abduct $v$ only depending on the $X$. Your definition in Eq. 6 says for any $v \sim P(V|X)$. In my example, both $v_1, v_2$ serve as valid abductions for the $x$. For the above explanation to hold you should have had the definition as for any $v \sim P(V|X, Y)$. But in one of my previous responses you had indicated that abducting with *just* the covariates is the correct thing to do. Your text in the paper and explanation provided above seems disconnected. 

3. I used APS and RAPS as examples, but I assume the authors are familiar with other state-of-the-art conformal predictors. To validate the merits of your conformal formula, it must be compared directly against these leading conformal predictors. I see comparing your approach against the state-of-the-art as the only way to establish its effectiveness.

4. The $\alpha$ coverage does ensure that the correct label is included in the prediction set at least $1 - \alpha$ of the time. In the absence of strong evidence, it is reasonable to conjecture that CF monotonicity holds. Therefore, all existing conformal predictors indirectly address CF harm and serve as benchmarks for your work.
We would like to thank the reviewer for their follow-up message.

2. We meant for any $v$ such as $X=x$. However, note that, in the inequality in Eq. 6, the random variable $Y$ is given $f_{Y}(v)$ and thus, in the example brought up by the reviewer, for $v_2$, one has $Y = y_2$. Then, for $u$ and $v_2$, the inequality in Eq. 6 is $\Bbb{1}\\{ f_{\hat{Y}}(u, v_2, C_\lambda(x)) = y_2 \\} \geq \Bbb{1}\\{ f_{\hat{Y}}(u, v_2, C_{\lambda'}(x)) = y_2\\}$ and, for $u$ and $v_1$, the inequality in Eq. 6 is $\Bbb{1}\\{ f_{\hat{Y}}(u, v_1, C_\lambda(x)) = y_1 \\} \geq \Bbb{1}\\{ f_{\hat{Y}}(u, v_1, C_{\lambda'}(x)) = y_1\\}$.

3. The conformal predictor used by Straitouri et al. is the only one that has been empirically validated using the same decision support paradigm we use in a large-scale human subject study. Nevertheless, following the reviewer's suggestion, we will apply our framework to APS and RAPS in an Appendix in the revised version of the paper. 

4. We would like to first clarify that the $\alpha$ coverage guarantee does not take into account whether the human expert successfully predicts the true label independently, and that is the reason why a conformal predictor with $\alpha$ coverage may cause CF harm $\alpha' < \alpha$. Further, in the example brought up with the reviewer, our framework can be used to verify that the RAPS conformal predictor causes CF harm smaller or equal than $\alpha'$. To this end, one needs to use the nonconformity score of RAPS to construct the prediction sets in Eq. 1 rather than the nonconformity score $1-m_{y}(x)$. In doing so, note that our framework does not change the way in which the RAPS conformal predictor constructs prediction sets and thus the prediction set size and accuracy do not change. As discussed under point 3, following the reviewer's suggestion, we will apply our framework to APS and RAPS in an Appendix in the revised version of the paper. 


Based on the above clarifications and promised changes, we hope you are willing to reconsider your score.
1. Understood, thank you. Apologies for misunderstanding the notation. No further questions.

2. I do not understand this point. You've defined both $X$ and $Y$ solely in terms of the exogenous noise $v$. Why would $X$ vary with $u$? What do you mean by the phrase: "for any $u, v$ such that $X=x$"?

I understand that you do not make assumptions about $f_X$ or $f_Y$. So, they can be man y-to-one in general. Let us consider a case where $f_X(v_1) = f_X(v_2) = x$ for $v_1 \ne v_2$, but $f_Y(v_1) = y_1 \ne f_Y(v_2) = y_2$. It seems your definition asserts the following condition:

$1\set{f_{\hat{Y}}(u, v_2, C_\lambda(x)) = y_1} \geq 1\set{f_{\hat{Y}}(u, v_2, C_{\lambda'}(x)) = y_1}$

Is my interpretation correct? How do you explain the discrepancy between $v_2$ and $y_1$?

3. I disagree with the authors that only the method by Straitouri et al. represents the current state-of-the-art baselines. They should compare their approach with other well-established, such as APS, RAPS, and other advanced conformal predictors. Figure 2 in the provided PDF does not show comparisons with existing leading conformal methods. Such comparisons are crucial to establish the merits of your approach in context of the existing literature.

4. I agree with the authors that CF harm may occur in cases when "a human expert successfully predicts the true label independently, but the corresponding prediction sets do not include the true label." However, many conformal predictors precisely aim to address this issue by ensuring an $\alpha$ coverage guarantee. Such harm is already addressed in my opinion. 
My main concern has been the following: Suppose we consider the RAPS conformal predictor (for example), which provides an $\alpha$ coverage with CF harm $\alpha' < \alpha$ and has an average prediction set size of $\delta$. If I were to use your Conformal algorithm instead, aiming for the same CF harm $\alpha'$, what would the corresponding prediction set size, and the accuracy be?

I am hesitant to recommend an accept, and I retain my score.
We would like to thank the reviewer for their follow-up message.

**[(W1)]** Following the reviewer's advice and to avoid misunderstandings, we will explicitly point out that we follow Peters et al.'s notation and will also discuss the differences between Peters et al.'s notation and Pearl's notation in the revised version of the paper.

**[(Q2)]** After re-reading the original concern raised by the reviewer under Q2, our rebuttal, and the reviewer's follow-up message, we are unsure we understand how to answer to "I would expect some simple example/illustration to demonstrate how harm-controlling sets you discuss help solve a problem that is not handled by other existing methods". Nevertheless, in what follows, we try our best and, if we miss the reviewer's point, please, follow-up further.

We view harm control as a prerequisite, not an end goal, for any decision support system in high-stakes domains. In this context, we get inspiration from the medical domain where, before measuring the effectiveness of a new treatment, one measures the harm caused by the treatment. In this context, it is also worth pointing out that, in contrast to our work, existing methods do not leverage the specific monotonicity assumptions of decision support systems based on prediction sets to (partially) identify counterfactual harm directly from observational data. 

Further, in the case of decision support systems based on prediction sets, we would also like to clarify that the counterfactual and the interventional monotonicity assumptions do not allow us to identify and partially identify counterfactual benefit. In this context, it would be interesting to identify properties under which counterfactual benefit is (partially) identifiable.
Dear authors,

Thanks for your detailed response. You have addressed some of my concerns, and I will just share some follow-ups:

(W1) Although I am familiar with the textbook you mention, I was not aware of the do() notation being used for counterfactuals. It does seem that your definition is then correct, I am a happy for you to keep the notation you prefer (I thought there was a genuine mistake). However, two things to keep in mind: (1) I am not sure if there is any place where you mention where your notation is coming from, which would be a very good thing to add. (2) My comment about the notation is still pertinent, in the sense that readers familiar with the framework of the Causality book (Pearl, 2000) will be confused by the notation (this book has 27,000 citations on Google Scholar, if you consider that a relevant metric). For this reason, it may be helpful to have a sentence in the preliminaries that mentions the difference in the notation.

(Q2) 

> We would like to clarify that our goal is not to minimize counterfactual harm but rather to find the largest harm-controlling set ...

I am aware of the scope of the paper. However, given that you are spending significant energy on studying this topic, and are cleary doing some good work, I would expect a better justification for how this effort fits into the bigger picture. In other words, I would expect some simple example/illustration to demonstrate how harm-controlling sets you discuss help solve a problem that is not handled by other existing methods. I hope you can share some thoughts, since I would be quite interested to learn about this.
We would like to thank the reviewer for their follow-up message. Below, we provide a point-by-point reply. Please, do not hesitate to follow up further if something is unclear.

1. In Eq. 4, the true label is not a function of the predicted label. Here, note that $(\hat{Y})$ is not part of the superscript. The expression refers to the probability of the random variable $\hat{Y}$ under the distribution $P^{\mathcal{M}; do(\Lambda=\lambda) | \hat{Y} = \hat{y}, X = x, Y=y}$.

2. In the definition of CF monotonicity we follow from Straitouri et al. [18], for a fixed $X=x$, the inequality in Eq. 6 needs to hold for any $u$ and $v$ such that $X=x$. However, for different $v$’s such that $X=x$, one may have different $Y$ values since, in the SCM in Eq. 3, we do not make any assumption on $f_X$ and $f_Y$.

3. and 5. Given a conformal predictor with coverage $1-\alpha$, our framework can be used to predict whether the conformal predictor causes average counterfactual harm smaller than a user-specified bound $\alpha' \in [0, 1]$, with $\alpha' \leq \alpha$. In our experiments, we applied our framework to the SOTA conformal predictors by Straitouri et al. [17,18]. Therefore, in Figure 2 in the one page pdf attached with the rebuttal, we show the (a) average set size and (b) coverage vs the average counterfactual harm achieved by the SOTA conformal predictors by Straitouri et al. [17, 18]. Given this clarification, we are not sure what it would mean to compare these SOTA conformal predictors with our method in terms of prediction set size.

4. In the benchmark datasets we used in our experiments, we cannot identify counterfactual monotonicity violations using the mixture of MNLs because the mixture of MNL is not a causal model—it is just a prediction model used to estimate the accuracy of the predictions made by humans using prediction sets. However, we would further like to clarify that counterfactual harm may occur even if there are no violations of counterfactual monotonicity. If there are no violations of counterfactual monotonicity, the average counterfactual harm is given by Eq. 8. Therein, it is apparent that, as long as there are samples for which a human expert successfully predicts the true label on their own but the corresponding prediction sets do not include the true label, counterfactual harm will occur. Further, if there are violations of counterfactual monotonicity, the right-hand side of Eq. 8 is a lower bound on the counterfactual harm. Then, based on the results shown in Figures 2 and 3, we can conclude that there exists counterfactual harm irrespective of whether counterfactual monotonicity holds.
Thanks for the rebuttal. I have carefully read your responses. Please see below for the follow up questions.

1. I am particularly interested in understanding the notation $Y=y(\hat{Y})$. Why is the true label expressed as a function of the predicted label?

2. While I understand that $\hat{Y}$ is not fixed. but why is $Y$ not fixed?

3. If conformal predictors with $1-\alpha$ coverage guarantee gives us an $\alpha$ guarantee on CF harm already, wouldn't conformal predictors directly serve as baselines for your approach? In that case, comparing existing SOTA conformal predictors with your method in terms of prediction set size becomes crucial. I’m disappointed that such comparisons were not included in the rebuttal, despite being explicitly asked for.

3. I understand that in observational datasets, it’s not possible to detect CF monotonicity violations. However, in your benchmark datasets, where predictions were modeled based on prediction sets using MNL, did you encounter instances of such violations? Identifying such examples would strongly support the practical motivation for your problem setting. Currently, I am struggling to convince myself that CF harm occurs in practice.

4. I am unclear about the results provided in the PDF. The experiment I requested was as follows: Consider the SOTA CP algorithms in the literature and evaluate the following metrics—CF harm, conformal accuracy, and prediction set size. How does your conformal approach, which explicitly addresses CF harm through the formula $1 - m_y (x)$, compare to other SOTA methods that do not? I’m particularly interested in understanding the gap your approach fills empirically in the existing SOTA methods.
We would like to thank the reviewer for their follow-up message. In the revised version of the paper, we will include our responses to points 3 and 7 into the main text and we will discuss the related work suggested by the reviewer alongside Wang et al. (2024) and Liu et al. (2024).
Thanks to the authors for their rebuttal. I believe my points have been addressed sufficiently, and I appreciate the additional work in showing the role of the validation set. I suggest to work their responses of their points 3 and 7 into the main text.

Concerning the point of another reviewer on the discussion of the relationship between prediction performance and harm in decision support systems, alongside Wang 2024 and Liu 2024 it would be appropriate to discuss van Amsterdam 2023 at al "When accurate prediction models yield harmful self-fulfilling prophecies", which contains formal results on the topic.