Utility-Directed Conformal Prediction: A Decision-Aware Framework for Actionable Uncertainty Quantification
Authors: Santiago Cortes-Gomez
Carlos Miguel Patiño
Yewon Byun
Steven Wu
Eric Horvitz
Bryan Wilder
Venue: ICLR 2025 Poster (2025)
Keywords: Decision-focused learning
decision making
uncertainty quantification
healthcare
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: There is increasing interest in ``decision-focused" machine learning methods which train models to account for how their predictions are used in downstream optimization problems. Doing so can often improve performance on subsequent decision problems. However, current methods for uncertainty quantification do not incorporate any information at all about downstream decisions. We develop a framework based on conformal prediction to produce prediction sets that account for a downstream decision loss function, making them more appropriate to inform high-stakes decision-making. Our approach harnesses the strengths of conformal methods—modularity, model-agnosticism, and statistical coverage guarantees—while incorporating downstream decisions and user-specified utility functions. We prove that our methods retain standard coverage guarantees.  Empirical evaluation across a range of datasets and utility metrics demonstrates that our methods achieve significantly lower decision loss compared to standard conformal methods. Additionally, we present a real-world use case in healthcare diagnosis, where our method effectively incorporates the hierarchical structure of dermatological diseases. It successfully generates sets with coherent diagnostic meaning, aiding the triage process during dermatology diagnosis and illustrating how our method can ground high-stakes decision-making on external domain knowledge.

Published conference paper ICLR 2025
UTILITY-DIRECTDD CONFORMAL PREDICTION: A DECISION- AWARE FRAMEWORK FOR ACTIONABLE UNCERTAINTY QUANTIFICATION
Santiago Cortes-Gomez, Carlos Patiño', Yewon Byun, Zhiwei Steven Wu, Eric Horvitz & Bryan Wilder Machine Learning Department, Carnegie Mellon University
"University of Amsterdam
"iirosooft
(scortesg, yewonb, zstevenwu, bwilder]ecs. cmu.
carlos. patino. paz@student. ul horvitzém crosoft. com
ABSTRACT
Interest has been growing in decision-focused machine learning methods which train models account for how their predictions are used in downstream optimization problems. Do- ing so can often improve performance on subsequent decision problems. However, current methods for uncertainty quantification do not incorporate any information about downstream decisions. We develop a methodology based on conformal prediction to identify predic- tion sets that account for downstream cost function, making them more appropriate inform high-stakes decision-making Our approach harnesses the strengths of conformal methods cecthodd--oduuariiyy porating downstream decisions and user-specified utility functions. We prove that our meth- ods retain standard coverage guarantees. Empirical evaluation across a range of datasets and utility metries demonstrates that our methods achieve significantly lower costs than stan- dard conformal methods. We present a real-world use case in healthcare diagnosis, where method effectively incorporates the hierarchical structure of dermatological diseases. The method successfully generates sets with coherent diagnostic meaning, potentially aid- ing triage for dermatology diagnosis and illustrating how our method can ground high-stakes decision-man makinn employing domain knowledge.
model-agnosticism, and statistical coverage guarantees-whil incor-
INTRODUCTION
Uncertainty quantification in classification problems is increasingly addressed through a model-agnostic tech- nique known conformal prediction Vovk et al. (2005); (Aggllopoulos & Bates (2021). Instead of outputting a single label and confidence, conformal prediction outputs a set outcomes with guaranteed statistical cov- erage, ensuring that the true outcome is included in the set with pre-specified, target confidence level. This procedure for quantifying uncertainty through inference about sets of outcomes, has been leveraged to enable safer decision-making based on machine learning model outputs. As an example, in dermatology diagnosis, conformal analysis might predict that the patient's 's illness is member of a set of possible mutually exclusive classes, such as [eczema or psoriasis]. That is, a guarantee is provided that the true condition is highly likely to be included in the predicted set, making inference about the set a more robust estimate than raw model outputs about singular outcomes. Recent efforts have pursued uses of conformal prediction develop Al systems that
can support decisions in high-stakes situations (Vemula et al.) 2018 Dvijotham et al. 2023 Yin et al. 2022 where the decisions restricted to actions within the prediction set are deemed as safe.
We note that statistical coverage of the true label is not always a strong enough desiderata to guarantee in- formative sets for decision making. In many applications, prediction sets should ideally be aligned with utility function that captures preferences about downstream decisions and outcomes. For instance, in medical diagnosis, predicted set of discases may be more actionable if the different suggested diagnoses share the same treatment, costs and benefits of treatments, or if it is straightforward to conduct additional tests for each diagnosis the set to discriminate among them.

---

Published as a conference paper at ICLR 2025
Indeed, there has been a growing body of work that bridges machine learning predictions and decision prob- lems modeled by optimization programs, Among this literature there is growing interest for the so called decision-focused machine learning (Wilder et al. 2019 Mandi et al., 2020: Wang et al. 2021] Shah et al.
2022 Elmachtoub & Grigas, 2022 Ren et al. (024). where models are trained by maximizing a downstream optimization problem in the core loop of machine learning. In contrast, conformal methods are designed to be agnostic to the model, loss function, and downstream decision task; their generality is key feature that allows this method to be used flexibly as a post-processing step that is completely decoupled from the model.
Inspired by the spirit of approaches with the goal of incorporating information about downstream decision problems, retaining the modularity and ease of use of conformal prediction, while enabling users to align prediction sets with task- specific utility functions. We are not proposing an end-to-end approach to train a model to output predictions that are going to yield higher utility on a downstream application. Rather we show how to generate prediction sets that minimize a user-specified cost function while maintaining a high-probability coverage guarantee. To achieve this, we propose two families of algorithms. The first family, extending existing methods for conformal prediction, incorporates a penalty based on the cost function of a prediction set. This creates a collection of conformal predictors indexed by hyperparameter which weights cost function (the penalty) against a standard non-conformity score. The hyperparameter is then optimized on a validation set to find the value with minimum cost function. While this method often performs well, can struggle adapt to the combinatorial structure of downstream cost function (e.g., non-linear interactions between elements of the prediction set) and adds complexity in the form of hyperparameter tune. Therefore, address these limitations, we introduce a second family of algorithms which instead solve an optimization formulation whose solution is the set with the minimum loss subject to statistical coverage constraints, i.e, the sought after predicted set. Algorithmically. this is instantiated by solving lightweight optimization problem for each test instance, followed by post- processing a function of the base classifier's predictions to enforce the desired statistical coverage. We prove that both approaches provide standard coverage guarantees and find empirically that they are often able to find sets with significantly smaller costs than standard conformal methods.
learning. we introduce methods that bridge the strengths of conformal
Incorporating the cost function into uncertainty quantification allows us to address the goal of producing more informative prediction sets. For instance, in the aforementioned case study on dermatology diagnosis, we show how specifying a natural cost function (a coverage function) enables us to generate prediction sets with more clinically coherent interpretation. Figure provides an illustrative example of a prediction set output by a standard conformal approach compared to that of our method. We observe that the different labels within the standard conformal prediction set would imply completely different downstream actions for diagnosis and treatment, as benign vs. malignant diseases need to be dealt with differently. Even though the set contains the true label, it is not informative for decision making. By contrast, the prediction set output by our method con- tains diagnoses within a common family of malignant epidermal diseases, while satisfying the same coverage guarantees. This hierarchical homogeneity of the prediction set aids clinicians in making follow-up decisions (c.g., reflecting about the discase process category at hand, prioritizing follow-up tests, prescribing treatments, etc.). We accomplish this by specifying a cost function (detailed in Section 3.2) based on coherence of the prediction set with respect to expert-defined hierarchy for dermatological diagnosis. Such hierarchical ab- stractions are a common feature of medical diagnosis and many other real-world classification tasks, and expert decision making more broadly, and a valuable characteristic of our method is that it can be employed to align prediction sets with the abstractions that human experts use to make decisions based on predictions.
To summarize, our contributions are as follows:
We develop three methods for constructing conformal predictors which generate sets with low average cost function based on a pre-defined utility function, thereby producing more desirable sets from the perspective of downstream use. We start by considering cost functions which are separable, meaning that they can be decomposed into the sum of a penalty for each label included in the prediction set. We then generalize these ideas to the non-separable case, where the cost function may contain nonlinear interactions between elements of the set. In each case, we prove that our algorithms produce sets with the desired level of statistical coverage.
We conduct experimental analyses across several datasets, including both standard image classifica- tion datasets and a real-world case study in dermatological medical diagnosis. Our proposed algo- rithms result in prediction sets with improved costs across all settings, often by a substantial amount,
2

---

Published conference paper ICLR 2025
e.g., reductions of 60-75% loss. Our method successfully captures patterns homogeneity in- formed by external domain knowledge. In particular, for the case study, methods produce predic- tion sets that have diagnostic meaning, demonstrating how our method leverages uncertainty quantifi- cation to ground high-stakes decision-makes maaoon-maiin----- in this case in the healthcare domain. Our implemen- tation is publicly available at link
Base Conformal Prediction
Greedy Optimizer on Coverage (Ours)
Predicted Set
Sebortheic Keratosis
Predicted Set
Benign
Epidermal
Actinic Keratosis
Jadassohn
Basal Cell Carcinoma
Malignant
Epidermal
Squamous Call Carcinoma
Malignant
Epidermal
Besel Call Carsnome Marphallom
Solid Cycto Basal Cell Cardioma
Lupus Erythematosus
Squamous Cell Carcinoma
Neutrophilic Dermaloses
Psoriasis
Non-Neoplastic
Intaammatory
Scables
Seborheic Dermatiis
Stasis Edema
Figure 1: Illustration of how proposed methods compare to the standard formulation of conformal prediction. The real-world example is taken from the Fitzpatrick dataset where the true disease is squamous cell carcinoma. The standard conformal prediction method results in of nine elements that span benign, malignant, and non-ncoplastic diagnoses. Our method provides formulation with set of five elements within the malignant melanoma category.
1.1 RELATED WORK
Decision-Focused Learning A growing body of literature focuses on aligning machine learning model pre- dictions with downstream optimization objectives, either under the predict-then teeeittthen-optimize framework (Mandi ct al. 2020; Elmachtoub & Grigas| 2022. Wang et al. 2021). or using end-to-end approaches like decision- focused learning (Wilder al. 2019 Shah et al. 2022 Ren al., 2024). Of particular relevance our work is Horvitz & Klein (193)), which introduces utility- theoretic formulation of inference and decision making at progressively higher-levels of abstraction about outcomes (i.e., abstraction defined in terms of moving from single hypotheses to sets of mutually exclusive outcomes). This work is particularly interesting and motivating to us, as is proposed as solution to similar real-world case study as ours but using ideas related to Bayesian inference framed in the context of utility- -hhoortttc bounding formulation.
Uncertainty Quantification and Conformal Prediction The literature on uncertainty quantification is vast, from Bayesian methods, such Bayesian neural networks (MacKay) 1992.
1995 Neal 1996. Wilson Iz-
mailov 2020) and ensemble methods (Wasserman) 2000 Lakshminarayanal et al. 2017). 7), to frequentist ap- proaches, such as quantile regression (Koenker& & Bassett Jr 1978] Koenker [20l Yu et al. 2003 Tagasovska & Lopez- Paz, 2019 Pouplin et al. 2024) and conformal prediction (Vovk et al. 2005 Romano al. 2020b). Quantile regression particularly noteworthy, as can be adapted to produce conformal predictors for continuous variables. However, the study of the continuous case lies outside the scope of this work. Of utmost relevance to our work the literature on conformal prediction.
al. 2020 Angeloppolos
Conformal prediction has gained popularity in recent years model-agnostic, computationally low-cost procesroc method for principled uncertainty quantification Vovk et al.] (2005). In particular, [Romano al. 22200) and (Angelopoulos
al. (2020b) explore how create prediction sets whose size adapts to the confidence levels output by the model. When the model is more certain about a given prediction, the predic- tion set becomes smaller. Specifically, Angeloppulos et al. (2020b) generates adaptive sets that are smaller on average, a task that represents particular case of the type of problems we aim to solve, Conformal prediction has been used in the context of decision making the medical setting, for example Lu et al. 2022) propose
3

---

Published conference paper ICLR 2025
conformal predictor that provides group conditional guarantees. They are interested in provide prediction sets that are usable across groups in order to ensure fair decisions, thus increasing clinical usability and trustworthi- ness in medical AL. In other words, this work is focused on guarantee faimess rather than general formulation of fallity-fousset prediction sets.
At the intersection of machine learning for decision-making and conformal prediction, and thus most closely related our work, is conformal risk control Angelopoulos al. (2022). The approach is generalization of conformal prediction that allows for controlling the expected value of desired loss at preset level. The method can achieve true statistical coverage in multi-label classification or, more generally, for any loss function that decreases as the size of the prediction set increases. Although this method offers flexibility, it does not directly optimize the expected value of the chosen function, and depending on such function, controlling for the expected value of the chosen loss can be at expense of statistical coverage. Notably, our method imposes no restrictions on the type of losses that can chosen, allowing it to handle cost functions that do not decrease monotonically with respect to set size and for which it is impossible to guarantee absolute upper bound. We also note there is wealth of literature that studies uncertainty quantification in similar settings (Qi et al. 2021
Chenreddy et al. 2022; Kallus & Mao 2023: Sadana al., 2024 Chenreddy & Delage 2024 Patel et al. 2024 Yeh et al. 2024).
2 PROBLEM FORMULATION
Let Y' a finite set of labels. Define the cost function of a prediction set as a function L : 2 + R. In our framework, L evaluates the utility of a given prediction set, where a suitable prediction set for decision-making is one that has low value of C. Let P be joint distribution over X' x y. Our methods will act as post- processing steps to pre-specified classification model f that estimates p(y]az). Throughout this paper, we will use p(y(æ) (z) and (z) interchangeably. We do not assume anything about how the model f is trained. In particular, it need not be aware of the cost function L.
The ultimate goal is to leverage to inform decisions via set S of suggested labels, accounting for both utility and uncertainty. Thus, for every x, we would like to produce a set of labels Sy(=) that fulfills the following two properties:
1. For a random sample (z, y) drawn from P. it must be true that E St(r) with probability at least a This condition must be fulfilled without any assumptions on the correctness of f or the distribution
2. ((r,v)- ((((s)() is minimized. This quantifies that, on average, the set Sy(z) is optimal from an utility- seityy-theoreicall perspective.
Standard conformal methods guarantee the first property and our goal in this paper is to target the second (cost function minimization) as well while still maintaining such coverage guarantees.
Empirical Coverage
Statistical Coverage for iNaturalist Dataset
0.99
Legend
Greedy Optimizer Max
0.95
Greedy Optimizer on Coverage
Separable Penalized Ratio
Standard Deviation
0.90
3-
4-
Reference Lines
0.9
0.95 Desired Coverage (1 at
0.99
Figure 2: Empirical statistical coverage on the iNaturalist dataset across various loss functions. The observed coverage is close the expected preset value of a for all 10 trials per loss function. We similar behavior for all the other datasets.
"This not conditional statement, instead is a high probability event over the joint distribution of tuples (Y, X)
4

---

Published a conference paper at ICLR 2025
METHODS
We start by considering simplified setting in which the cost function has particular separable structure, which we can leverage to precisely characterize to our problem. We then consider nonseparable loss functions and extend such ideas the general setting.
3.1 SEPARABLE LOSS
As a motivating example, consider cost function in the medical domain where healthcare worker must diagnose a condition based on a set of tests. Each test has an associated penalty related to costs, patient discomfort, and procedure complexity. Notably, the penalty of each test does not depend on the presence of other tests in the set. Formally, we say that the cost function L. is separable if it can be decomposed as ((S,) - =yees {vy, where f(y) is a cost function defined on individual labels rather than on the set of labels as whole.
One starting point for algorithm development in this setting, is the standard conformal prediction toolbox, in particular, the split conformal algorithm. Split conformal uses a held-out calibration set to provide model- agnostic pro-pesoess step which generates prediction sets with statistical coverage at a predefined level. While this method naturally addresses the first (I) of the two conditions we seek from our prediction sets, raises an important question: can we leverage split conformal to also ensure low loss L?
At the core of the split conformal algorithm the definition of a non-conformity score s(x. y). Larger scores encode worse agreement between x and y (Angelopoulos & Bates] (02T). Naturally, the selection of a non- conformity score is key because the prediction sets will be a function of it. There has been recent work (Angelopoulos & Bates) 2 exploring the choice of particular score as engineering question; in gen- cral, the non-conformity scores are a function of the estimates p(y(x) thus leveraging the notion of similar- ity between the pair (I, y) captured by the model. The split conformal algorithm works by outputting sets S. {y ((r,v) < h- where fi- is the empirical (m-1)(1-a) quantile of independent draw of size n of nonconformity scores, such draw is known in the literature as the calibration set.
Returning to our goal of producing prediction sets with low cost function L. one option is to incorporate the loss of the prediction set (())()) into the score s(x, y). This can be done explicitly if the loss C is separable as (()) can be decomposed into a penalty E(y) for each individual y. Formally, let x xxX)
(o) ..., [x)) the permutation that orders the estimates ((B)(1)) (a)) from greatest to lowest, define p(z, y) Ei=
Ei P(Vrr (ax) (r)) where (x)(xx) y and define L(y) as Σx= (a)) Particularly, is the nonconformity score used in the popular "adaptive prediction sets" algorithm Romano et al. (2020); as labels are added from most to least likely until the true label y is reached, P computes running total of the amount of esstmaated- probability mass contained the set. Notably, L keeps running total of the cost function incurred when adding elements to the prediction set in that same order. Our proposed nonconformity score is simply sx(z, y) := p(x, y) + ALy),, where À >>00 0 is a hyperparameter. After applying the split conformal algorithm on a held-out calibration set, the resulting prediction sets for given À are given by S(() = {v Sx y) < f/aa) } where
+/ is the empirical (m++)(1-a)) quantile of an exchangeable draw of size n the scores 8x(2, y) Note how a higher value of skews the distribution of the scores; for any pair (I, y) the relevance of the similarity captured by the estimate p(y|æ) is going to have less weight than the total loss of the proxy prediction set (M), (x,,) captured by L(y).
The question of how to select the appropriate À remains. One possible approach is to perform a grid search for the optimal value using a separate validation/test split distinct from the calibration dataset. Since maximizing utility is the primary objective, this hyperparameter tuning strategy indeed will converge to the true sought parameter as summarized by the following proposition.
Proposition 1. Let X E H where H is a finite set, assume as well that for every X and every x. L(S) (x)) &
B. Let S be the X that minimizes 1/ Er= ((S)(x)). for an iid draw of size from P. Let X be the X that minimizes the population level quantity E(C(S)) (x))) Fix any 8 E (0,1) then with probability at least I 5
B(C(S) - B[[(S)(x))]] < &2B log
2/H( 5
1
2n
5

---

Published a conference paper at ICLR 2025
A consequence of the previous preposition is that X can be learnt via empirical risk minimization. That being said, doing so makes the estimated Ả dependent on the data. Thereby breaking the exchangeability assumption between a test point (X, Y) and the original sample. This assumption is crucial for applying the split conformal algorithm to ensure valid statistical coverage. Consequently, after learning Å, the quantile fi must be recalculated using an independent sample.
Taken all together, we then proposed the following algorithm. Firstly, split the data in three folds: a validation set, test set, and a calibration set. For each value of À in H estimate the quantile f on the val set. Then, we estimate the cost function on the test set and then select the À with the best test loss (*). Finally, re-estimate the quantile A on the fresh calibration set to guarantee valid statistical coverage.
Although our proposed sample splitting impacts the learning rate stated in Proposition the final rate of convergence remains Of relative to the total sample size.
3.1.1 HYPERPARAMETER-
SOLUTION
The prior solution depends on a hyperparameter À that must be tuned. We also have no formal guarantee that the optimal prediction sets for our problem can be expressed via the penalized family of nonconformity scores. However, it possible to develop score derived from more principled approach that does not require the selection of any dyper-parameters As starting point to derive this score, suppose that there is oracle access to the true estimand ply z). Then, conditions and 2 can be explicitly stated as the following program:
min Ex([(Sx)] s.t P(Y E Sx) 11-a S=Xxy
Where Sy are the fibers of S over x, formally, S₂ = {y: (z, v) E S}. This program, under the assumption of separable loss, has closed-fone solution via the Neyman Peaiaanne lemma. Define H = {(z, v): p(ylæ)
[(y) > ta
where to is the a quantile for p(uا= That is, (x) is the prediction set given by all labels where the ratio of probability loss exceeds a threshold chosen ensure coverage.
Proposition 2. The set H minimizes E,, [[(x)) while fulfillin P(Y E H(X)) 21-a
We note that Sadinle et al. (2019) characterized solutions for the special case where the loss function is the size of the set (i.c., all labels have equal cost): here we provide general solution for arbitrary separable losses.
Since we do not have access the ground truth p(ylæ), we instead use f as a plug-in approximation for p(y|z) in the set H defined above. Since f may not be correct, we may no longer have the correct coverage level, which we correct with the following conformalization step. Define s(x, y) p(vl=)
fly) let fa be the empirical
a quantile of an observed iid draw of scores s(z, y). Define Sf(=) := {y p(v=) frperparameter--ree method as the Separable Penalized Ratio throughout this work. We can guarantee using standard arguments that the separable penalized ratio algorithm satisfies statistical coverage, as demonstrated in Figure and formalized in the following proposition.
fly) > fa). We refer this
Proposition 3. Let s(z, y) and Stz²) be defined as above. Then
1
1-a P
E S <1 a n + 1
3.2 NON-SEPARABLE LOSSES
Unfortunately, not all relevant set losses L are separable. In real-world scenarios, we often want to define losses that depend interactions among the elements in the set. For example, consider running example of medical diagnosis, where we may prefer prediction sets in which the labels come from a small number of categories within a domain-specifie hierarchy, representing more clinically similar diagnoses. This can be formalized as (())())) = Ecec T(S)())) n C)) where C is a set of (possibly overlapping) categories. In this context, we prefer prediction sets which intersect with as few categories as possible. This loss is not separable because there are diminishing penalties to including a new label in the set once a given category is already covered (as discussed formally in the appendix).
6

---

Published a conference paper at ICLR 2025
While loss separability was instrumental to the family of non-conformity scores proposed as a solution for the separable case, we now present a conformal- approach for the non-separable case. Drawing inspiration from the separable scenario, we introduce a non-conformity score that generalizes the one outlined in Proposi- tion [ The key idea is to linearize the non-separable cost function by summing the marginal increase in loss when adding each additional element to the prediction set.
Let x (x) = (o) o((x)) be the permutation that orders ((ya(ar from greatest to smallest. Define p(z, y) = Ei= (( (x), where (x)
(this coincides with the homonymous quantity defined in the previous section). Now, let S: = (Vo,(x) : E [i]} be the set containing the first i labels in the permutation Ei= (C(S.) 1() as the sum of the marginal gains obtained from adding elements using the order given by the permutation a. The proposed non-conformity score is then given by (Iz 2) = p(a, y) + (L))..
and define L(y) (())
As in the separable case, this heuristic generates sets that ensure coverage while maintaining low loss. The motivation for selecting À remains consistent with that of the separable case. We can also employ grid search hyperparameter tuning on a validation dataset, as the approach outlined in Proposition does not rely on the separability of the loss.
The definition of the previously presented score, circumvents the non-separabiliity of the loss by fixing the order in which to add labels to the prediction set (based the confidence score outputted by the model f), after which the loss can be decomposed into a series of marginal increments.
3.2.1 HYPPERARRAEEERR-RRE SOLUTION
Just in the separable loss case, we would prefer approach that does not rely hyperparameter tuning and which can benefit from structural information about the optimization problem at hand. The main draw back from the penalized conformal predictor proposed for the non-separable case is that the fixed order (which does not depend on the loss) upon of which it is based may be suboptimal if it does not adapt to the combinatorial structure of the cost function. To address this issue, let us revisit the program in Equation [ and suppose that we have access to the true estimate ply z). As in the separable case, it would be ideal to solve this program. coosed-forn solution exists.
However, since the separability assumption no longer holds, it unclear whether Suppose instead that we have access to an algorithm which can optimize over the cost function L. Then, we could solve the following optimization problem separately for each value of z to produce the prediction set Sg:
min ((S2) s.t. 5.Cy
p(yl)) 1- WES.
The prediction set Sy defined by solving this problem for each would correspond to feasible solution to the problem in Equation Moreoever, it would minimize the cost function on each instance. However, actually attempts offer stronger coverage guarantee: conditional coverage guarantee where Sz contains the true label for every æ, instead of the marginal guarantee in condition 2] Unfortunately, guaranteeing conditional coverage is in general impossible without access the ground truth p(y]æ=) (Foygel Barber et al. 2021 Vovk 2012), so we will be unable to implement this solution exactly.
Our proposal is to use the optimization oracle to solve the above optimization problem at each test instance with the plugin estimates p(y| and then conformalize the resulting sets to provide at least marginal coverage guarantees. While there are many ways to implement this meta-algorithm (depending on the optimization approach which is appropriate any given cost function L), we focus here on a particular implementation via an efficient greedy algorithm which we find works well for many losses in practice.
Suppose that L(S) & M. Consider the order in which labels are chosen by the following greedy algorithm for [eekkvee et al. 2007) solving the plug-in optimization problem:
M u
5i+1 f())
arg max DEV/St -a ((.))
1 - p(ylz))
7

---

Published as a conference paper at ICLR 2025
where p(S:) == Eyes, Pu) We then define a non-conformity score as s(x, y) = -E: Hyo(() z), where is the permutation that orders the labels according to the greedy construction of the set S (x)) outlined in equation 2 where k is such that y Vo(k)- Our final algorithm will simply rum split conformal on this noncon- formity score. The crucial difference from the previous approach is the order in which elements are added to the prediction set, which is calculated by simulating the above greedy algorithm each instance (instead of simply sorting the predicted probabilities). This procedure has standard marginal coverage guarantees:
Proposition 4.
Let Ta,n be the empirical
[n+1)(1-o) quantile from an iid draw of size n of the scores s(x, y).
Define S(e)) = {y= : s(x,)) v) & T. Then for any (x.y) E P.
1 1-a a < P(y Stz)) & - a+ 1
n n
4 EXPERIMENTS
4.1 SETUP
We conduct series of experiments to evaluate our proposed methods for both separable and non-separable loss cases, with the following setup.
Datasets and models. We evaluate our algorithms for both separable and non-separable settings using the following four datasets: CIFAR-100 (Krizhevsky & Hinton| 2009), iNaturalist (Van Horn et al.] 2018). Ima- geNet (Deng et al. 2009). and the Fitzpatrick dataset (Groh et al. 202IJ. The iNaturalist dataset consists of a large collection of images of living organisms, with labels organized according to subtree of the natural taxonomic hierarchy. Our classifier is trained to predict the lowest level of abstraction in this hierarchy, which is the class of the organism (here, "class" refers to the taxonomic rank). These classes are further grouped into phyla, which in turn belong to broader taxonomic categories known as kingdoms. The Fitzpatrick dataset con- tains 16,577 clinical images, annotated with both skin condition and skin type labels based on the Fitzpatrick scoring system, along with two additional aggregated levels of skin condition classification derived from the skin lesion taxonomy developed by |Esteva al. (2017). Training a classifier on the Fitzpatrick dataset is par- ticularly challenging (Groh et al., 2021). with good models having top-1 accuracy hovering around 30%. In order to study how the performance of our methods varies with the quality of the underlying classifier, we also include a cleaned version of the dataset where we excluded classes with a validation accuracy below 0.30, retaining 64% of the dataset (reducing the number of classes from 114 to 49) and increasing model accuracy to 0.5. In all experiments, we use EfficientNet-BOO TTan 2019), with a fixed at 0.1, as our underlying classifier to estimate the conditional probability p(y|z)). We report the median of the means for the loss function of interest across different validation and calibration data samples for each dataset. Additionally, we present the statistical coverage conditioned on set size Anzgelloooos et al. (2020a), in order to investigate that the marginal coverage guarantee does not mask unwanted behavior, such as providing perfect coverage for a specific subgroup of the population at the expense of others. Finally, we report the average value of p(ylıc) for the true label y as a function of the set size S(r)). which is proxy for adaptability of our methods (Angelopoulos & Bates 2021
Cost functions, In the separable case, the total loss was defined as the sum of penalties assigned to the con- stituent elements of a given set. We applied a uniform random assignment from the set 4 : E [4] to each label across all datasets. In the non-separable case, we evaluate two different loss functions over the label spaces: maximum distance and coverage loss. Each of these losses is defined in terms of pre-specified hierarchy. For the iNaturalist and Fitzpatrick datasets, we use the expert- -effind hierarchies that accompany each dataset. For CIFAR-10M 100 and ImageNet, we synthetically generate the hierarchy using hierarchical clustering on the repre- sentations obtained from the classifier's 's final layer. The maximum distance is defined as the largest pairwise graph distance along the hierarchy between any two elements within a predicted set. The coverage loss is de- fined as the number of categories that are intersected by predicted set, where each category corresponds to group of labels at the second-to-lan level of the hierarchy. Both losses capture notions of homogeneity within the predieted sets with respect to the hierarchies.
Comparison against other forms of uncertainty quantification Our methods, being conformal-based, do not rely on a precise estimation of p(ylz)) However, to benchmark against other approaches to uncertainty quantification (many of which rely on properly estimating p(y(z)) we conducted experiments to compare our method against predictions sets obtained from Bayesian estimates of p(y|z). We chose Bayesian due to its
8

---

Published a conference paper ICLR 2025
1.0
6-
5-
0.8
5.
0.6
3-
0.4
2-
2
0.2
1-
Ritpattick
RtzpatrickClean
0.0
Fltzpatrick
FttzaarickClean
Rizpatrick
RtzpatrickClean
Base Conformal
Separable Penalized Ratio
Greedy Optimizer on Coverage
Penalized Conformal (optimal
Greedy Optimizer on
Figure 3: Comparisons between different methods for the 1) separable loss case using the sum of penalties in each set the loss metric (left), 2) non-separable loss case using maximum penalty in the set as the metric (middle), and 3) hierarchy coverage function loss case using the intersection of the set with each branch of the hierarchy as the metric (right). Numbers the median-of-means across 10 different runs. See Appendix D for all datasets.
Sum Penalties
Penalties
Coverage
80
reptinci
Cawtaage
Loss
60
coeforral
contoral tnatina
0.6
40
0.1
0.3
0.4
0.1
0.2
0.3
0.4
0.1
0.2
0.3
0.4
Accuracy
Figure 4: Measured impact of base model accuracy on downstream optimization values on all losses considered for the cleaned Fitzpatrick dataset. Shows median-of-ment -oammns across 10 different runs for each configuration. Results demonstrate that all methods improve with a higher quality model-i.e., a model with higher accuracy.
widespread adoption within the uncertainty quantification community. The results can be found on the appendix in Figure
4.2 RESULTS
In Figure 3 (and on the appendix) together with Tables 4 lower cost function than the hase conformal method in all considered datasets. This validates hypothesis that engineering a non conformity score that aligns with a downstream loss results in an utility aligned confor- mal predictor. the separable case, we observe that our method outperforms the baseline conformal prediction sets, both when using the Separable Penalized Ratio and Penalized Conformal algorithms (Table 4 and Figure 3.. For the non-seaarbble case, we also observe that our method outperforms base conformal prediction for both considered loss functions (Table [3] and Figure 3 Additionally, in Table [ (and Tables [4] pendix), we demonstrate how our methods successfully provide valid coverage across all set sizes, ensuring robust coverage that is relatively uniform across all predictions. Similarly, Figure well as in Figure in the Appendix) illustrates how our methods achieve adaptability, yielding larger sets for instances where the model is less confident in its predictions. These results demonstrate that our framework indeed provides principled approach to take advantage of latent hierarchical abstractions, enhancing an agent's 's ability to navigate complex problems, illustrated in Figure 1
5 6
we observe that our methods have significantly
13 in the Ap-
We further note that, penalized conformal methods, when appropriately tuned, tend to outperform the other methods we tested on CIFAR 100, ImageNet, and iNaturalist, likely due to the large dataset sizes. This can be attributed to the fact that both the separable penalized ratio and the greedy optimizer approach assume the Bayes classifier is known, whereas penalized conformal methods do not. However, we posit that for the FitzPatrick dataset, there was not enough data to reliably estimate the optimal X, making our principled methods perform better and a suitable off- she-f solution for an initial iteration on this dataset.

---

Published conference paper ICLR 2025
True Labe Softmax
Penal ized Conformal
Greedy Optimizer Max
Greedy Optimizer on Coverage
Dataset
0.5
Fitzpatrick
FitzpatrickClean
2-4
5-9
10+
2-4
5-9
10+
1
2-4
5-9
10+
Size
Figure 5: Mean and standard deviation from the predicted softmax score for the class as a function of predicted set size. The downward trend indicates that the predicted set size increases with the difficulty of the classification task for the model, highlighting the relationship between model uncertainty and set size.
Size
Base Conformal count
Penalized Conformal
Greedy Coverage
Greedy Max
coverage
count
coverage
count
coverage
count
coverage
1 2 to 4
46
0.87
55
0.87
29
1.0
22
0.82
77
0.95
196
0.91
50
0.96
37
0.89
5 to 9
102
0.88
219
0.89
48
0.98
36
0.92
10 to
588
0.95
383
0.90
677
0.98
713
0.96
Table 1: Conditional statistical coverage by set size for the cleaned version of the Fitzpatrick dataset and target coverage of 0.9. Penalized conformal method results are reported for the optimal À value. We also included the number of samples that belong in each size bucket. The average set sizes for all methods and datasets are available in Table I0
We additionally investigate whether method can be used in situations where the underlying classifier is noisy. Such robustness can be useful for application settings where practitioners only have access to imperfect, noisy models due to practical concerns (c.g., small amounts of labeled, domain-specific data, etc). In Figure 4 we conduct ablation studies on the Fitzpatrick dataset with a classifier whose training was stopped at different levels of performance to understand the impact of the base model accuracy on downstream optimization values. As expected, we observe that the expected loss of our method decreases with improved classifiers (i.c., having a better base classifier better). But more importantly, we observe that our method benefits from decision- focused conformalization even when the underlying classifier is very noisy, demonstrating that our method outperforms base conformal methods at any level of aceuracy of the underlying classifier. This demonstrates that even with noisy classifiers, our methods will still produce prediction sets with higher utility and desired coverage.
CONCLUSION
We introduce methods that extend traditional conformal prediction to take into consideration utility functions that represent consequences of real-world actions and outcomes. The approach can be harnessed to provide decision-maken with guidance in the form of high-confidence prediction sets that contain homogeneous ele- ments with respect or, at the very least, not significantly different in their outcomes according with the utility metric.
external utility metric, ensuring that the actions taken based these sets effective
We illustrate the operation of our methods via a real-world case study in dermatology diagnosis. In this context, the utility metric reflects homogeneity concerning the hierarchy of dermatological pathologies. That is, good sets are those that ideally form subsets within this established hierarchy. Our method produces prediction sets with valid statistical coverage and a coherent clinical interpretation. The sets generated correspond to subsets of these hierarchical abstractions, aligning with, and potentially supporting, diagnostic and therapeutic reasoning. Our method outperforms existing approaches, and appear robust challenges such scarce and noisy data, as demonstrated by the performance on the Fitzpatrick dataset. We hope that our work will provide a valuable stepping stone for future research that leverages uncertainty quantification to support decision makers in high- stakes settings.
Reproducibility Statement We are committed to making our research reproducible, allowing future work to build upon our findings. All code and instructions necessary to reproduce all experiments and results is available
10

---

Published a conference paper ICLR 2025
at [hidden]. Further, all artifacts (i.e., data, model scores, metadata) and hyperparameter configurations conpeggaaamtterrcoooigaaatotoooo require for running the experiments is also publicly released at [hidden].
AKNOWLEDGMENTS
We thank Naveen Raman for comments on the manuscript. This work was supported in part by the AI2050 program at Schmidt Sciences (Grant #G-2264474) and the AI Research Institutes Program funded by the Na- tional Science Foundation under Al Institute for Societal Decision Making (AI-SDM), Award No. 2229881.
Carlos Patiño's presentation of this paper at ICLR 2025 was financially supported by the Amsterdam ELLIS Unit.
REFERENCES
Anastasios Angelopoulos, Stephen Bates, Jitendra Malik, and Michael I Jordan. Uncertainty sets for image classifiers using conformal prediction. arXiv preprint arXiv:2009. 2020a.
Anastasios N Angelopoulos and Stephen Bates. gentle introduction to conformal prediction and distribution- free uncertainty quantification. arXiv preprini 107.0:2
Anastasios N Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. arXiv preprint arXiv:2208.028814 2022.
Anastasios Nikolas Angelopoulos, Stephen Bates, Jitendra Malik, and Michael Jordan. Uncertainty sets for image classifiers using conformal prediction. arXiv preprint arXiv:2009. 1XX1:2009.14193,, 2020b.
Abhilash Chenreddy and Erick Delage. End-to-end conditional robust optimization. arXiv preprint arXiv:2403.04670, 2024.
Abhilash Reddy Chenreddy, Nymisha Bandi, and Erick Delage, Data-driven conditional robust optimization. In Advances Neural Information Processing Systems, volume 35, pp. 0025-95255553 -153,,5377220 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Krishnamurthy Dvijotham, Jim Winkens, Melih Barsbey, Sumedh Ghaisas, Robert Stanforth, Nick Pawlowski, Patricia Strachan, Zahra Ahmed, Shekoofeh Azizi, Yoram Bachrach, et al. Enhancing the reliability and accuracy of ai-enabled diagnosis via complementarity driven deferral to clinicians. Nature Medicine, 29(7): 1814-1820, 2023.
Adam N Elmachtoub and Paul Grigas. Smart "predict, then optimize". Management Science, 68(1):9-26. 2022.
Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-level classification of skin cancer with deep neural networks. nature, 542(7639): 115-118, 2017.
Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. The limits of distribution- free conditional predictive inference. Information and Inference: A Journal of the IMA, 10(2):455-482, 2021.
Matthew Groh, Caleb Harris, Luis Soenksen, Felix Lau, Rachel Han, Aerin Kim, Arash Koochek, and Omar Badri. Evaluating deep neural networks trained clinical images in dermatology with the fitzpatrick 17k dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 1820-1828, 2021.
Eric Horvitz and Adrian Klein. -bility-based abstraction and categorization. In Uncertainty in Artificial Intelligence, pp. 128-135. Elsevier, 1993.
Nathan Kallus and Xuan Mao. Stochastic optimization forests. Operations Research, 71(1):1-18, 2023.
11

---

Published conference paper ICLR 2025
Roger Koenker and Gilbert Bassett Jr. Regression quantiles, Econometrica: journal the the Econometric Society, pp. 33-50, 1978.
Roger Koenker and Kevin F Hallock. Quantile regression. Journal of economic perspectives, 15(4):143-156, 2001.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto, 2009.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Jure Leskovec, Andreas Krause, Carlos Guestrin, Christos Faloutsos, Jeanne VanBriesen, and Natalie Glance. Cost-effective outbreak detection in networks. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 420-424- 2007.
Charles Lu, Andréanne Lemay, Ken Chang, Katharina Höbel, and Jayashree Kalpathy-Cramer. Fair confor- mal predictors for applications in medical imaging. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 18008-12015 2022.
David C MacKay. Probable Networks and Plausible Predictions - A Review of Practical Bayesian Methods for Supervised Neural Networks. Network: Computation in Neural Systems, 6(3):469-55) 100669555-.10 1995.
David John Cameron MacKay, Bayesian Methods for Adaptive Models. PhD thesis, California Institute of Technology, 1992.
Jayanta Mandi, Peter J Stuckey, Tias Guns, et al. Smart predict-and-optmmize for hard combinatorial optimiza- tion problems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 1603-1610, 2020.
Radford M. Neal. Bayesian Learning for Neural Networks. PhD thesis, University of Toronto, 1996.
Yash P Patel, Sahana Rayan, and Ambuj Tewari. Conformal contextual robust optimization. In International Conference on Artificial Intelligence and Statistics, pp. 2485-2492 PMLR, 2024.
Thomas Pouplin, Alan Jeffares, Nabeel Seedat, and Mihacla van der Schaar. Relaxed quantile regression: Prediction intervals for asymmetric noise. arXiv preprint arXiv:2406.03255. 2024.
Meng Qi, Paul Grigas, and Zuo-Jun Max Shen. Integrated conditional estimation-oppiiizztion arXiv preprint 1rX1v:2110.13311, 2021.
Kevin Ren, Yewon Byun, and Bryan Wilder. Decision-focused evaluation of worst-case distribution shift. arXiv preprint arXiv:2407.03557, 2024.
Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adaptive coverage. Ad- vances Neural Information Processing Systems, 53:3581-35911 2020.
Uday Sadana, Abhilash Chenreddy, Erick Delage, Antoine Forel, Emma Frejinger, and Thibaut Vidal. A survey of contextual optimization methods for decision-making under uncertainty. European Journal of Operational Research, 2024. Forthcoming.
Mauricio Sadinle, Jing Lei, and Larry Wasserman. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association, 114(525):223-225) 2019.
Sanket Shah, Kai Wang. Bryan Wilder, Andrew Perrault, and Milind Tambe. Decision-Focusee learning without making: Learning locally optimized decision losses. Advances in Neural Information Processing Systems, 35:1320-1333 2022.
Natasa Tagasovska and David Lopez-Paz. Single-model uncertainties for deep learning, Advances in neural information processing systems, 32, 2019.
12

---

Published conference paper ICLR 2025
Mingxing Tan. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint 19X,::1905.1196., 2019.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7769,8778, 2018.
Anirudh Vemula, Katharina Muelling, and Jean Oh. Social attention: Modeling attention in human crowds. In 2018 IEEE international Conference Robotics and Automation (ICRA), pp. 4601-4607. IEEE, 2018.
Vladimir Vovk. Conditional validity inductive conformal predictors. Asian conference on machine
learning, pp. 475-490. PMLR, 2012.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in random world, vol- ume 29, Springer. 2005.
Kai Wang. Sanket Shah, Haipeng Chen, Andrew Perrault, Finale Doshi-Velez, and Milind Tambe. Learn- ing mdps from features: Predict-then-optimzz for sequential decision making by reinforcement learning. Advances in Neural Information Processing Systems, 34:8795-8806, 2021.
Larry Wasserman. Bayesian model selection and model averaging. Journal of mathematical psychology, 44 (1):92-107, 2000.
Bryan Wilder, Bistra Dilkina, and Milind Tambe. Melding the data-decisions pipeline: Recision-focused learning for combinatorial optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1658-1665, 2019.
Andrew Gordon Wilson and Pavel Izmailov. Bayesian Deep Learning and a Probabilistic Perspective of Gen- eralization. Advances Neurul Information Processing Systems, 2020.
Christopher Yeh, Nicolas Christianson, Alan Wu, Adam Wierman, and Yisong Yue. End-to-end conformal calibration for optimization under uncertainty. arXiv preprint arXiv:2409.20534 2024.
Tao Yin, Chenzhengyi Liu, Fangyu Ding, Ziming Feng, Yuan, and Ning Zhang. Graph-based stock corre- lation and prediction for high-frequene trading systems. Pattern Recognition, 122: 122:108209, 2022.
Keming Yu, Zudi Lu, and Julian Stander. Quantile regression: applications and current research areas. Journal of the Royal Statistical Society Series D: The Statistician, 52(3):331330,, 2003.
A APPENDIX
A.1 PROOF PROPOSITION ONE
Proof. Formally, Let P absolutely continuous to == / x pl, where μ is the Lebesgue measure and μ the counting measure. Let H v(x,v): y (x)) and (z)) {v E H(x)).
E[C(H(=)) J ((H())))(z)dz
Σ (ly)(<))
yEH(z)
((y)p(x)dr
((yp(a)d
и
[
Ely)dv
H
13

---

Published conference paper ICLR 2025
Therefore, as a consequence the Neyman- SaymannPeaasoo Pearson Lemma, we can guarantee that the set H that minimizes
Su (lyddv while fulfilling In dP
is
p(yla) ((y)
x(y,)): 2 ta
Where la is the quantile for r(vl)) Egg
A.2 PROOF PROPOSITION TWO
Proof. It is worth notice that for both the maximum and coverage cases ((Sx(z)) < B for every À and every x (different constant in each case though). Then, If the of parameters H. is finite, then combining Hoeffding's and the union bound:
P(sup É(A) - B[C(A)])) <
Thus, if en
(2 /8) 2n
then P(supxlê(A) - B[C(A)]] > cn) < and thus with probability at least
6:
[[((A) & (()) +en & L())+ B[C(A*) + 2
A.3 PROOF PROPOSITION THREE
Proof. It follows as a consequence of the independence of the calibration data points and the application of the split conformal algorithm Angelopoulos & Bates (2021): Vovk et al. (2005)
A.4 PROOF PROPOSITION FOUR
Proof. It follows consequence of the independence of the calibration data points and the application of the split conformal algorithm Angelopoulos & Bates (2021); Vovk et al. (2005)
A.5 PROOF COVERAGE FUNCTION IS NOT SEPARABLE
Proof. Suppose separable, i.e, C(S) = E, ins {ly) for some particular this would imply that C(1y)) Ely). Let C be one of the categories used to define L, then:
((C) Σ I(CnC)
C.EC
=1
However, L((v)) E(y), thus L(C) [C), a contradiction if C' has more than one element.
B EXPERIMENTAL DETAILS The training details for the neural network classifier by dataset are:
14

---

Published conference paper ICLR 2025
Dataset
Epochs
Batch Size
Learning Rate
Validation Fraction
Vision Backbone
Fitzpatrick
120
256
0.001
0.1
EfficientNet
Fitzpatrick Clean
120
256
0,001
0.1
EfficientNet.BB
iNaturalist
15
256
0.001
0.1
EfficientNet BO
CIFARI00
32
256
0.001
0.1
EfficientNet BO
ImageNet
256
0.001
0.1
EfficientNer_Bl
Table 2: Hyperparameters used to train the classifiers for each dataset. The configurations are identical except for the number of training epochs.
Dataset
# of Classes
Dataset Size
Base Model Accuracy
Fitzpatrick
114
8289
0.37
Fitzpatrick Clean
49
4709
0.54
CIFAR100
100
50000
0.71
iNaturalist
51
100000
0.90
ImageNet
1000
50000
0.72
Table Details about the datasets we use calibrate and evaluate the sets. For the iNaturalist dataset, we use the labels up to the class level. We divide each dataset into calibration, validation, and test sets with split of 50-25-25% of the dataset respectively.
For both separable and non-separable conformal-based methods, the grid used for mning À was (0.001, 1,
C BAYESIAN COMPARISON
Using variational inference, we trained an EffcientNet Tan 2019) where the last linear layer of the net had Gaussian prior NO, 1) on weights. The differentiable objective used to train this Bayesian neural network, was the negative evidence lower bound and at each training step the weights and the posterior distribution (being more precise the variational approximation) were updated accordingly. Finally for the prediction phase, for each instance z we draw 100 samples from the posterior distribution to produce estimates of p(y æz 0), subsequently we then produce a Monte Carlo approximation of p(y(z). The neural network was trained for 25 epochs using learning rate of 0.001 and batch size of 64 on the FitzPatrick dataset Groh al. (2021).
We now proceed to define the sets S(z)- Let (2) = (ar (z)( (z))) denote the permutation that orders p(y: .r) from greatest to smallest. The prediction set Sp defined as S (*), (ox (w)), where is the smallest integer such that Σ (() æ) > 1 1-a. However, this procedure does not inherently provide statistical coverage. To address this limitation, we conformalized the scores Ek= ((Ve.(=) x) by applying our penalized conformal prediction method with À = and using the Bayesian estimates p(y|z)) The results of this procedure are summarized in Figure
15

---

Published conference paper ICLR 2025
Sum of Penalties
100
Lo05
I
50
Base Conformal
Separable Perialized Ratio
Perialized Conformal
Base 8 Bayesian
Maximum Penalty
0.6
Extimation plyjo
0.4
Non Mon-Bayesian Bayesian
0.2
0.0
Base Conformal
Penalized Conformal
Greedy Optimizer
Base Bayesian
Greedy Oplimizer
Hierarchy Coverage
5.0
2.5
Base Confermal
Penalized Conformal
Groedy Optimizer Method
Base Bayesian
Greedy Optimizer
Figure 6: Comparison of methods when Bayesian model the source of p(y(z). The plots show our methods work well for Bayesian and non-Baye Ba-Besssian approaches because they outperform the base methods in both cases.
Dataset
Base Conformal
Penalized Conformal (optimal X) Separable Penalized Ratio
CIFARI00
29.146 (2.339)
6.915 (0.477)
4.442 (0.374)
ImageNet
150. (12.889)
25.553 (2.551)
5.277 (0.549)
iNaturalist
32.949 (4. (82)
3.855 (0.582)
2.043 (0.474)
Fitzpatrick
86.373 (8.68) (8.680)
72.986 (7.315)
60.805 (1 (11.300)
FitzpatrickClean
34.356 (3,790)
22.257 (2.589)
17.481 (2.236)
Table 4: Comparison our proposals for the separable loss case against base conformal predictions for dif- ferent datasets using the sum of the penalties in each set as the metric. The numbers are the median-of-means across 10 different runs.
Results It can be observed that the Bayesian approach only fails to provide true statistical coverage but also produces prediction sets with a higher average loss. Furthermore, our conformal-basee techniques demonstrate remarkable robustness and flexibility by effectively correcting these deficiencies when applied top of the biased Bayesian estimates of p(y z). This highlights the advantages of our method, as it addresses both the lack of coverage and suboptimal set quality, showcasing its effectiveness once again.
D EXPERIMENTAL RESULTS
16

---

Published conference paper ICLR 2025
150
Base Conformal
Penalized Conformal (optimal A)
Separable Penalized Ratio
Loss
100
I
I
0
CIFAR100
ImageNet
iNaturalist Dataset
Fitzpatrick
FitzpatrickClean
1.0
Base Conformal
0.8
Penalized Conformal (optimal X)
Greedy Optimizer on Max
0.6
Loss
0.4
0.2
0.0
CIFAR100
ImageNet
iNaturalist Dataset
Fitzpatrick
FitzpatrickClean
6
Base Conformal
Penalized Conformal (optimal X)
Greedy Optimizer on Coverage
Loss
2
CIFAR100
ImageNet
INaturalist Dataset
Fitzpatrick
FitzpatrickClean
Figure 7: Comparisons between different methods for the 1) separable loss case using the sum of penalties in each set as the loss metric (top), 2) non-separable loss case using maximum penalty the set as the metric (middle), and 3) hierarchy coverage function loss case using the intersection of the set with each branch of the hierarchy as the metric (bottom). Numbers are the median-of-ment across 10 different runs.
Dataset
Base Conformal
Penalized Conformal (optimal X)
Greedy Optimizer 0.419 (0.003)
CIFAR 100
0.475 (0.003)
0.229 (0.003)
ImageNet
0,432 (0.003)
0.150 (0.002)
0,313 (0.003)
iNaturalist
0.591 (0.002)
0,076 (0.001)
0.570 (0.003)
Fitzpatrick
0.692 (0.005)
0.678 (0,005)
0.502 (0.007)
Fitzpatrick Clean
0.657 (0.004)
0.597 (0.004)
0.515 (0.007)
Table Comparison between different methods for the non- separable loss case across different datasets using the maximum penalty in the set as the metric. The numbers are the median-on -f
ofan--n--e means across 10 different runs.
17

---

Published a conference paper ICLR 2025
Dataset
Base Conformal
Penalized Conformal (optimal X)
Greedy Optimizer
CIFAR 100
3.134 (0.030)
1.054 (0.017)
2.270 (0.023)
ImageNet
3,365 (0.032)
0,960 (0.002)
1.675 (0.013)
iNaturalist
6.072 (0.033)
1.071 (0.003)
4.981 (0.046)
Fitzpatrick
6.134 (0.080)
5,562 (0.057)
2.940 (0.033)
Fitzpatrick Clean
4.634 (0.054)
3.338 (0.092)
3.144 (0.034)
Table 6: Comparison between different methods for the coverage function loss across different datasets using the intersection of the set with the hierarchy as the metric. The numbers are the median-of-mean across 10 different runs.
Dataset
0.01
0.01
0.1
1.0
10,0
COFART00
15.039 (1.121)
6.915 (0.477)
5.644 (0.812)
5.846 (0.570)
6.354 (0.476)
ImageNet
25.553 (2.551)
8.671 (0.477)
7.093 (0.963)
8.389 (1.223)
8,231 (1.035)
iNaturalist
3,855 (0.582)
2.818 (0.310)
2.514 (0.317)
2.092 (0.362)
2.371 (0.256)
Fitzpatrick
71.937 (9.291)
72.986 (7.315)
83.053 (8.408)
77.088 (8.263)
75.293 (9.907)
Fitzpatrick Clean
20.517 (3.350)
22.257 (2.589)
24.088 (2.636)
23.515 (2.816)
23.780 (2.213)
Table 7: Comparison between different values of the hyperparameter À for the proposed Penalized Conformal The numbers are the median-of of-means across 10
method on separable loss case across different datasets. different runs.
Dataset
0.001
0.01
0.1
1.0
10.0
C0
0.471 (0.003)
0.443 (0.002)
0.314 (0.001)
0.261 (0.003)
0.229 (0.003)
ImageNet
0.426 (0.003)
0.374 (0.003)
0.237 (0.002)
0.160 (0.001)
0.150 (0.002)
iNaturalist
0.328 (0.003)
0.173 (0.001)
0.076 (0.001)
0.012 (0.000)
0.000 (0.002)
Fitzpatrick
0.688 (0.004)
0.678 (0.005)
0.687 (0.003)
0.686 (0.003)
0.685 (0.004)
Fitzpatrick Clean
0.606 (0.012)
0.597 (0.004)
0.603 (0.005)
0.596 (0.009)
0.600 (0.010)
Table Comparison between different values of the hyperparameter À for the Penalized Conformal method on the non-separable loss case across different datasets. The numbers are the median-of sitiian-of-manss of the maximum penalty per set across 10 different runs.
Dataset
0.001
0.01
0.1
1.0
10.0
CIFARTON TOO
3.079 (0.028)
2.756 (0.026)
1.704 (0.009)
1.096 (0.012)
1.054 (0.017)
ImageNet
3.279 (0.034)
2.640 (0.031)
1.435 (0.012)
1.053 (0.004)
0.960 (0.002)
iNaturalist
2.401 (0.022)
1.432 (0.009)
1.071 (0.003)
0,940 (0.001)
0.902 (0.008)
Fitzpatrick
6.017 (0.070)
5.562 (0.057)
5.629 (0.082)
5.621 (0.062)
5.613 (0.050)
Fitzpatrick Clean
4.446 (0.104)
3.587 (0.041)
3.456 (0.046)
3.338 (0.092)
3.419 (0.105)
Table 9: Comparison between different values of the hyperparameter for the Penalized Conformal method on the non-separable loss case across different datasets. The numbers are the median-of-means of the coverage penalty per set across 10 different runs.
Dataset
Base Conformal
Penalized Conformal
Greedy Coverage
Greedy Max
CIFAR 100
14.6
5.20
22.2
23.9
Fitzpatrick
41.8
34.7
64.2
64.8
FitzpatrickClean
17.0
10.6
29.4
25.3
ImageNet
74.8
15.1
152
146
iNaturalist
17.1
1.20
23.1
22.5
Table 10: Average set size by method and dataset. The Penalized Conformal method (with an optimized parameter) always results in smaller sets than the Base Conformal method. The table also shows that the greedy methods have larger set sizes than the other two methods, which indicates these methods optimize the utility function the expense of larger sets. However, the predictive value of the sets from the greedy methods is strong, as shown by their conditional coverage and utility function values.
18

---

Published conference paper ICLR 2025
Size
Base Conformal
Penalized Conformal
Greedy Coverage coverage
Greedy Max
count
coverage
count
coverage
count
count
coverage
1
1041
0.94
3285
0.94
1059
0.90
1025
0.94
2 to 4
2665
0.97
4225
0.93
2168
0.97
2145
0.97
5 to 9
2152
0.99
2542
0.96
1224
0.99
1198
0.99
10 to 49
4948
0.99
1873
0,94
5051
0.99
4563
0.99
50 to 99
698
0.99
15
1.00
1929
0.99
2517
0.98
Table 11: Conditional statistical coverage by set size for the CIFAR-100 dataset and target coverage of 1 0.9. The results for the penalized conformal method are reported for the optimal value. We also include the number of samples that belong to each size bucket. We also include the number of samples that belong to each size bucket.
Size
Base Conformal coverage
Penalized Conformal coverage
Greedy Coverage coverage
Greedy Max
count
count
count
count
coverage
1446
0.99
20553
0.95
1089
0.99
1122
0.99
2 to 4
3776
1.0
2740
0.89
784
1.0
736
1.0
to 9
3600
1.0
427
0.89
3458
1.0
1440
1.0
10 to 49
13516
1.0
74
0.96
16353
1.0
18967
1.0
50 to 99
206
1.0
0
0.0
908
1.0
226
1.0
Table 12: Conditional statistical coverage by set size for the iNaturalist dataset and a target coverage
0.9. The results for the penalized conformal method reported for the optimal À value. We also include the number samples that belong to each size bucket.
Size
Base Conformal count
Penalized Conformal coverage
Greedy Coverage count
Greedy Max coverage
coverage
count
coverage
count
967
0,95
3608
0.95
894
0.94
882
0.95
2 to 4
1854
0.98
2657
0.94
1744
0.98
1703
0.99
5 to 9
1452
0.98
1744
0.96
1253
0.99
1286
0.98
10 to 49
2952
0.99
2866
0.95
2226
0.99
2169
0.99
50 to 99
1186
0.99
625
0.95
992
0.98
1148
0.99
100+
1260
0.99
327
0.95
990
0.99
1159
0.99
Table 13: Conditional statistical coverage by size for the ImageNet dataset and target coverage of 0.9. The results for the penalized conformal method are reported for the optimal À value. We also include the number of samples that belong to each size bucket.
Size
Base Conformal count
Penalized Conformal coverage
Greedy Coverage coverage
Greedy Max
coverage
count
count
count
coverage
19
0.95
8
0.75
24
0.79
22
0.86
2 to 4
42
0.90
33
0.97
41
0.71
31
0.84
5 to 9
54
0.89
84
0.92
57
0.82
55
0.91
10 to 49
538
0.89
725
0.87
131
0.85
227
0.90
50 to 99
459
0.92
295
0.94
711
0.94
298
0.91
100+
1.00
0
0.00
103
0.97
447
0.96
Table 14: Conditional statistical coverage by set size for the Fitzpatrick dataset and a target coverage of a 0.9. The results for the penalized conformal method reported for the optimal À value. We also include the number of samples that belong to each size bucket.
19

---

Published conference paper ICLR 2025
Softmax
Penalized Conformal
Greedy Optimizer on Max
Greedy Optimizer on Coverage
1.0
Detaset
CIFAR100
0.5
ImageNet
INaturalist
2-4
5-9
10+
2-4
5-9
10+
2-4
5-9
10+
Set Size
Figure 8: Trend how the predicted softmax score the true class decreases as the set size increases. The downward trend indicates the size of the sets grows the model has more trouble classifying an image.
20


Reviews:
[{"review_id": "g5leeXUVL0", "reviewer": "Reviewer_rCi4", "summary": "This paper presents a novel framework for decision-focused uncertainty quantification, integrating conformal prediction with downstream decision-making considerations. By introducing decision loss into the prediction process, the authors create a conformal method that offers both standard statistical coverage and improved utility for specific applications, such as healthcare diagnostics.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The setting of this paper is interesting that taking the decision loss in conformal prediction pipeline.\n2. This paper effectively bridges conformal prediction and downstream decision making by incorporating user-specified utility functions, providing both theoretical guarantees and practical applicability. The authors comprehensively address both separable and non-separable decision losses.", "weaknesses": "1. The motivation for the study and its illustrative example could be more clearly developed. The authors present the example of the Fitzpatrick dataset to explain their approach, but the rationale for selecting a loss function that reflects the hierarchical homogeneity of dermatologic pathologies remains somewhat unclear. It would benefit the reader if the authors further explained why this hierarchical approach yields a more interpretable clinical result. In addition, I suggest moving this example to the introduction or background section to establish the relevance of the study more clearly.\n\n2. The method relies on applications that require a decision loss function; however, the necessity and specific contexts for using a decision loss function should be more thoroughly explained. The authors claim that their method \"outperforms existing approaches\" in terms of decision loss; however, since the loss function is central to their conformal prediction adaptation, one would expect it to outperform basic conformal approaches that don't consider decision loss. An explanation of why certain applications requires such a loss function would clarify the broader applicability. For example, in Section 3.1, the authors present a medical example where penalties are associated with the cost and complexity of each test. However, the experimental datasets largely use homogeneity-focused loss functions, so it is unclear why this type of loss should apply to datasets such as CIFAR100. The authors might consider providing additional examples of potential applications and elaborating on appropriate loss functions for each.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6bn8AYaTgH", "reviewer": "Reviewer_2y3o", "summary": "This paper presents a new framework that merges conformal prediction with decision-focused learning to create prediction sets that optimize decision-making while maintaining statistical coverage at various levels. The method addresses a gap in current techniques, which often ignore how predictions affect real-world decisions, especially in high-stakes areas like healthcare. The authors propose two algorithms: a penalty-based approach for separable losses and an optimization method for non-separable losses without hyperparameters. They provide theoretical and empirical evidence, including a healthcare case study, to show the effectiveness of their approach.", "questions": "- For the penalty-based approach in separable losses, can you elaborate on the efficiency of grid-search tuning for large datasets? Are there faster alternatives?\n- Could you clarify how the fixed order of adding elements to prediction sets affects performance for complex utility structures? Would alternative scoring or ordering strategies improve flexibility for non-separable losses?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Originality: Novel integration of decision-focused loss into conformal prediction, addressing a key gap in uncertainty quantification for high-stakes applications.\n- Quality: Robust theoretical grounding and empirical results, demonstrating significant decision-loss reduction across multiple datasets.\n- Clarity: The paper is well-structured, with a logical flow from motivation to problem formulation, methodology, and results. Also explains previous related concepts well\n- Significance: Highly relevant for domains like healthcare, enhancing prediction sets to support actionable, utility-aligned decisions.", "weaknesses": "- Lacks comparisons with alternative uncertainty methods like Bayesian inference, limiting context on the framework’s unique advantages.\n- Tuning for separable loss is computationally intensive; more efficient tuning methods would improve usability.\n- Focuses heavily on healthcare; additional applications in other high-stakes areas would demonstrate broader applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "faSFTz3ES5", "reviewer": "Reviewer_c9L1", "summary": "This paper tackles a problem in AI decision-making and how to make AI predictions more useful for real-world decisions while maintaining reliability. Instead of just giving a set of possible predictions (like in conformal prediction methods), this work creates prediction sets that make practical sense for decision-makers. The paper suggests a solution that balances probability and cost using tunable parameters and another elegant solution that doesn't need to find this tunable parameter and automatically finds the best balance of likelihood and cost associated with the class.", "questions": "- The greedy approach overall seems to be doing worse than the approach that uses a learnable $\\lambda$ parameter. Could the authors please clarify the advantages of the greedy approach? \n- Can you provide theoretical bounds for the performance between greedy optimizer vs optimal solution for non-seperable losses ? \n- Is there a relationship between base model calibration and decision loss? \n- Did you consult medical experts about the clinical relevance of your prediction sets?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- This is a strong and mathematically sound work that extends traditional conformal prediction work by generating sets of utility functions that capture the associated likelihoods and costs, which is crucial for decision-making.", "weaknesses": "- The results section is limited regarding insights provided and evaluated only on 4 common datasets. It would also be a nice insight to see the prediction sets for the experiments as a qualitative analysis. \n- It will be interesting to see how this approach works for more challenging datasets with complex hierarchical structures. \n- Limited ablation studies\n-There is no discussion on the loss functions choices and design beyond maximum distance and coverage", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wcfkECzPSz", "reviewer": "Reviewer_arZB", "summary": "The paper aims to generate prediction sets that maximize a utility function while having coverage guarantees. It extends standard approaches in conformal predictions with novel conformity scores that integrate the utility of the prediction set. Several utility functions are considered, and the approaches differ depending on whether they are separable or inseparable. The marginal coverage guarantees of conformal prediction can be preserved in both cases. Experiments on three classification datasets and diverse utility functions show that the methods achieve higher utility than the conformal prediction approach that uses the standard non-conformity score.\n\nMy evaluation of the paper is mixed. On the one hand, I find the notion of utility of prediction sets very interesting and novel (to my knowledge). On the other hand, I find the paper confusing in some parts, especially in its framing as a \"decision-focused\" approach, which refers to a stream of literature that provides structured predictions by integrating a constrained optimization problem.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The main novelty of the paper is providing prediction sets that maximize a user-specified utility function. If the utility function is the size of the set, we recover the traditional conformal prediction setting. However, the approach is more general. It can consider for instance the hierarchy / categories of the labels in the prediction set. The example in Figure 1 works quite well to illustrate the value of the obtained sets, which are more informative for users.\n\nFrom a methodological perspective, the algorithms and non-conformity score are quite simple (which is an advantage) and are shown to preserve the coverage guarantees of conformal predictions. The experiments also suggest that they provide good performance.", "weaknesses": "I have two main concerns.\n\nFirst, I find the framing of the paper as a decision-focused approach misleading and confusing. This is for two reasons. It hides the novelty of the paper, which is its focus on prediction sets with utility. It also makes it seems that the paper is focusing on decision-making in the same vein as the cited works of  Mandi et al., 2020; Elmachtoub & Grigas, 2022; Wang et al., 2021, etc. However, this is not the case because (a) the loss function is completely independent of the true label (hence, there is no notion of accuracy / task loss / decision loss / regret of the prediction set) and (b) the prediction sets are unconstrained since the coverage guarantee is achieved by the conformalization procedure. Decision-focused learning typically deals with the challenges of having to output a decision that is both heavily constrained (linear or combinatorial constraints) and provides good task loss thanks to end-to-end training.\n\nAn interesting direction to make the problem task-focused / end-to-end would be to consider Equation (1) as the task and train the ML classifier in an end-to-end fashion to minimize this task loss.\n\nI argue that the paper's main focus is conformal predictions with utility and does not have much to do with decision-focused learning. My second main concern is that the current analysis of the prediction sets is a bit light. The experiments do not show how large the prediction sets are (which is a common utility metric in conformal prediction) nor their achieved marginal and conditional coverage. There is likely to be a trade-off between these metrics (size vs. achieved coverage vs. user-specified utility) for having truly informative prediction sets. I would also expect to see benchmarks from the existing literature on conformal predictions, such as the adaptive approaches cited in the paper.\n\nI also have a few minor comments:\n- line 53 \"*the work to date on decision-focused learning has largely neglected uncertainty quantification, concentrating instead on constructing models to optimize point predictions for specific decision tasks*\". This is incorrect. Point forecasts are the focus when the task loss is linear in the predicted parameters. It is not always the case, see among others:\\\nQi et al. (2021). Integrated conditional estimation-optimization.\\\nChenreddy et al. (2022). Data-driven conditional robust optimization.\\\nKallu & Mao (2023). Stochastic optimization forests.\\\nSadana et al. (2024). A survey of contextual optimization methods for decision-making under uncertainty.\\\nas well as the following papers, who all use conformal predictions:\\\nChenreddy et al. (2024). End-to-end conditional robust optimization.\\\nPatel et al. (2024). Conformal contextual robust optimization.\\\nYeh et al. (2024). End-to-End Conformal Calibration for Optimization Under Uncertainty.\n- line 171. The first property states a conditional coverage guarantee (“*for an instance*”), whereas the objective is marginal coverage (what conformal predictions guarantee and what I believe is shown in Figure 2). It could help to formally introduce the problem using Equation (1), which includes a marginal coverage guarantee.\n- The short proofs of Proposition 3 and 4 should be moved to the paper body to clearly show that the coverage guarantee is achieved by the split procedure.\n\n[EDIT:] the authors' response and changes to the manuscript have addressed my main concerns. Therefore, I have raised my score from 3 to 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IQEgCasdn9", "reviewer": "Reviewer_yMsv", "summary": "This paper introduces new methods for uncertainty quantification in classification problems, focusing on conformal prediction techniques that optimize decision loss. Three methods are proposed to address different types of decision losses (separable and inseparable) and are with or without hyperparameters. These approaches retain the coverage guarantee of traditional conformal prediction while achieving significantly lower decision loss, as demonstrated through empirical evaluation on five datasets.", "questions": "- The authors modify the conformal score by incorporating an additional term to account for decision loss. Wouldn't this make the conformal score less informative and potentially increase the size of the conformal sets? In general, doesn't attempting to minimize decision loss tend to result in larger sets which carry less predictive value? Furthermore, for certain losses, is it possible that adding an extra element to the conformal set could directly reduce the decision loss? To address these concerns, I recommend that the authors report the average size of the conformal sets in their experiments.\n\n- The authors focus on split conformal prediction, but there are other conformal prediction methods available [1,2]. Can the proposed modifications be applied to these other methods, or are they specific to split conformal prediction?\n\n- Following the previous point, the lack of adaptiveness in split conformal prediction is a known issue [1]. How does the proposed method affect conditional coverage? Is the decision loss minimized uniformly across all subgroups, or are there significant discrepancies? Since the authors emphasize medical applications, it seems essential to assess the method's performance on specific subgroups in such contexts.\n\n[1] (2019) Conformalized Quantile Regression\n\n[2] (2022) A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper's goal is clear.\n- The simplicity of this method, along with the effort to propose a method without introducing a new hyperparameter, will favor its adoption in practical applications.\n- Empirical results show notable improvement over the standard conformal prediction method.\n- The methods are theoretically well-supported.", "weaknesses": "- The discussion of related work is insufficient, focusing exclusively on conformal prediction literature while neglecting other uncertainty quantification methods (quantile regression, Bayesian methods, ensemble methods...). As the goal of the paper is to \"Incorporate the decision loss into uncertainty quantification\", this omission makes it difficult to assess the significance of the paper beyond conformal prediction.\n\n- Building on the previous point, the paper lacks comparisons with existing methods, even though relevant approaches in conformal prediction appear to address the authors' objectives [1]. Other methods outside of conformal prediction also exist [2], and while they may not offer the same theoretical guarantees, comparing their empirical performance would help assess the impact of the proposed methods. Additionally, traditional conformal prediction could be directly applied to classifiers that focus on minimizing decision loss [3].\n\n- Some sections, such as Figure 2, require improved clarity for better readability.\n\n- The experimental details required to reproduce the results are missing. Specifically, classifiers trained on these datasets can be sensitive to variations in training hyperparameters.\n\n[1] (2022) Conformal Risk Control\n\n[2] (2024) Relaxed Quantile Regression: Prediction Intervals for Asymmetric Noise\n\n[3] (2022) Decision-Focused Learning without Differentiable Optimization: Learning Locally Optimized Decision Losses", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)