Conformal Risk Training: End-to-End Optimization of Conformal Risk Control
Authors: Christopher Yeh
Nicolas Christianson
Adam Wierman
Yisong Yue
Venue: NeurIPS 2025 poster ()
Keywords: conformal risk control
decision-focused learning
conditional value-at-risk
CVAR
Primary area: probabilistic_methods
TL;DR: We train models end-to-end with risk control for the broad family of optimized certainty equivalent (OCE) risks.

Abstract: While deep learning models often achieve high predictive accuracy, their predictions typically do not come with any provable guarantees on risk or reliability, which are critical for deployment in high-stakes applications. The framework of conformal risk control (CRC) provides a distribution-free, finite-sample method for controlling the expected value of any bounded monotone loss function and can be conveniently applied post-hoc to any pre-trained deep learning model. However, many real-world applications are sensitive to tail risks, as opposed to just expected loss. In this work, we develop a method for controlling the general class of Optimized Certainty-Equivalent (OCE) risks, a broad class of risk measures which includes as special cases the expected loss (generalizing the original CRC method) and common tail risks like the conditional value-at-risk (CVaR). Furthermore, standard post-hoc CRC can degrade average-case performance due to its lack of feedback to the model. To address this, we introduce "conformal risk training," an end-to-end approach that differentiates through conformal OCE risk control _during model training or fine-tuning_. Our method achieves provable risk guarantees while demonstrating significantly improved average-case performance over post-hoc approaches on applications to controlling classifiers' false negative rate and controlling financial risk in battery storage operation.

Conformal Risk Training: End-to-End Optimization of Conformal Risk Control
Christopher Yeh, Nicolas Christianson, Adam Wierman, Yisong Yue
Department of Computing and Mathematical Sciences
California Institute of Technology
Pasadena, CA 91125
fcyeh, nnbrissichrstti... adamw,
Abstract
While deep learning models often achieve high predictive accuracy, their predictions typically do not come with any provable guarantees on risk or reliability, which are critical for deployment in high-stakes applications. The framework of conformal risk control (CRC) provides a distribution free, finite-sample method for controlling the expected value of any bounded monotone loss function and can conveniently applied post-hoc to any pre- rre-trained deep learning model. However, many real- world applications are sensitive tail risks, as opposed to just expected loss. In this work, we develop method for controlling the general class of Optimized Certainty- Equivalent (OCE) risks, broad class of risk measures which includes as special cases the expected loss (generalizing the original CRC method) and common tail risks like the conditional value-at-risk (CVaR). Furthermore, standard post-hoc CRC can degrade average-case performance due its lack of feedback to the model. To address this, we introduce "conformal risk training," an end-to-end approach that differentiates through conformal OCE risk control during model training fine-tuning. Our method achieves provable risk guarantees while demonstrating significantly improved average-case performance post-hoc approaches on applications to controlling classifiers' false negative rate and controlling financial risk in battery storage operation.
Introduction
We study the problem of training deep learning models that used for potentially risky downstream decision making. For example, in high-stakes tasks such as tumor classification, doctors need models that achieve both good overall classification accuracy and provably bounded false negative rate, ensure that the health risk of a false negative prediction (i.e., misclassifying tumor as benign) is sufficiently prioritized in model predictions. In such settings, important design unified approach (trained model and decision-making policy) that simultaneously controls for the desired level of risk while maximizing the utility of downstream decisions.
One promising paradigm risk control. Given a (pre-trained) model whose predictions are used by decision policy parameterized by À E A. the goal is to choose À such that [[[())] < for some loss function and risk level a. Many common goals can be framed as risk control problems: bounding the false negative rate of classifier, producing predictive uncertainty target coverage level, ensuring factuality of large language model outputs, etc. [6, 29].
that satisfy
The conformal risk control (CRC) method [6] introduces sufficient criterion for solving the risk control problem when the loss function monotone. While CRC is elegant and simple, the original formulation has several limitations. First, CRC is limited to controlling the expected loss of L, while more general notions of risk are critical for nigh-stakes problems in the real world. Notably, the
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

original paper on CRC [6] poses an open question on how to tackle more general notions of risk beyond expected loss, such as the conditional value-at-risk (CVaR). Second, because CRC is purely applied post-hoe (i.e., given a pre-trained model) to model outputs without providing any feedback to the model, it may significantly degrade model performance.
In this paper, we propose a theoretical and algorithmic framework, called conformal risk training, that extends CRC to enable end-to-end training and is applicable to tail risk formulations such as CVaR. Our key contributions are as follows:
1. First, develop risk control method for controlling the general class of optimized certainty- equivalent (OCE) risks [10, 11], a broad class of risk measures that includes as special cases the expected loss (generalizing the original CRC method) and the conditional value-at at-rssk (CVaR) [50]. In particular, the CVaR formulation partially answers an open question posed by the original CRC paper [6]. The key insight is that any OCE risk can be bounded by monotone transformation of the loss, thereby preserving the monotonicity required for CRC-style methods.
2. Second, we propose conformal risk training. method that trains a model and the conformal risk control procedure end-to-en mddto-een. Our method substantially generalizes the method of conformal training of uncertainty sets [47, 51] to the conformal risk control setting. This trains the model to be "risk aware"-iee., it learns to produce predictions that maximize performance while minimizing downstream risk.
3. Finally, we demonstrate empirically that fine-tuning models using conformal risk training leads to significant performance improvements while guaranteeing satisfaction of risk constraints. We present results for maximizing model specificity while controlling false negative rate on tumor image segmentation task, as well as maximizing average profit while controlling tail-risk of losses in battery storage operation task.
Related work. Previous works that explore post-hoc conformal risk control either bound the expectation of monotone loss function, provide a high-probability bound for (possibly) more general losses. We use the term "post-hoc" to refer to procedures that are applied to the outputs of a pretrained prediction model without further fine-tuning of the model. Within the class of bounds on expected losses, conformal prediction (CP) [49, 45, 4] bounds expected miscoverage loss for set-valued predictions, whereas conformal risk control (CRC) [6] generalizes CP to bound the expectation of any bounded monotone loss function. For high-probabiln
risk bounds, Risk- Controlling Prediction Sets (RCPS) [9] bound the expected loss of set-valued predictors for monotone losses, whereas Learn Then Test (LTT) [5] bounds more general risks and non-monotone losses at the cost of typically looser bounds from having to apply family-wise error correction procedure. Recently, [16] developed a high-probability bound for the family of distortion risk measures which includes CVaR. While [6] shows how to convert a -ig--probability risk bound to an expected risk bound, it is not clear that their methodology can be extended to directly bound more general risks like CVaR. As such, to the best of our knowledge, our work is the first conformal-style approach that provides a certain (as opposed to high-probability) bound on risk measures beyond expected loss for monotone loss functions.
Several prior works within the CP literature have also explored calibrating model uncertainty during training. Conformal training [47] and related works [21, 17, 39] introduced methods for incorporating CP differentiably during model training by treating part of each minibatch as pscudo-calibration set. These works primarily focus on reducing the size of calibrated prediction sets. In contrast, our conformal risk training method is the first to incorporate conformal risk control differentiably during model training and compatible with more general performance objectives such as reducing the false positive rate of a classifier or minimizing an expected decision loss. We show in Appendix E that conformal training is a special case of our method.
Beyond the conformal prediction literature, a number of works in the machine learning literature (e.g., [33, 20, 35)) have introduced risk-sensitive objectives into learning: while these methods may, for example, reduce CVaR risk, they do not come with risk control guarantees.
Finally, our work is related to methods from "ppredicttheennoptimiee" [22] and decision-focused learning [37, 44], especially the growing literature on decision-focused uncertainty quantification (UQ). These methods aim to generate prediction sets that optimize some downstream decision- making objective while still maintaining calibration. Several of these methods are applied post-hoc to (potentially) decision-agnosti models [48, 18, 31], whereas others have combined conformal training with decisionnfocusee learning [25, 51]. Our conformal risk training method builds upon

---

this decision-focuue UQ literature: whereas these prior works have largely focused on set-valued predictions and their associated risks, our method allows for more general risk measures.
Outline. This paper is structured as follows. Section 2 introduces our overall problem setting and reviews the standard CRC result. Section 3 introduces broad generalization to the conformal control of OCE risks, of which standard CRC is special case, and shows that this framework can be extended to more general assumptions in the specific case of CVaR risks. Section 4 introduces our conformal risk training procedure and discusses how the gradient of the risk-controlling parameter can be computed. Section 5 highlights key experimental results, and Section 6 concludes. Additional experimental results are presented in Appendix A, and all proofs are deferred to Appendix C.
Notation. [N] denotes the set {1,2, .N.. [x]] denotes the function max(0, 1[:] is the @ denotes partial derivative; d denotes dz
indicator function, while 1, is length-d vector of ones. 8y
the total derivative. (x) is the subdifferential of a function f at a point x.
2 Preliminaries and Background on Conformal Risk Control
A primary goal in the training and deployment of machine learning (ML) models is to achieve both good overall performance and ensure reliable deployment through the control of some notion of risk. To achieve this dual goal, existing approaches follow a two-stage, "Pretrain, then Risk Control" approach. First, an ML model with parameters training objective, such as cross-entropy for classification. Then, after training. the decision-maker applies a post-hoc risk control procedure to the model to guarantee provably bounded risk. Formally, let L: θ x A + I denote (random) mapping from model parameters 0 E O and an "aggressiveness" parameter E A to some loss. The decision-maker seeks to choose the parameter À to ensure that risk-the expectation of the loss L-is bounded at chosen level
θ C RD is trained to minimize standard
E (O, A) S a.
For instance, a tumor image segmentation problem, L may denote the fraction of false negative pixels randomly drawn image, and a is desired upper bound on the false negative rate (FNR). The aggressiveness parameter X E [o, I] can be chosen as the threshold used to distinguish positive predictions from negative ones. Thus, smaller À will yield more positive predictions and lower false negative rate, and a greater À will yield more negative predictions and a higher false negative rate. See Example for more detailed description of this task.
number of post-hoc approaches to control the risk of pretrained machine learning models have been proposed in the literature. Most notable is the approach of conformal risk control (CRC) [6]. which gives a distribution-fres finite-sample methodology to provably enforce the risk bound (1), In the remainder of this section, we will omit the model parameters 0 as an input to the loss function L, since the results hold beyond the case of controlling the risk of machine learning model decisions.
CRC assumes that the decision- maker has a dataset {Li..... LN} of previous loss functions, and that the goal is to control the marginal loss of the next instance: E[Ln+1 S a. This can be done under several mild assumptions.
A IR is closed and bounded set with minimum value Amin min A. The set (L; A + IR. Bizl is of exchangeable, left-continuous random functions. Furthermore, B: A > R is a left-continuous function that almost surely upper bounds each Li pointwise, so that for all i [N + 1). Pr(vA A [x(A) < B(A)) =
Assumption
Assumption 2. The functions ( are almost- nondecreasing, and B is nondecreasing. Assumption 3. The risk control problem feasible. That is, the desired risk level o chosen such that B[ (Amin)) S a.
Under these assumptions, CRC gives the following approach for selecting À to control risk. Proposition 1. Under Assumptions and 2, let E IR be desired risk level and define the set Ả as
A (AEAIM(A)
where h(A) N+1 B(A)
1
+BA)+
+
1=1
Then, for any X E Å, we have EE (A)] S a. Furthermore, if Assumption 3 holds, then choosing = max(Xmin sup ) ensures risk control: EELN+1 a.
3

---

We make three brief remarks, First, the CRC procedure we describe above in Proposition 1 is actually a mild generalization of the original method in [6]; in particular, we allow the risk upper bound B to be function of X, whereas prior work assumes B to be a constant. In practice, this allows us to achieve tighter bounds on the risk. Second, we assume that the functions L, are left-continuous and nondecreasing and we choose À via the supremum, following the convention of [34]; on the other hand, the original CRC paper [6] assumes right-continuous and nonincreasing functions L; and chooses conservativeness parameter À via the infimum. These approaches equivalent up to reflection of the parameter X. Finally, observe that due to monotonicity of the function h(A), the risk controlling parameter À can be computed in practice through bisection search; see Algorithm 3 in Appendix B for a full description of this approach.
Standard CRC (Proposition 1) provides sufficient criterion for choosing the parameter À to ensure bounded marginal (i.e., expected) loss. However, it does not address the control of more general notions of risk; in the next section, we will show that this framework can be broadly extended the control of the general family of optimized certainty equivalent risks.
3 Conformal Risk Control for Optimized Certainty Equivalents
The previous section describes the control of expected loss to achieve the risk bound (1). However, in real-world, high-stakes applications, decision-maken may seek to control their risk beyond just the expected loss, especially if they are sensitive to losses of particular magnitudes. In general, they are faced with the problem of controlling their loss L under some chosen risk measure R, which maps from random variables to IR:
R[[(A]] a
While the CRC methodology described in the previous section can be extended achieve the more general risk control (3) in certain special cases, such as when R is a quantile (see [6, Section 4.2]), there are many other important notions risk which this approach cannot directly accommodate. For instance, in [6, Section 4.2], the authors pose the question of whether CRC or some other approach can be extended to enforce the risk control bound (3) when R is the conditional value-at-risk (CVaR), common risk measure in financial and energy systems applications [42, 32, 38, 36).
In this section, we answer this question in the affirmative, proving that in fact, CRC can be extended to control any risk in the broad family of optimized certainty equivalent risks, defined as follows.
Definition 1 ([10, 11)). A risk measure R mapping a real-valued random variable X to R is an optimized certainty equivalent (OCE) risk measure if R[X] can be expressed as
R[X]
inf t E - t)]. TER
where o R (+oo) is disutility function that is nondecreasing, closed, and convex with p(0) = and Oo(0).
The family of OCE risks includes number of practical and popular risk measures, including mean- variance and entropic risks [33]. Moreover, the conditional value-at-risk can also be shown to be an OCE risk measure with the disutility function ΦCVaR (z) +-g[=]+
Definition 2 ([41, 421). Let X be real-valued random variable, and let F be its cumulative distribution function. The conditional value-at-risk of X at level 8 10, denoted CVaR' [X], is the average value of X on its 8-tail, or the 1 - ô fraction of its largest outcomes. If X has
density, then CVaRt X|= E[X F(X)= 2 ö). For general random variables X, CVaR [X] can be expressed via the variational formula
CVaR*[X] inf t EER
1
LEX -
In Theorem 1, we show that the CRC methodology described in Proposition can be broadly generalized to accommodate any OCE risk measure.
Assumption 4. The OCE risk control problem feasible. That is, the desired risk level a is chosen such that R R|LN+ +1 (Amin)] I

---

Algorithm 1 (Post-hoc) Conformal OCE Risk Control (CORC)
Require:
parameter space A [Amin Amax), functions { L, h upper-bound B, risk threshold R, numerical tolerance
hyperparameter & R
function CORC(A, {Li B, a, e, disutility & R)
Amin. Ā Amax
while X X> do
À + (A X)/2
if h(,, f) I then A A else X+
>h is defined in (4)
return
function CONFORMALCVVARCOTTRO
[Ia] B, a, 6, quantile level ó [0, 1))
o < B(Amin) or t & [B(Amia); al then return Amin
return CORC(A, [L,]- W a, e, @cvaRo)
Theorem 1. Suppose Assumptions I and hold, and fix or, R. Let R be an OCE risk measure
with disutility function Φb Define the exchangeable random functions {Liz : A - B] by
(A)( = + (TTx(A) - 1), and let Be(A) = 1. @ (3(2)=) - t). Define
1
Ba(A) Σ L.dAN d=1
N
he(A)) = N +1
(4)
and let dr = === (A) < a). For every E Åo R[LN+i(A]] S a. Furthermore, if Assumption 4 holds, then choosing x= max Amin: sup År} ensures risk control: R[Ix+1(A)) S
The key insight underlying this theorem is that for any OCE risk, h2 is nondecreasing in X. This structure gives rise the conformal OCE risk control (CORC) algorithm shown Algorithm 1, which computes À using bisection search.
Theorem is a strict generalization of the original CRC methodology in Proposition 1. By choosing the disutility o(x) = ø, the risk measure R is simply the expectation, and we recover the setting of controlling expected risk. In this special case, for all & E R. we have
Vi E [N] L
= L,. B, h₂(A) = h(A), A=
thus recovering Proposition 1 exactly. Moreover, Theorem I answers positively the question from [6] of whether CRC can be generalized to control the CVaR, since the C'VaR is an OCE risk measure. In fact, it is possible to obtain an even more general result in the specific case of controlling the CVaR. Specifically, we may relax the condition in Assumption that the losses L; are nondecreasing.
Assumption All {L.) and are monotonic X. Note that within the set of functions we allow for some functions to be monotonically nondecreasing (e.g.. L1, Lg),
{B} U N+1 (
while others may be monotonically nonincreasing (e.g.. L2, B).
Even under this milder assumption on structure losses L;, we can obtain the following risk control guarantee for the CVaR.
Theorem 2. Suppose that Assumptions I and 5 hold. Fix any 8 10, and E RR. Define 7 in Theorem I using the disutility function ΦCVaR (x) = - [x]+ Then, for every 1 E [B(2mi)) of and E A, we have risk control bound CVaR [[N+1(A)] Sa.
While the above result is written specifically for case of CVaR, we note that the result can be extended to some, but not all, other OCE risk measures; see Appendix C.3 for further details. Theorem motivates the conformal CVaR control algorithm, which we present in Algorithm 1. Note that Theorem 2 is only non-vacuous when B ((Amin) I ar; otherwise, assuming feasibility of the risk control problem (Assumption 4), we can still use Amin to obtain the desired bound.
Compared the standard CRC result (Proposition 1). Theorems and 2 both involve additional hy- perparameter t. For the risk bounds to hold, should not depend on the calibration data {L,,... Lx} used to select the risk control parameter Å. In practice, we recommend using additional held-out set of losses [I/,..., } (e-g., the training set), and picking the that yields the largest risk control parameter this set.
5

---

Algorithm 2 Conformal Risk Training
function
set {a,, u h) parameter space A [Amin Amax) upper-
bound B : A + R, disutility Φ : R R. risk threshold a E IR, numerical tolerance >> 0, initial model parameters 8)
for mini-batch D S [M] do
Randomly split batch: (Dett. D Pmma
Define loss functions L,(0, X) L(ri, Vi, 2) for Deal Compute ((0) = CORC(A, (L,(0,)))EEDa B, a, 6, 6)
CORC defined Algorithm
for E peed do
Define cost functions (, X) := f(2,, 0,
Compute (suh)gradient of objective d0; == de, a(0))/d0
Update 0 using (sub)gradients: Die d0,
While the conformal OCE and CVaR risk control procedures we have proposed in this section enable controlling substantially more general risks than the original CRC framework, such a post-hoc risk control procedure may still come with substantial cost upon ML model deployment. For example, applying CRC to control the false negative rate of a model for tumor segmentation may come at the cost of large false positive rate (see Figure in our experiments). Because CRC is designed be applied post-hoe to a pretrained model, there is no existing approach to remedy this degradation in performance. To improve performance while guaranteeing controlled risk, a better approach would train or fine-tune models subject to a constraint enforcing risk control. Designing methodology to accomplish this goal is the focus of the next section.
4 Conformal Risk Training
Returning to the setting described at the start of Section 2, the loss function L typically depends on the output of an MI. model with parameters
O To explicitly denote this relationship, define h(0, X) (equation 2) and ha (8, X) (equation 4) accordingly in terms of ==1- The CRC (Section 2) and CORC (Section 3) procedures for picking À treat the model parameters 0 as fixed. Thus, call them "post-hoc" procedures.
write L(0, A), and {L, })
However, as we show in our experiments, this separation of pre-training model parameters 0 and performing risk control post-hoc leaves substantial room for improvement. Instead, in this section, we consider the problem of jointly optimizing the model parameters and the risk control parameter A, which we call the end-to-end optimal risk control problem:
min XEA
[[((0,))]
s.t. R[T(0, X) Sa.
where c: +:0% x A + IR is a cost function that measures model performance and is differentiable almost everywhere. Note that are specifically concerned with settings where the cost function f depends on the risk control parameter X. For a standard training objective such as cross-entropy where l only depends on the model parameters but not on the end-to-end optimal risk control problem (5) reduces to standard empirical risk minimization.
The following example concretely illustrates the roles of & and L for the problem of optimizing specificity in tumor image segmentation while controlling false negative rate.
Example 1. In tumor image segmentation, an input image X E X' dx3 (represented as
flattened array of RGB pixels) has a corresponding binary label Y E = (0, 134, where indicates
the presence of a tamor a given pixel location. Let fo X + [0, 1]d denote a segmentation model parameterized by 0 that outputs predicted probubility that each pixel is tumorous. If E [0, 1] is the decision threshold, let (0, y denote the binarized prediction, i.e., for all j E [d]. (0, A); 1X) 2 A). Then, the fraction of false negative predictions is
I(0,A) A) = IY A (O, X) =1
1 Y
Σ 1[fo(X), 2
Σ 1[fo(X), A1, (6) J:
|
J-Y==1
where |-] counts the number of ones in binary vector. Clearly, L is nondecreasing in X. The expected loss E. (0,, 2)) gives the FNR of the model fo, and we can pick to control the FNR to be less than some target threshold

---

However, in addition to controlling FNR. doctors and patients may also want to reduce false positives (i.e., optimize specificityy. We can approximate the fraction of false positives an image with
((),A)
Σ Y=
(x(X);
)
la Y
T
where T is a temperature hyperparameter.
As in the post-hoe CRC setting, usually do have access the exact distribution of (nor the distribution of f). Instead, we only have a dataset of i.i.d. (or exchangeable) samples & and L; for i [N], from which form the end-lo-end optimal CORC problem:
1 N
min OEO, AEA
((0,)) s.t. (0, a.
We can equivalently express (7) as a bi-level optimization problem. The outer problem minimizes over 0, whereas the inner level chooses cist-contoolling A:
N (0,,)) s.t. he(0, A) S a.
min DEG
Σ (000,(00) N
1
where x(0) == arg min
(8)
N
i=1
We shall assume that Assumption 4 holds, so in case the inner optimization problem in (8) is infeasible, we set X(0) Amin to ensure risk control.
To solve (8) via gradient descent, we need to compute the gradient
de; (0, A(0)) = 00 (0, (()) + ax (0, ((0)
0l:
ae,
aA
(0).
(9)
dØ
do
The key challenge lies in computing the derivative d.A this derivative can be computed exactly in many common settings.
aB (0). Fortunately, as we will show in Section 4.1,
Assuming for now that
compute do d (0), we propose the conformal risk training method
(Algorithm 2) for solving the general end-to-end CORC problem.! Inspired by the conformal training method [47], conformal risk training splits each minibatch of training data into two halves, Deal and Drrdd We use the pseudo-calibration
Deal form the loss functions L, and compute the risk control parameter A()), while we use Dpred to form the cost functions é, Note that after training model with conformal risk training, we use fresh calibration dataset (and thus fresh set of Lı...., LN) to compute X()) for use on a test input.
4.1 Computing the gradient in conformal risk training
This section discusses the computation of the gradient requires differentiating through the optimal solution of the lower-level CORC problem in (8), which may in general be nonconvex. Fortunately, it is possible to compute this gradient in many practical settings. In the following (informal) theorem, we give a high-level description of two cases which can compute this gradient; Appendix C.4 for formal statement of the theorem and its proof.
de (0). This is challenging in general, it
Theorem (Informal version of Theorem 3, Appendix C.4), Suppose Assumptions 1 and 2 hold and the inner problem in (8) is feasible. Under mild differentiability conditions, we may obtain
closed-form expression for dAA. a9 (0) in the following two cases:
1- are strictly decreasing in X, and (B) U ( are piecewise constant in X; (ii) if (4) are strictly convex or strictly monotone in X, and (B) ( are convex in X.
(i) (
Note that when considering certain OCE risks like the CVaR in Theorem 3, the nondecreasing assumption (Assumption 2) can be relaxed to monotonicity (Assumption 5) in a similar manner as done in Theorem 2.
The gradient for conformal training [47] follows as a special case of (i) above (see Appendix E). We now show that case (i) also applies immediately to the FNR loss considered in Example 1.
'The name "conformal risk training" was coined by David Stutz on his blog. However, to the best of our knowledge, the first actually develop concrete methodology for end-to-end conformal risk control.
7

---

False Negative Rate
Faise Positive Rate
10a
0.8
10
0.0
past-hoc CRC
cross-entrapy
conformal training
10
0.01
0.05 0.1
0.01
0.05
0.1
0.01
0.05
0.1
Figure Results for the tumor image segmentation problem (Section 5.1) across different FNR thresholds a with 10 random seeds. (left) All three methods show FNR controlled at level a. (middle) Whereas only applying CRC post-hoc results very large FPR, our method (conformal risk training. in green) is able to significantly reduce FPR without sacrificing FNR. (right) Our method generally picks a higher classification threshold than the baselines, suggesting that it is less conservative.
Example 2. Recall the setting of Example I. Given a dataset Deat {X2,, YO drawn ex- changeably from P, let L,(0,), X) = [Y; Y1(0, X)I/|Y Clearly, every Ls is bounded by B Following the conformal risk control procedure, using the form (6) the loss, and noting that each & is strictly decreasing in X, we may pick
N 1+ ++E+ [ (Y:)
<N1 Sa )
A(0 max A 10 1.
Σ
1 Y
1[6(X,),
(10)
N
Assuming that all values fo (Xi), are unique, then according to the proof of Theorem 3(i) (see Appendix C.4), the optimal value x(0) = So [o(X;); for some specific (i, Thus, the gradient is
dA(()) d
d6 db fo(X));
Experiments
In this section, we present experimental results for our conformal risk training method on two problems: (1) controlling false negative rate in tumor image segmentation [6], and (2) controlling CVaR of losses in grid-scale battery storage operation [19]. Code to reproduce our results are available on GitHub, 2 and additional experimental results and details reported in Appendices A and D.
5.1 Controlling false negative rate in tumor image segmentation
We adopt the colonoscopy gut polyp image segmentation problem setup explored in [6, Section 3.1] and described in Example 1. We use pre-trained PraNet [23] as our model fa. and we split images from 4 public datasets
[13], CVC-ColonDB [12], ETIS- ETIS-LaribPolypDDB [46], Kvasir-SEG [28]) into training, calibration, and test splits. In Figure I, we compare the FNR and false positive rate (FPR) on the set across three different models: (1) "post-hoc CRC" applied directly to the pre-trained PraNet; (2) ""cross-etropy"" refers to the fine-tuning PraNet using cross-entropy classification loss and then applying CRC: and (3) "conformal risk training" refers to fine-tuning PraNet using our method described in Section 4. For each model, we try 10 different random seeds for dividing the calibration and test splits, and we vary the target FNR a across three different values (0.01, 0.05, 0.1).
As Figure shows, all three models have their expected FNR controlled at the target level However, for the "post-hoc CRC" and "cross "cross-entropy" baselines, applying post-hoc CRC comes at significant cost the FPR, reaching as high as 80% FPR when the target FNR 1%. In contrast, our conformal risk training method reduces FPR on average by 23-42% across the levels. Furthermore, our method yields larger average values of À than the baselines, suggesting that our method reduces conservativeness while maintaining the risk guarantee.
risk-

---

- 0.9
0.95
0.99
12.5
CVaR A03
7.5
post hoc conformel CVaR control
5.0
fine-tune task loss
conformal risk training
0.0
6
10
5
10
10
0.9
6 0.95
6 0.99
100
% relativ increase
in moan proftt
50
model
25
fine task conformall risk training
-25
10
10
10
Figure 2: Results from the battery storage problem (Section 5.2) across different CVaR quantile levels 5 and risk control thresholds a, with 10 random seeds. (top) All three methods show CVaR risk controlled at the target level a. (bottom) Comparison of the relative increase in profit (i.e., negative task loss) achieved by different methods over the post-hoe CRC baseline. Higher values better.
5.2 Controlling CVaR tail risk from battery storage operations
Energy resource operators seek electricity price forecasting models which deliver optimal expected profit while ensuring that the risk of losses due to poor predictions is adequately controlled. We adopt a grid-scale battery operation problem [19], where battery operator predicts electricity prices Y E IR' over a Ttsppp horizon and uses the predicted prices to decide how much to charge (2 E RT) or discharge (2 E R ) the battery, subject constraints the battery's state of charge expressed in terms of the net charge = E RR. The input features include the past day's prices and temperature, the current day's energy load forecast, and other date-related features. The battery has capacity C, charging efficiency % and maximum charge/discharge rates ctn and cout_ The task loss function ff represents the multiple objectives of maximizing profit, battery health by discouraging rapid charging/discharging (with weight cump), and flexibility to participate in other markets by keeping the battery near half its capacity (with weight ((ex):
f(u 2) = (2 - To y tamp (|>"||²|||=ouu|)
+ 11) + flex ||?
with the constraint set Z containing all (sin zou 2et)
3T that satisfy the constraints
C net 5
C
0 052m 0 S out < out
<
VE VE
in out
27
2
2
Following [19], we set T 24, C = 1, 0.9, cm 0.5, out 0.2, ees 0.1, and ramp 0.05.
I e Z, then clearly A= E Z for all A 10 10. We can thus implement X-dependent decision- maker via "decision-scaling rule 20 A) >ε)(z), where f0(x) = arg min f(jo (x). 2) and do : - y is a neural network pre-trained, to minimize mean squared error in predicting electricity
prices.² 3 We use the task loss A) f(Y,2o <x, A)) as our training objective. which is strictly convex in X. and we apply CVaR control to the financial risk term L(0, A) == A( (X) 20 (X)) Ty. We construct monneane-increasing upper bound B(A)= == 100Å based on the empirical observation that decision-maker &g satisfies (2) (x: 2out (x) y S 100
training and validation sets. Note that Theorem enables us to control the CVaR of L(0, A), which may either be monotone nondecreasing nonincreasing in in contrast, the traditional CRC method (Proposition does not apply, as it only allows for controlling the expectation of L when it is nondecreasing.
'We do not train model to directly map features to decisions because must satisfy constraints. Instead, train ğo predict electricity prices, and then define the decision fo(2) as function of (o

---

Table 1:
Sensitivity of task loss and tail risk relative changes hyperparameter on the battery storage problem. Values given show mean + I standard deviation of the task loss E (xx,y)>Dlf(Y (xx())) and financial tail risk (X. (1 (X) =0th( (X)) YY for 10 models je trained with different random seeds. / is calculated as = (1 + At) where to denotes the optimal value of tuned on the training set.
Ar
8 = 0.9
0.96
0:90
= 0.9
0.95
0:90
= 0.91
0.90
0.90
-0.5
-8.6 3.2
-4.3 1.6
-15 0.5
-0077 4 4.0
-3.9
6.36
-21.4 T.9
-7.7 4: 2.5
4.3
1.8
21.3
10.7
6-3
21.5
8.9
0
-866 32
-43+18 1.6
0.5
-10.7 4.0
-4. 1.3
63
-21.6
7.
29
0.1
8.6
3.2
4.3+1..
0.8
2.9
10.7 4-40
4.6+
6.3
21.5
9.1+2.
0.5
-8.6
32
1.6
0.
7.9
-10.7
-1.1 1.0
63
-21.6
-333
-0.5
1.6 0.3
16
1.2
3.9
4.0
2.8 0.6
T.1
8.0 1.5
5.6 L1
CVaR [
0.3
1.6+0.3
1.3+
0.
4.0+00.8
3.3+
7.
8.1
6.5+1.22
0
0.3
1.6
1.3
0.8
0.8
3.3 0.6
7.0
8.1 11111 1.5
6.6 1.2
0.1
0.3
1.6
1.2+002
0.
4.0
3.4 0.6
7:0
811-15 1.5
6, 1.2
0.5
1.3
1.
7.
0.0 0.0
0.0 0.0
0.8 088+00..
0.0
0.0++.00
2.0 D.7
0.1
0.1
4.1
Because L and B are both convex in À and differentiable almost everywhere in (0, A), X(0) is the solution to a convex optimization problem, and Theorem 3(i)) allows us to compute the derivative dX(0)/d0 by differentiating through the KKT conditions of the convex optimization problem [1].
We compare test set tail risk and task loss across three different models: (1) "post-hoe conformal CVaR control" applied directly to the pretrained price prediction model ÿo: (2) "fine-tune task loss" refers to fine-tuning go using decision-focused task-loss [19] and then applying conformal CVaR control; and (3) "conformal risk training" refers to fine-tuning De using the procedure described in Section 4. For each model, we try 10 different random seeds for dividing the validation and test splits, and we vary the target CVaR tail risk E (2, 5, 10) and the quantile level 8 E (0.9, 0.95, 0.99).
As Figure (top) shows, all three models have their CVaR tail risk controlled the target level a. However, in Figure 2 (bottom), it is clear that whereas fine-tuning using the task loss does not improve average task loss when controlling for tail risk, our conformal risk training method achieves between 7.2% and 22.6% mean improvement in task loss (i.e., higher average profit) compared to the "post-hoc conformal CVaR control" baseline at all tested risk thresholds a and quantile levels
Sensitivity to hyperparameter. As noted in Section 3, conformal CVaR control requires picking hyperparameter
[B( a). which we set the value to that yields the largest risk control parameter \ on the training set. To understand the sensitivity of conformal CVaR control to the choice of f, we computed task loss and the empirical CVaR of financial risk under both relative (Table 1) and absolute (Table 3) perturbations of f away from to. Empirically, we find that the task loss and CVaR are not very sensitive to changes in &; the task loss from perturbed & tends to be close to (within 1 standard deviation of) the task loss from tg.
6 Conclusion
We have developed the conformal OCE risk control method, which a strict generalization of the original CRC procedure. In particular, this allows to directly control the CVaR tail risk, unlike previous works which only control expected losses or provide high-probability bounds. We have also developed conformal risk training, which generalizes the conformal training procedure from conformal prediction to the setting of conformal OCE risk control, and our experiments show significant improvements in model performance over applying post-hoc CRC alone.
Limitations and future directions. The main limitations of conformal OCE risk control are the same limitations that apply to standard CRC: the risk control guarantee only applies to monotone and exchangeable losses. For conformal risk training, while we derive the exact gradient in some common cases, we do not provide complete characterization of when the gradient exists.
Future work may examine the tightness of the conformal OCE risk control bound and consider generalizations of CRC to other families of risk measures such as distortion [16] or coherent risk measures [7]. We believe that conformal OCE risk control will be of particular interest to high-stakes applications in finance, robotics, and LLM alignment, where provable tail risk guarantees are critical.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by NSF grants (CCF-2326609, CNS-2146814, CPS-2136197, CNS- 2106403, NGSDI-2105648), an NSF Graduate Research Fellowship (DGE-2139433), gifts from Amazon, OpenAl, and Latitude AI, and the Caltech Resnick Sustainability Institute.
References
[1] A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and J. Z. Kolter. Differentiable Convex Optimization Layers. In H. Wallach, H. Larochelle, A. Beygelzimer. F. Alché- Buc, E. Fox, and R. Gameett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https / http://aappeeaeeeeeeeeeeeee wwt.pprrr..iipp...m nips cttpps.cc/pppppa cc/paper cctpaper_fileee.. fille// paper/2019 htee//ahh/Seea
htt.::aasee....e-aaatatt....... html.
[2] A. Agrawal, S. Barratt, S. Boyd, and W. M. Moursi. Differentiating through cone program. Journal of Applied and Numerical Optimization, 1(2):107-115, Aug. 2019. doi: 10.23952/jano.
1 1.2019.2.02. URL https: / ano. biemdas com/ arrhivees9v111
[3] B. Amos and J. Z. Kolter. OptNet: Differentiable Optimization as a Layer in Neural Networks. In Proceedings of the 34th International Conference Machine Learning, pages 136-145. PMLR, July 2017. URL https: http::s:///oceeennngg/ mlr, www..presssvoo h0t00//0//////// amwsaao.s..... html. ISSN: 2640-3498.
[4] A. N. Angelopoulos and S. Bates, Conformal Prediction: A Gentle Introduction. Foundations and Trends@ in Machine Learning, 16(4):494 591, 2023. ISSN 1935-8237. doi: 10.1561/ 2200000101. URL http: / //tx doi. org/ 10. 2225002200000001..
[5] A. N. Angelopoulos, S. Bates, E. Candès, M. Jordan, and L. Lei. Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control. Technical report, Apr. 2022. URL http://arxiv.ot http://awww...rrrr/////////// 01052. arX0v:2110001002 [cs, stat] type: article.
[6] A. N. Angelopoulos, S. Bates, Fisch, L. Lei, and Schuster. Conformal Risk Control, Apr. 2023. http:/// org/ abs 2200. 02814. arXiv:2208.025 028814 [cs, math, stat].
[7] Artzner, F. Delbaen, J. Eber, and D. Heath. Coherent Measures of Risk. Mathematical Finance, 93):2033-228, July 1999. ISSN 0960-1621 1467-9965. doi: 10.1111/1467-9965.000 0000600. URL wiley. www.coocooccccecccccccco.c...oot /tt///.oo111001/ 111.:.....1111111111111111111111 09665.00063
[8] Barratt. On the Differentiabilitty of the Solution to Convex Optimization Problems, Nov.
2019. URL http:://axxii/ org/ abs//804 05098. arXiv:1804.05098 [math].
[9] S. Bates, A. Angelopoulos, L. Lei, J. Malik, and M. Jordan. Distribution-fee free, Risk-controlling Prediction Sets. Journal of the ACM, 68(6):33:1-43:33 Sept. 2021. ISSN 0004-5411. doi: 10.1145/3478535. URL https: http:///oo.or///..1/1 110./1175//3833
[10] A. Ben-Tal and M. Teboulle. Expected Utility, Penalty Functions, and Duality in Stochastic Nonlinear Programming. Management Science, 32(1 14:445-644,, Nov. 1986. ISSN 0025- 1909, 1526-5501. doi: 10.1287/mnsc.33 32.11.1445, URL htttts://pubsonee ine informs.
org/doi/11 /00 1287/ 2037/mmsc... 32. 11. 1445.
[11] A. Ben-Te Tal and M. Teboulle. An Old-New Concept of Convex Risk Mea- sures: The Optimized Certainty Equivalent. Mathematical Finance, 17(3):449 2007. ISSN 1467-9965. doi: 10. 111111/j.1677999989 011.9999..00070011.....
2007
URL https:// _eprint:
onlinelibrary. wiley. com/doi/abs /10. 1111/jijjj////
[12] Bernal, Sánchez, and Vilariño. Towards automatic polyp detection with a polyp appearance model. Pattern Recognition, 45(9):316631882, Sept. 2012. ISSN 0031-3203. doi: 10. 10. pato://..atcoo.201111 atticle/ppi/$00333001100..11
URL. https: httpps:/www... http:::://ww.scienneiieeeee sciencedirect. wttpiieeetttctteeeeeeeeeeeeeeeee cww/coo////ooo/o ence/

---

[13] J. Bernal, F. J. Sánchez, G. Fernández- Esparrach. D. Gil, C. Rodríguez, and F. Vilariño. WM- DOVA maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized Medical Imaging and Graphics, 43:99-111, July 2015. ISSN 0895-6111. doi: 10.1016/j.com compmediaa 2001..00.000 UtRlhttps::/wwe http://wwwwcceeeedreeetttt com/sciene hennsscinne///e/ /t//s/iiineaarricllccc/ article/ wwteciceeppi/////eee pwtpS00956111111 1110000677.
[14] D. P. Bertsekas, editor. Convex Optimization Theory. Athena Scientific, Belmont, Mass, 2009. ISBN 978-1-886529-31-1
[15] D. Bessis and F. Clarke. Partial subdifferentials, derivates and Rademacher's Theorem. Trans- actions of the American Mathematical Sociery, 351(7):2899-29)) -292,, 1999. ISSN 0002-9947, 1088-6850. doi: 10.1090/S0002-9944 99979999-9229------ URL https:/ //ww. org/ ///raa//////// 0000--51-077/000--000000
[16] C. Y.-C. Chen, J. Shen, Z. Deng, and L. Lei. Conformal Tail Risk Control for Large Language Model Alignment, Feb. 2025. URL http:///axiia org/abs/ aabbs/500.2222 arXiv:2502.20285 [cs].
[17] A. H. Correia, F. V. Massoli, C. Louizos, and A. Behboodi. An Information Theoretic Perspective on Conformal Prediction. Advances in Neural Information Processing Systems, 37:10100M 101041, Dec. 2024. URL https: ptatee://proceedingg neurips cc cctpaper_fflllee files/ haw.///ppr/////// 2024
Certtt-ooneeeeeeeeee html.
[18] S. Cortes Grntes C. M. Patiño, Y. Byun, S. Wu, E. Horvitz, and B. Wilder. Utility-Directed Conformal Prediction: A Decision Aware Framework for Actionable Uncertainty Quantification. Oct. 2024. URL https: http:///pppeevviieee
met/forumm/ i 10Mmn 1hSB0.
[19] P. L. Donti, B. Amos, and J. Z. Kolter. Task-based End-to-end Model Learning in Stochastic Optimization. In Advances in Neural Information Processing Systems, volume 30, Long Beach, CA, USA, Dec. 2017. Curran Associates, Inc. URL httLhttp//ppeeee mips cc/paper/ 7132-task- based- eored-end-to--- aoo-end-modell. momem-eeaeeennggg------ee in- htt---tooccatii---ppiiiiiiiiiiii waw-ttaaa-baaeeedeee
[20] J. C. Duchi and H. Namkoong. mance via distributionally robust optimization. The Annals of Statistics, 49 (3), June 2021. ISSN 0090-5364. doi: 10.1214/00A0S22044 URL https:
Learning models with uniform perfor-
/ //projecteuclid dearning-modelle with with-uniform www.://wmmoooee...addaaarrrrorrr 10. 10.111422--A020000 full.
[21] B.-S. Einbinder, Y. Romano, M. Sesia, and Y. Zhou. Training Uncertainty-Awar Classifiers with Conformalized Deep Learning. In Advances in Neural Information Processing Systems, New Orleans, LA, USA, Nov. 2022. doi: (0.488500ar2Xi.22205..55 org/abs/22055 05878. arXiv:2205.0587 [cs, stat].
URL hutp:://are
[22] A. N. Elmachtoub and P. Grigas. Smart "Predict, then Optimize". Management Science, 68(1):9-26, Jan. 2022. ISSN 0025-1909. pubsonline, informs. org/doi/100 1287/mmsc..... 2020. 3922.
doi: 10. 00877//msc22220.3999 Publisher: INFORMS.
URL https:// ///
[23] D.-P. Fan, G.-P. Ji, T. Zhou, G. Chen, H. Fu, J. Shen, and L. Shao. PraNet: Parallel Reverse Attention Network for Polyp Segmentation, July 2020. URL http::/ttattttae http:://xxxvvvvro or0// 11392. arXiv:2006.1 1392 [eess].
[24] S.-H. Gao, M.-M. Cheng, K. Zhao, X.-Y. Zhang, M.-H. Yang, and P. Torr. Res2Net: A New Multi-Scale Backbone Architecture. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(2):652-662, Feb. 2021. ISSN 1939-3539. doi: 10.1 00.1000/PAM.2211.2999777 URL https: /ttpttts:///eeepllorr..e ieee. org/ /document/8821313.
[25] K. He, D. Adam, S. Han-Oh, and A. Liu. Training-Awar Risk Control for Intensity Modulated Radiation Therapies Quality Assurance with Conformal Prediction, Jan. 2025. URL http: //arxiv. org/ /250 08963. mxXiv2551.00990 [cs].
[26] L. J. Hong. Estimating Quantile Sensitivities. Operations Research, 57(1): 130, Feb. 2009. 10180//00 ISSN 0030-364X. doi: 10. 1287opre. 1080.0531. URL htl.httpa:/ppaboonnlle ine. informs org/
doi/10. 128. opre. 1080. 0531.
Publisher: INFORMS.
12

---

[27] S. loffe and C. Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference Machine Learning, volume 37, pages 448-456, Lille, France, July 2015. PMLR. URI.https http:http//aae http:///axvvvv.. org/ absaaaa/.../////////////aaaaa 03167. Series Title: Proceedings of Machine Learning Research.
[28] D. Jha, P. H. Smedsrud, M. A. Riegler, P. Halvorsen, T. de Lange, D. Johansen, and H. D. Johansen. Kvasir-SEG: Segmented Polyp Dataset. Y. M. Ro, W.-H. Cheng, Kim, W.-T. Chu, P. Cui, J.-W. Choi, M.-C. Hu, and W. De Neve, editors, MultiMedia Modeling, pages 451-462, Cham, 2020. Springer International Publishing. ISBN 978-3-030-377342. doi:
[29] Z. Jiang, A. Liu, and B. V. Durme. Conformal Linguistic Calibration: Trading-off be- tween Factuality and Specificity, Feb. 2025. URL http://arxis arggabs/2502. 19110. arXiv:2502.199 19110 [es].
[30] D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations, San Diego, CA, USA, May 2015. URL http: //axxiv. org/ hatg//ba////// aa/aa/b//11// 141226980..
[31] S. Kiyani, G. Pappas, A. Roth, and H. Hassani. Decision Theoretic Foundations for Conformal Prediction: Optimal Uncertainty Quantification for Risk-Averse Agents, Feb. 2025. URL http: //arxiv..vvv/////////a org/abs/ 22t./b2b/500.....// 02561.. arXiv:2502.02561 [es].
[32] Krokhmal, T. Uryasev, and J. Palmquist. ditional value-at-risk objective and constraints. 43-68, Mar. 2001. ISSN 14651211. doi: 10.21314/0R.22022.557. URL http://www
Portfolio optimization with con- The Journal of Risk, 4(2):
http///ww..skk..o/////../../...t risk cot.ioo-opmmiiizaiioooo
[33] Lee, S. Park, and J. Shin. Learning Bounds for Risk-sensitive Learning. In Ad-
vances in Neural Information Processing Systems, volume 33, pages 13867-13879. Cur-
www.nnuurp.... ptt.rpppcccppppp///00////////// ran Associates, Inc., 2020. URL Abstraat..... hhtml.
9f60ab2b55468f hat.r//66..00000bbb6ooee666666e1
[34] J. Lekeufack, A. N. Angelopoulos, A. Bajesy, M. I. Jordan, and Malik. Conformal Decision Theory: Safe Autonomous Decisions from Imperfect Predictions, May 2024. URL http: 05921. arXiv:2310.05921.
//arxiv. org/
[35] L. Leqi,, A. Huang, Z. Lipton, and K. Azizzadenesheli. Supervised Learning with General Risk Functionals. In Proceedings of the 39th International Conference on Machine Learning, pages 12570- 12592. PMLR, June 2022. URL https: https://proceedings. mlr press/v162/ 1eqi22a. html. ISSN: 2640-3498.
[36] A. N. Madavan, N. Dahlin, S. Bose, and I.. Tong. Risk-Based Hosting Capacity Analysis in Distribution Systems. IEEE Transactions Power Systems, 39(1):355-365 Jan. 2024. ISSN 1558-0679. doi: 10. 109/TPWRS 2023.3238848 URL https: : //teeexppelreeeee/e www.or.reeeeeeeeeee.ee.e.eee.... org/ focument/1004056: Conference Name: IEEE Transactions on Power Systems.
[37] J. Mandi, J. Kotary, S. Berden, M. Mulamba, V. Bucarey, T. Guns, and F. Fioretto. Decision- Focused Learning: Foundations, State of the Art, Benchmark and Future Opportunities. Journal of Artificial Intelligence Research, 80:1623-1701, Aug. 2024. ISSN 1076-9757 doi: 10.1613/ jair. 1.5320. URL https::/twwttt jair org/ hts///idxxxx..// phpppppppaa////pp /utpee/.ppp//aaarrrrrrt/eeee///s
[38] M. Ndrio, A. N. Madavan, and S. Bose. Pricing Conditional Value at Risk-Sensitive Economic Dispatch. In 2021 IEEE Power Energy Society General Meeting (PESGM), pages 01-05, Washington, DC, USA, July 2021. IEFE, ISBN 978-1-66540--07-- doi: 10. 10.1109/ESGM44689.220 266774474... URL. URL ieee org/document/ 9637845/.
[39] S. Noorani, O. Romero, N. D. Fabbro, H. Hassani, and J. Pappas. Conformal Risk Mini- mization with Variance Reduction, Feb. 2025. htt: //axxiv.. http://aaxxvv..r//bbea///e/a 01696. arXiv:2411.00 1.01699 [cs].
13

---

[40] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, Desmaison, Köpf. E. Yang. Vitio, M. Raison, Tejani, S. Chilamkurthy, B. Steiner, L.. Fang, J. Bai, and S. Chintala. PyTorch: an imperative style, high-performance deep learning library. Proceedings of the 33rd International Conference on Neural Information Processing Systems, number 721, pages 8026-8037. Curran Associates Inc., Red Hook, NY, USA, Dec. 2019.
[41] R. Rockafellar and S. Uryasev. Journal of Risk, 2(3):21-41, 2000. ISSN 14651211. doi: 10.21314/J0R.0000 038. URL http: / http://www..kkkk. risk. joet/jorrallof-ooo httpa//d---itteeemmemaaaaap////p optimization-contee tianal-vall--ereee..
Optimization of conditional value-at-risk. The
[42] R. T. Rockafellar and S. Uryasev. Conditional value-at-risk for general loss distributions.
Journal of Banking & Finance, 26(7):1443-1471, July 2002. ISSN 0378-4266. doi: 10.1016/ S0378 18678-426600210021--6 URL https://wwwt http://wwwwccieeeieeeee...t wwt.cooo//////// htt.e//cc/cceeece//etiillleee
p11//0377642662002716
[43] O. Ronneberger, P. Fischer, and T. Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. In N. Navab, Hornegger, W. M. Wells, and A. F. Frangi, editors, Medical Image Computing and Computer-Assisted Intervention MICCAI 2015, Lecture Notes in Computer Science, pages 234-241, Cham, 2015. Springer International Publishing. ISBN
978-3-319-24574-4 doi: 10. 10.1107/77--333199
[44] U. Sadana, A. Chenreddy, E. Delage, A. Forel, E. Frejinger, and T. Vidal. A survey of contextual optimization methods for decision-making under uncertainty. European Journal of Operational Research, Mar. 2024. ISSN 0377-2217. doi:
URL
https://www, http://wwww.cincnceeedeec sciencedirecte wtt.:/ccc/cccccceeeeeee.eeeeaeee http:///ciiieeeaa.a.aa//////////
[45] G. Shafer and V. Vovk. A Tutorial on Conformal Prediction. Journal of Machine Learning Research, 912):371-42) 2008. URL http: httb:://mmmzz http://irooorrooo....////o. wwwggppaprs/vve. shaf er08a. html.
[46] Silva, Histace, Romain, X. Dray, and B. Granado. Toward embedded detection of polyps in WCE images for early diagnosis of colorectal cancer. International Journal of Computer Assisted Radiology and Surgery, 92): 293, Mar. 2014. ISSN 1861-6429, doi: htRps:/ //doi org/10 10. 1007 / 11111-1111-99290
[47] D. Stutz, Krishnamurthy, Dvijotham, A. T. Cemgil, and A. Doucet. Learning Optimal Conformal Classifiers, May 2022. URL /wwivv.xvvvv/////////a/////aa org /210 09192. arXiv:2110.09192 [cs, stat].
[48] Teneggi, M. Tivnan, W. Stayman, and J. Sulam. How Trust Your Diffusion Model: A Convex Optimization Approach to Conformal Risk Control. In Proceedings of the 40th International Conference on Machine Learning, pages 11940-33960) PMLR. July 2023. URL https: httpp://proceedingg mlr https///2222tteeegge2e html. ISSN: 2640-3498.
[49] V. Vovk, A. Gammerman, and G. Shafer. Algorithmic Learning in a Random World. Springer- Verlag, New York, 1st edition, 2005. ISBN 978-0-387-00152--, doi: 10.1007//106715. URL springer... www/co...////// 10110007//bb /b007111
[50] W. Xu, X. Gao, and X. He. Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalents. In Proceedings of the 40th International Conference on
Machine Learning, pages 38400-38427. PMLR, July 2023. URL httttt:/ppocceeddigg
mlr. ppwss/v202/xu234.. html. ISSN: 2640-3498.
[51] C. Yeh, N. Christianson, A. Wu, A. Wierman, and Y. Yue. End-to-End Conformal Calibration for Optimization Under Uncertainty, Sept. 2024. URL http: argg/bs//2e /20/ /t////2422.2222 20534. 2rXiv:2409.20534 [cs, math].
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction are supported by formal theorems in the main paper, with proofs in the appendix.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the Conclusion (Section 6), we discuss the limitations of the two methods (conformal OCE risk control and conformal risk training) introduced in our work. Due to space constraints, we discuss empirical computational efficiency in Appendix D. and we do not believe that our work imposes any significant computational efficiency burdens.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All proofs are given in Appendix C. For Theorem 3 where we only provide an informal statement Section 4, the full set of assumptions listed Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In the main paper, we provide pseudocode for our main algorithms and we list the datasets we use. Our code made publicly available on GitHub (see Section 5 for the URL), and we provide experimental details in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
16

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code to reproduce our experiments is available on GitHub (see Section 5 for the URL).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
/gticc/gg///////eee gwwiee//odeSubbleee "CodeSummissionP oisiiniccyy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips. wttpccccppccppppttcccccccccccttt wttp///ww/eeeeeeeeeeeeeeeeeeeees CooeeSumm (owoeeSamissioooooo...oo ssisss ssionPolicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
All experiment details are given in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All of our figures and tables report results from 10 random seeds. Our figures use standard boxplots to help readers interpret the data, whereas our tables report mean + stddev. All sources of randomess are documented in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources used to conduct experiments are detailed in Ap- pendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics and have followed its guidance the best of our efforts.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
18

---

We discuss how our work can help control risks in high-stakes applications such as for medical diagnoses. We do not anticipate significant negative societal impacts.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any new datasets or model checkpoints.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All of the pre-existing datasets we use are properly cited, and all of our code uses standard open- Python software and packages.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
19

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcoee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code is released on GitHub with permissive license. Our paper does not introduce any new data or model checkpoints.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our experiments do not involve crowdsourcing or research with human sub- jects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
20

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This research does not involve LLMs any important, original, or nonstandard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy (https: petttss//neurree http///ei....//eee / for what should or should not be described.
21

---

input
post-hoc CRC
eross-entropy
conformal risk training
0.01
post-hoc CRC
cross-entropy
conformal risk training
(b) a = 0.05
post-hoc CRC
cross-entropy
conformal risk training
(c) a 0.1
Figure 3: Predictions randomly selected images from the test for the tumor image segmen- tation problem (Section 5.1). Black and white pixels indicate correct outputs. False positives are shaded teal; false negatives are shaded red.
A Additional experimental results
Tumor image segmentation. Figure shows a random selection images from the test set, along with the corresponding predictions made by the baseline methods ("post-hoc CRC" and "cross- entropy") compared to our "conformal risk training" method. Evidently, our method significantly reduces false positives (shown in teal) without significantly increasing false negatives (shown in red), especially when the false negative rate controlled at a very low a. Note that the marginal risk control guarantee offered by the CRC procedure (Algorithm 3) ensures that false negative rate will be
22

---

Table Comparison of cross entsopy loss, false negative rate (FNR), and false positive rate (FPR) across baseline methods ("post-hoc CRC" and "cross-entropy" and our conformal risk training (CRT) method. The a values in the different columns indicate the level which FNR controlled by CRC. The value in parenthesis following "CRT" indicates the FNR threshold enforced during conformal risk training.
cross-entropy
0.01
0.05
0.1
loss
FNR
FPR
FNR
FPR
FNR
FPR
post-hoc CRC
0.1
0,009 +: 0.003
0.811 = 0.011
0.018
0.008
0.521 0.022
0.101
0.019
0.256 = 0.085
cross entropy
0.003
0.049
0.050
0.014
0.106
0.101
0.017
0.092
CRT (o- .01)
11.9 00++00 0.4
0.008 + 0.005
0.465
0.057
CRT (o- .00
5.8 4:4::2. 2.5
0.049 0.014
0.385 + 0.054
CRT
0.106 0.021
0.152 0.01
0.95
1.0
0.6
model
post- conformal control
0.4
fine-tune task loss conforma Iooormal risk training
0.0
10
10
Figure 4: Values of the decision scaling factor À for the battery storage problem (Section 5.2) across different CVaR quantile levels 8 and risk control thresholds a, with 10 random seeds. Higher values indicate more aggressiveness and larger charge/discharge decisions.
controlled at the target level on average over test examples, while individual examples might exceed the target false negative rate. This is why in certain cases, our method ("conformal risk training") might exceed the false negative rate of other methods (e.g.. in the 7th column of the a = 0.05 case in Figure 3), while on average, guarantees the target false negative rate.
We would also like to emphasize the decision-focused nature of our method. As shown in Figure 1, further training the pre- pre-trained image segmentation model with the PraNet weighted cross-entropy loss [23] and then applying post-hoc CRC (shown in orange) does not improve model performance directly applying post-hoc CRC the pre-trained model (shown in blue). This observation is verified in Table 2, where we see no meaningful difference in performance between the "post- hoc CRC" and "cross-entropy" rows. In contrast, at given false negative rate (FNR) risk level o, our conformal risk training method achieves lower false positive rate (FPR) but incurs higher cross-entropy loss.
Battery storage operations. In Figure 4, we show values of the decision scaling factor À obtained through our conformal risk training approach compared with the baselines described Section 5.2. In general across values of the CVaR quantile level 6 and risk control threshold it appears that our conformal risk training method yields values À that are larger on average than the alternative methods. This implies that method is able to learn to deploy more aggressive decisions, and employs a greater portion of the battery capacity when charging/discharing than the alternative methods. In other words, by learning to use larger scaling factor À while preserving the risk constraint, the electricity price forecasting model trained via conformal risk training becomes more "calibrated" in regard to risk. In addition, as the value of 5 increases, the scaling factor X decreases across all methods, reflecting the increased need to make more conservative decisions in settings where losses on the extreme tail must be controlled.
Conformal CVaR control performs better when the calibration set is larger. As shown Figure 5, a larger calibration set generally reduces the task loss and achieves tighter CVaR bound closer to (i.c., not being overly conservative). The reason a larger calibration set size N helps is because the term N B₂(A) appears the expression for (A). Increasing N reduces the effect of the upper bound B and enables choosing larger (less conservative) X. However, there are diminishing gains as N increases.
23

---

100
300
0.9
0.95
0.99
-10
1366 lost
calibration
100
-30
613
100
1.0
1.0
Figure 5: Comparison financial tail risk (top) and task loss (bottom) as a function calibration set size on the battery storage problem (Section 5.2), across different CVaR quantile levels 5 and risk control thresholds a, with 10 random seeds. Here, post-hoc conformal CVaR control is applied to the pre-trained prediction model ya.
Table Sensitivity of task loss and tail risk to absolute changes in hyperparameter on the battery storage problem described in Section 5.2. Values given show mean + stddev of the task loss [xx,y)>D[[(Y, Xáo (X)) and the financial tail risk CVaR (X,) [M (X) 20th (X)) for 10 models to trained with different random seeds. is calculated as f tp + At, where to denotes the optimal value tuned on the training set. the case that to + At E (Amin)), a). no values are given, indicated by "_".
0 ==2
a =
a=
Ar
0.95
0.99
0.9
0.95
0.99
0.9
0.95
0.99
6.5 0.9
333
-3
E
-86 -88
433
-18
7.
10.77
4.5
35.66.3
21.5
9.0 2.6
-6.8
19
4.4 1.4+11
-1.0+007
+..9
-11.3 113++30
4.5 0.8
35.4 35.4 63
222 1.5
93 2.1
0.0 0.0
0.0
0.0.1..0
1..4
-.9
0.0 0.0
1.2
-2.9
4.4 0.3
CVait [2]
0.1
3.0
6.4
0
1.3
166 0.3
1.3022
3.20..8
1.0008
33106
70
81 1.5
6.111 1.
2
0.1 0.1
03:00.1
0.0
34
4.4+0.6
2.4
69+21
86- 13
6.1
0.0
0.0
0.0
0.1
0.7
0.0
6.2
8.4
3.4
0.0:0000
0.0
0.8.03
0.0
0.0:2:00
202
01101
01201
41113
In Table 3, we compute the task loss and CVaR financial tail risk on the test set under perturbations of the hyperparameter from the tuned value lg. Because Theorem 2 only holds when L [B(min)), a)), we exclude reporting results when the perturbed / is outside of the required interval. We find that while our la tuned based on the training set not always optimal for the test set, generally close to the optimal.

---

B Conformal Risk Control Algorithm
Algorithm 3 Conformal Risk Control (CRC) for left-continuous, nondecreasing losses
function CRC(parameter space A = Amin Amax], loss functions {L4 : A + +R upper-bound
B A R, risk threshold IR, numerical tolerance > 0)
f Amiu. +- Amax
while
do
À - A + 2
if N+1 (B()) + E L(())) s then A À else X t-
return À
C Deferred proofs
Throughout the proofs, we sometimes rely on the property that a disutility function ø R - R U {+oo] corresponding
OCE risk measure (Definition 1) is closed function. For completeness, we provide the definition of closed function below.
Definition ([14)). A function f IR - R {+oo} is closed if its epigraph epi {(x, t) E IR f(x) < t) is closed set in R?.
C.1 Proof of Proposition 1
Define the
N+1
À = (ΛEA) ((A)sa)
s.), where R(A) N +11-
Σ I,(A).
1=1
and recall the definition of the set from (2), which we state again below:
N = (AEA) A(A) S aat, where h(A) N B(A) Σ
ILO)
Since L; < almost surely for every E [N], Å S almost surely. Therefore, h(A) S a implies
R(A) S almost surely, so A S A' almost surely.
Define Ā = sup Å and X' == :usp Á'. We follow the convention that sup 0 = -c0. Since C À' almost surely, we have X almost surely.
We now consider cases based whether Á' is empty
1. Suppose A' = 0. Then Å = almost surely, so it is (vacuously) true that for all E A, E[ -
If Assumption 3 holds, then = X = -oo almost surely, so Å = max maxx-Amin A} Amin almost surely. Therefore, [E[ (Å)] a.
2. Suppose Á' is non-empty. In this case, X max A' because (1) (L. are left-continuous
functions, so /t is also left-continuous; (2) أ' contains at least one point; and (3) we assume that A is closed.
Let E be the random multiset of loss functions {Lı,..., LN+1), and observe that X a constant conditional E. The random functions Li exchangeable, which implies that
Tx41(X) E
Uniform (T'( ... [x)1(X')}
Therefore,
N+1 Lx(X)= R(()).
E [NN+(())
Σ
N
25

---

Note that R is random function, because the L 's are random. By the law of total expectation (i.c., by further taking an expectation over the random multiset E), have
[ (X) = [R(X)
Sa
where the expectation is over randomness from E,, and the inequality comes from the definitions of M and X. Since LN+1 nondecreasing almost surely and S X almost surely, we have
I [ S E (X)+ S a.
Since X sup Å and LN+1 nondecreasing almost surely.
A E|LN+1 (A)) s ([[L=-1(1) S a.
If Assumption 3 holds, then observe that both E[ (A)] a and E[ (Amin)] S a. Therefore, B[ a.
C.2 Proof of Theorem
Lemma 1. ff IR + RR-BUUU [++ is disutility function corresponding to optimized certainty equivalent risk (Definition 1), then & left-continuuss.
Furthermore, if only takes on real values, then 6 is continuous.
Proof of Lemma 1. From the definition of an optimized certainty equivalent risk (Definition 1), R- + R u } is closed, convex, nondecreasing function. By [14, Proposition 1.1.2]. closedness implies é is lower-scmicontinouu.
Next, we show that since @ is lower-semicontinuous and nondecreasing, it is left-continuous. Fix any To R. Since ø monotone, it has left-limit at np, which we shall call L:
= lim d(z).
Since d is nondecreasing, L < ((m)) Choose any increasing sequence {an R]LeN that converges to Tg. Then by definition of left-limit, we have
lim p((n) + n
Lower semicontinuity at 2o yields
@ (2)) I lim ((x) I lim inf ((x,)
= lim )(x,) L. n n
2-2
n n0000
We have thus shown < ((za) < L, so d(xv) L, is left-continuous To. Since 2o arbitrary, Φ is left-contimuous on all of R.
For the case where ø is real-valued (i.e., is never infinite), then it is well-known fact that convex real-valued functions are continuous.
Proof of Theorem 1. Fix any R. By Lemma is left-continuous. Since (ø, N+ B) are nondecreasing left-continuous functions, and the composition of nondecreasing left-continuous functions remains nondecreasing left-continuous, (LL,,), 1 B,) are nondecreasing left-continuous N+1
functions. Furthermore, we have B,(A) Z Lio(A) almost surely.
For any Ā E Å. we have
R[IN+1 inf E i + ((I 2)] E teR
((IN+1(A) - =E([N+1,((11 a.
where the last inequality comes from Proposition 1.
26

---

C.3 Proof of Theorem 2
Theorem 2 states that CVaR risk can be controlled when the loss functions L, are monotone, as opposed to the stronger nondecreasing requirement from Proposition 1. In this section, we prove the more general result (Proposition 2) that when L are monotone, any OCE risk measure whose disutility function can be written as the composition of another OCE risk measure's 's disutility bass and [+]+ can also be controlled. We then show that Theorem 2 follows as corollary of this more general result.
Lemma 2. If R R TR {+00} is a closed function (Definition 3), and g R - TR is continuous, then f o g is closed.
Proof of Lemma 2. Let (In, (n) be a sequence of points in epi(fog) such that (Zn, + (2, By definition of epigraph, (((() S ln. so (g(#n), (n) E epi f. Since is continuous, ((Ta) - g(x))
and thus (g(2n), fn) - (g(r), Since If is closed, epi f is closed, so (g(x), E epi f. In other words, f(g(z)) I t, so (z, E epi(f g).
Thus, we have shown that every convergent sequence of points in epi(f o g) converges within epi(ʃ o g), so epi(J o g) is closed. Therefore, f a g is closed.
Proposition 2. Suppose that Assumptions I and 5 hold. Fix any a E R. and let Øbase R I U [+00} be a disutility function for some OCE risk measure. Let R be the OCE risk measure corresponding to the transformed disutility function Φ(x) == Craso([z]
and define Ån as in Theorem using this transformed disutility function db. Then, for every 1 E (Xmmi)), a) and every
E AG we have the risk control bound R[LN+1(A). S a.
Proof of Proposition 2. First, we show that ø a valid disutility function. By definition of a disutility function, Obase is convex, closed, and nondecreasing. Φ is convex because it is the composition of convex, nondecreasing function &base and a convex function [:]+- ø nondecreasing because it is the composition of two nondecreasing functions. By Lemma 2, ó is closed because it is the composition of
a closed function bbase and a continuous function [. Furthermore, ((0) @base (00]-) = 0. Finally,
to show that 1 E (e) we consider two cases. If x < 0, then Φ(x)- - (0))- = ((0) (0) = 0 >>>>> x. If
x 2 O, then (xxx) - (00) = obase (a) 2 x since E aehase (0). Thus, for all x E R. d(x) (00) > x. which shows that 1 E (0). Therefore, Φ is valid disutility function.
Fix any 2 B(Amin). Following Theorem 1, we define
(a,(A)= = + ((L, - -t) = : B=
EE [N+1
((B(A) t).
We will establish that Li is nondecreasing and left-continuous. By Assumption 5, L, is monotone in X. If L; is nondecreasing in X, then clearly [A+(A) +]+ is nondecreasing in À as well. If L; is decreasing in X, observe that (x(Ainn)) S B(Amin) I t almost surely, so [A) - +]+ = 0 almost surely, for all X E A. Therefore, [L. - f|+ is almost surely nondecreasing in A. By Assumption 1, L; is left-continuous, so [A) t]+ is likewise left-continuous in X. &base is also nondecreasing (by definition of disutility function) and left-continuous (by Lemma 1). The composition of two nondecreasing left-continuous functions remains nondecreasing and left-continuous, so Φi (A)(A)
t) = @base (A) - + is nondecreasing and left-continuous in X. Therefore, Lip is nondecreasing and left-continuous.
By a similar argument, B, is also nondecreasing and left-continuous. Furthermore, since B(A) 2 (A) almost surely, and Φ is nondecreasing, we have B,(A) 2 (A) almost surely.
Thus, B, and ( (i N+1 satisfy Assumptions and 2. Therefore, by Proposition 1, for every À E A,
([L)+1(A)] inf E E + o (LN+I(A)- dER
E t + ø (LN+I(2) - -t]] E [[x+1
< a
27

---

We remark that if < a (which requires a 2 B(Amin)), then Å is almost surely nonempty. Note
that B& (Amin) = 1 S Every Li, is almost surely bounded above by Bt, so (i,i())n)) S ar almost surely. Therefore, Amin Å, almost surely.
On the other hand, if a, then A, is almost surely empty. This is because for every E A, we have
B.(A) 2 Lea(A) 2 Li (Amin)))))))))))))))))))))())( 2 t > almost surely,
so (A) N+1 (B((A) + E(((1)) > almost surely. Therefore, for > a. Proposition is almost surely vacuously true.
Proof of Theorem 2. Theorem 2 follows as a corollary of Proposition 2, where we set @hase &CvaR'. Then, #CvaR*:
((x) obasc ([a)+) ( 1 [[x]+] [2]+
&cvaR
C.4 Theorem and its Proof
We start by stating and proving several technical lemmas that will help in the proof of Theorem 3. Throughout this section, we abbreviate "almost everywhere" as "a.e."
Lemma 3. If L A - R is convex and Φ R R U {+oo} is convex and nondecreasing. then the function L : A R RUO {+oo} defined by L(),t) t + @(L(A) t) is convex in (X, t).
Proof. Let a E [0, 1]. and fix any (A1, (A2, 12) E x R. Suppose L is convex and á is convex and nondecreasing. Then,
Maa1 + (1 -1- a) a) + (1 a)(2)
ot + (1-a)t2 (L(()) - a)) (at (1 a)t2))
S at1 + - (1 (L-2))((22)) -2))
- (at1 + (1 a)t2))
ot + (l=o)\++(((((h))tt)) +111))d0l (12)) -22)
t2))
< ati + to -t) (1
al((1,41) + (22,
Lemma 4. Suppose L θ x A R is 0-1 step function in À of the form
L(0, (lg(0) X]
where g - R is continuous function whose level sets have measure zero. Then, L continuous a.e. in 0.
Proof. Fix g ({A]). Clearly, θ = U U W. Since the level sets of g have measure zero, W has measure 0.
and define the preimages ----(((-xx)))
( oo)), and W
Since g continuous, U and are open sets. For any 0 U, L(0, is constant, L is continuous on U. Likewise, L is constant and hence continuous on V.
Thus, L is continuous in everywhere except W, set of measure 0.
Lemma 5. Suppose L θ A + IR is piecewise constant, left-continuous, and nondecreasing in with the form
L(0, 2) = co) Σ ( (0) <
JEN
where (cg : + R)jR), and (g, O R)jEN are continuous functions, (c)(O)) are non- negative, and (9)(o)) is nondecreasing. Assume that the level sets of (gi)JeN have measure
28

---

Let à : R + R. be an OCE disutility function, and let t E R. Then, the function L ex -
defined as
:10,A)==+++0 (MM0))))) -)) - co(0) -11
-0(0)-113)(0)) 1(g,(0) <
JEN
contimuous a.e. in 9 piecewise-constant left-continuous nondecreasing in X, and can be written in the same form as L, i.e.,
L(0, A) (0) (,) X
JEN
where (Ei)SER, are continuous functions and (c,(0)) are nonnegative.
Proof. For all ls E N, define
& ==) = t co(0) -+ t+ Σ c((0)
Since é is nondecreasing function and (c)( are nonnegative, (c) (c))O)) is nondecreasing sequence of reals. (cz)jez, are continuous by assumption, and @ continuous by Lemma 1, so (c))JEz, continuous functions.
Observe that L is also piecewise-consen left- continonssu nondecreasing A:
Ł(,, A) = (0), A (-
(0), Ae (gx
Thus, we can write
İ(0, = 20( + Σ (0) (0) < Al.
JEN
where
(0) =10)==(())--4 VjeN. nondecreasing, (o)(0)) are nonnegative. Furthermore, (2,),Ez, inherit continuity from (c))JEz,.
(0) = (0),
Because (c, (0))
By Lemma 4, each 1 (0) < A| is continuous in 6. The product of continuous functions continuous a.e., so each term ()( 1 (0) (0) < A] is continuous a.e. in The countable sum of continuous a.e. functions remains continuous a.c., so is continuous a.e. in 0.
Now, we ready state the formal assumptions needed for Theorem 3.
Assumption 6. The functions B and ( of the form
B(A) = by +
' by [dd < A], L, A) = cco +
Σ JEN
(1)))(())_000 < A]
JEN
where
1. (Ci + R)EMM,,,, and (g5.3 θ BB)NNNNE are continuous functions;
2. (by E ER)EEN and (c)((O))E(M))eN are nonnegative;
3. (d, E <R)jEn and (O){()))EN)))))))))) for all i are nondecreasing sequences, and {dzbseN U
(O),(O))E((M)een are all unique; and
4. (gis) (EN))ENNN are differentiable a.e., and all of their level sets have measure zero. Assumption Suppose that the inner problem in (8) exhibits strong duality (e.g. g. Slater's condition holds), Let u(0) denote the optimal Lagrange multiplier arising from the dual problem to (8). Define
l, strictly increasing X
ε(0, = A N IN (0,,)),
eg is strictly decreasing in X
strictly convex in X
The following regularity conditions hold:
29

---

1. 0 É exists and is continuously differentiable in (0, X) in neighborhood around (0, (0)))
2. B U {
twice continuously differentiable in (0, X) in neighborhood around (0, (0)): 3. Φ is twice continuously differentiable neighborhoods around ((x()) - and L,(0, ((0) -
for each i E [N] and
{00 x()) + p(0)- 8 (0, X(0)) S
ahe ((0)
4.
ax
Ox²
8A
is invertible
(()) ahe ax
(0, X(0))
(, (())
With these assumptions stated, now state the full version of Theorem 3.
Theorem 3. Let Φ lR + R be an OCE disutility function. Suppose Assumptions and 2 hold and that the inner problem in (8) is feasible.
(i) Suppose Σi=1 & is strictly decreasing in X; the set {0 θ (0, a. has measure zero for all X; and {B) U {L,) are piecewise constant in X. Under the standard differentiability and regularity conditions in Assumption 6, may obtain closed-forn expression for dA (ii) Suppose Σi=1 & is strictly convex or strictly monotone X, and {B} U {Te)== are convex in x Under the standard differentiabiliyy and regularity conditions in Assumption 7, the derivative 42 d0 (0) exists and has a closed-form expression.
Proof. Setting (i). By Lemma 5, each (0, == t (f,,(O, A) t) is piecewise-constant left-continuous nondecreasing in and can be written in the form
(0, A) = (0) + Σ (0)(0) 19 (0) < A].
JEN
where (2g))iz,, continuous functions and (o))(()) piecewise-constan left-continuous nondecreasing and can be written in the form
nonnegative. Similarly, B, is
B₂(A) bo + > by
1[d < X]
JEN
where (by)jeN are nonnegative. Thus, (0, == NमI (B(A)+E>* + Lc.((.,))) is piecewise-
constant left-continuous nondecreasing in À and can be written in the form
(e, A) c0 (0) +
(0) 1 (0) < <].
JEN
The sequence of steps given by
(9) (b) (0))en sort cort__sccendin(((( beN (0)
which yields strictly increasing sequence because {di}seN U (9r, (0)) (O)))e are all unique by assumption. The nonnegative coefficients (0)) are defined appropriately in terms of (by)sez. and (GL))EE(N)EEE, and (c)" )JEz, inherit their continuity.
The functions )SEN are differentiable a.e and continuous; these properties are inherited from (dj) (en) (which are constant) and (gi.)kENNT,ee (which are differentiable a.e. and continuous by assumption).
By Lemma 5, each L,, is continuous in 8. B, is constant in 0, and therefore also continuous in O. he is a countable, weighted sum of B, and Li,r for i E [N], so it is continuous a.e. in 0.
Recall from (8) that
X(0) = arg min X
N Σ ((0,2) s.t. (, a.
Because each & is strictly decreasing in X, may equivalently write the optimization problem as maximizing subject the constraint on h, and leave the optimal solution X(6) unaffected:
A(0) max À s.t. (0, A) <
30

---

By assumption, the optimization problem is feasible and (b, # f-almost everywhere. Thus, for all I except on negligible set, one of the two following cases must hold:
1. (0, A) for every X A.
Then, X(0) max A. We showed above that (0,, max A) is continuous a.c. in O. Thus, for 0 a.e., there exists neighborhood (0) around @ for which he max A) < a for all 0' E (V().. Thus, x(0) max A for all 8' E (o))), so dA de (0) max A 0.
2.
There exists an index & E N such that
( (0) +
(b) (0) 169 (b) (0) < x(0) < <a
(h) (0) + (b) (b) (0) x()) > a.
169
In this case, because the objective seeks to maximize and (g) (1)) (0))jeN is a strictly increasing sequence, X(0) = 0
(h)
As shown above, 9i+11 is differentiable a.e. Now, suppose that 9 is differentiable at 0.
(h)
Consider any entry 8g of the parameter vector 0, and let ey denote the standard unit basis vector along coordinate q. Then the partial derivative of À with respect to 0g is
0x
X(0 + seq) X(0)
(0)
lim s+0
00,
4
X(0 + sC,)
9k+1 (0)
(h)
lim
8-0
96+1 (0 + +sey 96+1 (0)
(h)
lim
s-0
8
(h)
0gk+1
00g (0).
The third equality comes from observing that (c) jEEz and ) are continuous functions, there exists >> 0 such that for all E [0, e].
(h)
(h) (0 sey) + Σ (h) + seg) 1 (b) (0 seg) < <(0 sce)] <
k+1
0h (0 + seg) + Σ
(h) (A + seg) 119, (h) (0 sey) < X(8 + seg)l >
(h) Thus, for all E [0,e], X(0 seg) 9h+1 (0 seq).
BA 0g for all coordinates g, we have dA (0)
Since 08
00,
26 -d0 dd
(0).
Therefore, we have derived a closed-form expression for dA d
(0) almost everywhere.
Setting (ii). Since ø is convex and nondecreasing, (B,) U ( convex in À by Lemma 3. Therefore, he convex in A.
Recall from (8) that
x(0) == arg min
((, s.t. h,(0,)) S.
AEA
i=1
31

---

If E & is strictly increasing (or decreasing) in À but not strietly convex in X, observe that we may replace the objective & Σi with simply À (or -) in (8) without changing the solution A(0).
Concretely, let
A A E i= (),
if E is strictly increasing in A
((,
if l strictly decreasing in À
if Σ i=l L is strictly convex in À
Then,
x(0) - arg min x) s.t. he(0,A) aa
(1I)
Since X(0) is the solution convex optimization problem whose objective l is either strictly convex or convex and strictly monotone, the solution is unique. Every convex optimization problem can be equivalently reformulated convex conic problem [1]. Then, the implicit function theorem,
applied to the KKT equality conditions, gives sufficient conditions for de (0) to exist [3, 2, 1].
Specifically, define the Lagrangian L A x IR x > IR and the function A X
R22 by
C(,, p. 0) = E(0, A) + μ (he(0, A) a)
04 (A, 0)
g(A,M,)) 0) = 0 1| (h, A) -x)
The KKT equality conditions for primal-dual optimality of (A(0), u()) are precisely given by g(A(0), 0) O.
By the implicit function theorem [8]. if
(a) (11) satisfies strong duality (e.g., if Slater's condition holds);
(b) g(((),, (0),, 0) 0:
(c) 9 continuous differentiable in (A, p, 0) in neighborhood around (x(O), 0); and 0g
(d) 0(A.) (A(0), (0)), 0) is invertible,
then the derivative dA de
(0) exists and is given by
(0) do (0)
0g a(),
0g a0 (())),
(N())
Assumption 7 directly satisfies condition (a), and the KKT equality conditions satisfy condition (b). Condition (c) is satisfied if hr, at
ax* and ahe are continuously differentiable in (0, X) in ax neighborhood around (0, x())). Assumption 7.1 guarantees that this holds for at and Assumptions ax
7.2/7.3 guarantee that this holds for h, and 0h. Finally, note that
0x
0g a(A.u) ((O)) = Tx
8² L00 ((0)) + (1))) S² (0,
ah, (0, A())
OX
p(0) 8h, ax
(0, A()))
he A(0))
which Assumption 7.4 specifies is invertible, thus satisfying condition (d).
We now briefly remark on Theorem 3()'s use of Assumption 6.4 and its relevance in practice. Recall that Assumption 6.4 assumes Jt.j θ + IR is differentiable a.e. and its level sets have measure zero. The following two lemmas show that both of these conditions are easily satisfied gio is neural network with Lipschitz continuous activation functions and bias term in its final output layer. For instance, Example 2, (i,(()) = fo(X)), the output of a neural network so (albeit neural network with more complex U-Net [43, 23] architecture than MLP).
32

---

Lemma 6. A multi-laye perceptron (MLP) with Lipschitz continuous activation functions is locally Lipschitz continuous and therefore differentiable a.e. with respect to its parameters.
Specifically, let gK : + R denote a Klaayer MLP for some K E N, defined recursively as
(x) = gu (0(x))))== Ow 96-1(01 + bk)
Vk E [K]
9o E R² is the fixed input to the MLP.
Here, (Wk, bx) are the weight matrix and bias vector in each layer k, and we use the shorthand notation 01:k (Wi:k. 1:k: br.k), and = O1.K. If each o& is Lipschitz continuous activation function, then 9K is locally Lipschitz continuous and therefore differentiable a.e. in
Proof. We use induction on k. For the base case h = 0, go is constant and therefore (locally) Lipschitz continuous.
Now, fix some and suppose Jk-1 locally Lipschitz continuous neighborhood (\()) around with local Lipschitz constant Lk-1:
VO' (NO): 119)= 9h- 9h-1(0)]l - Lu 110 Ol1
Define n = ( 110'11 < do and πK-1 == (0)1 La-1 (ia so that 100-1(00)1 & -k-1 1 for all 0' E N()). Let Mr be the Lipschitz constant of og- Then, for all 8::
11 (0)l
-Wo 16') + bk)) - oR(We (k- + b&)l
S M& IIW 1 + b/ -(W 9k-1 (0) + bb)l
S Mr (1IW% 1 - W& Sk (0) + 1b/ 1x
S Mr (1IW% Ik 16') - We - (0)] + ||0' - -
We have
[ gh- (0)) WR 9
[W 98- 1 W% (0) W 9k-1(0) W 8k (0)
W Ik- (0') W/ 1k)1( + IW 1k-1(0) We gk 1
W 1 gx-1(0)] || Will 1
< ñ. -1 110' - 01| + 110' 0| Ā--1-
Putting these results together yields
+ (()) - gx (0)l S 1 1|0' - A||
Thus, 9k is locally Lipschitz in on the neighborhood N(() with Lipschitz constant Lg Me (foL-1 + -k-1
Hence, by induction, gK is locally Lipschitz in 0. Finally, by Rademacher's Theorem [15, Corollary 4.12], is differentiable a.e.
Lemma 7. Let W C RR" be an open set, and let h W - R be a continuous function. If g x + IR is defined by g(w, b) := h(w) +b, then for every c R, the c-level set of g. L. (ww b) W x IR g(w, b) c). has measure zero.
Proof. First, we establish measurability of Le. g is measurable because it is continuous, and T. = ({c)) is the preimage of the Borel measurable singleton set (c). Therefore, T.p measurable set, and its indicator function 1,, is measurable function.
33

---

For any m E N, let Am denote the Lebesgue measure on R".. Then,
/ Bu+
1n-) =
(w, b) dd,+i(w,b))
[ (w, dd1 dd)(w)
Tonelli's theorem
(1b R [lg(w,b) = dAn
\1(fc - h(w]]) dAn
(b (w, b) = = = fc h(w))
S Odhn
singleton sets have measure
C.5 Convexity of
In this section, for ease of exposition, we write h(A,t) instead of (A)).. The following proposition is used show that (12) is convex.
Proposition 3. Under the assumptions of Theorem or Theorem 2, h(A,t) is nondecreasing in and convex in t. Furthermore, if {Li) and B are convex, then h is jointly convex in (A, t).
Proof. First, show that i(A, 1) is nondecreasing in A.
In the setting of Theorem (i.e., Assumptions 1 and 2), functions (B, Lı....; nondecreasing. Since ø also nondecreasing, Li, and B₁ are also nondecreasing. Thus, is nondecreasing in A. In the setting of Theorem 2 (i.e., Assumptions and 5 with CVaR risk), we showed Ap- pendix C.3 that Li, and B, are nondecreasing. Thus, h is nondecreasing in X.
... Lx) } are all
Second, we show that ((,, 1) is convex in f. We start by showing that B₂(A) is convex in t. For any E [0,
Balt(1-a)e(A) = at + (1 (1- (1- (((()) at + (1-a)t)
= at + (1- + ((((((A) - f) (1- - -))
S at + +
-t)
S is convex
= =aB((A) + (1-
An identical argument shows that Li, is convex in t. Since the sum functions that are convex in f, is convex in f.
If ( and B are convex (in A), then by Lemma 3, (L,,) Since is their mean, it is also convex in (A,1).
N. and B, are all convex in (A, t).
D Experimental details
Computational resources hardware:
Our experiments were performed on two computers with the following
1. 2x AMD EPYC 7513 32-Core CPUs, 4x NVIDIA A100 GPUs (80GiB GPU memory each),
TiB RAM
2. Intel Core 19-1290KKS CPU, 2x NVIDIA RTX A6000 GPUs (48GiB GPU memory each), 125 GiB RAM
Our code (see supplementary ZIP file) written in Python and primarily relies on the PyTorch [40] deep learning library. We use the Adam optimizer [30].
34

---

D.1 Tumor image segmentation
Datasets We used images from 4 public datasets (CVC-ClinieDB [13], CVC-ColonDB [12], ETIS- LaribPolypDB [46], Kvasir-SEG [28]). yielding a total of 2, 188 images with per-pixel binary mask labels. Following a similar setup to [23], we used 1,450 images from the CVC-ClinieDB and Kvasir- SEG datasets to form training set; used the same training split as [23]. From the remaining 738 images, we created 10 different val (400 images) / test (338 images) splits from different random seeds.
Note that unlike [23], we did include the "CVC-300" dataset in our test set, as "CVC-300" is a duplicate of CVC-ColonDB.
Model fine-tuning We use a pretrained PraNet [23] model, which features Res2Net [24] backbone and a U-Net-like [43] decoder. During model fine-tuning, we only update the weights of the decoder; we keep the Res2Net backbone weights frozen. We tune the learning rate with grid search over (10 2, 10-3, 10-4, 10- 1) Models are trained with a batch size of 400 for up to 100 epochs with early-stopping after 10 epochs of no improvement on the val set.
We compare two different forms of fine-tuning, as shown in Figure
"cross-entropy" fine-tuning refers using the same multi-scale binary cross-entropy loss that [23] used to pretrain the PraNet model.
"conformal risk training" refers to using the gradient (9). Recall that Example 2 derives the dA(Ø) d
exact expression dg ao (X,), where j is the index of some pixel in some image i of the training minibatch. This gradient may have high variance because it is the gradient through the model's output on exactly a single pixel among an entire minibatch of images. practice we reduce the variance of the gradient estimator by averaging the model's gradient over other pixels with similar model outputs:
d1(0) d0
do fo(x.), (iS))T
Σ
M
for the M indices -= ((tx. ) whose values fo are closest to X(0). (We set M dynamically to be 0.5% of the number of positive pixels in each minibatch.) This approach is inspired by the variance reduction techniques introduced by [26, 39], which show reduced gradient estimation variance at the expense of possibly incurring some estimation bias. We leave it to future work to formally analyze bias-variance trade-off of this gradient estimator.
Computational efficiency Our conformal risk training procedure imposes negligible computational overhead compared to fine-tuning using the standard binary cross-entropy loss. For each minibatch of M images with pixels each, the only overhead is incurred by the binary search procedure from Algorithm 1. In practice, we implement this by sorting the model outputs {fo(X.)) (aee)) (el which requires (M Md) time complexity and took on average <50 milliseconds of wall-clock time per minibatch.
D.2 Battery storage operations
Dataset We use the same dataset as Donti et al. [19] in our battery storage problem. In this dataset, the target y I22 is the hourly PJM day-ahead system energy price for 2011-2016 for total of 2189 days. However, whereas Donti et al. [19] excluded any days whose electricity prices are too high (>$500/WWh), our conformal risk training method is specifically designed to handle tail-risk, so we did not exclude these "outliers." For predicting target for a given day, the inputs x E R! include the previous day's log-prices, the given day's sa'ss hourly load forecast, the previous day's hourly temperature, and several calendar-based features such as whether the given day is a weekend or a US holiday.
Unlike Donti et al. [19], we did not include the given day's 's hourly temperature as input features, as such features are unavailable in practice (although temperature forecasts may be available). To make the problem instance more challenging, we also added i..dd. J (O, v20) noise to the price targets.
35

---

We take a random 20% subset of the dataset as the test set; because the test set is selected randomly. is considered exchangeable with the rest of the dataset. For each of 10 seeds, we further use 65%/335% random split of the remaining data for training and calibration.
Decision constraints We note that we different, but equivalent, parameterization of the battery state-of-charge constraints. Donti et al. [19] use the task loss
fly, 2)
E ye zout)e + dex
x sttaee
C
+ eamp (1="11²= + =122000(²) 11²)
2
with constraints, for all t = -,...,,??
zgtat C/2,
state 2t 2t-1
state
2t + 72t
0 0 n < ct
S out < cou, 0 S 27 state SC
Instead of using a 2 stnte variable to represent state of charge, we equivalently express the task loss and constraints in terms of a variable net E R²' representing the difference of the state of charge from the starting point. Letting= = = (z =0u,, =)) denote the decision vector, we have
f(y 2) = y (2 o + Acx
|| |0=0 eamp (1="01| + ||
with constraint set
Vt E T
net =
Σ
y2
in out
2 (3 gut net) E R3T
gott
0 S x in < dm
0 out
< cou -C/2 Td ==/$$ C//2
This reparameterizztion is necessary for the decision variable satisfy "decision-scaling" requirement: if = E 2, then AA E [0, 1], A2 E 2.
Model and fine-tuning Our price forecasting model je is fully-connected multilayer perceptron (MLP) with hidden layers, each with 256 units, LeakyReLU activation, and batch normalization [27] layers. We use a batch size of 400.
We pretrain the MLP to predict prices using mean squared error (MSE) loss. During pretraining, we tune the learning rate across (10-4, 10 3.5 10-3, 3, 10 2.5 10-2 10 L0-1. and we use L2 weight deccay strength of 10-4, We pretrain for up 500 epochs with early-stopping after 20 epochs of no improvement on the val set.
During fine-tuning, we tune the learning rate with grid search over (10-2, 10-3 10-4, 10-5) and we keep the 10 10-4 L2 weight decay. We fine-tune for up to 100 epochs with early-stopping after 10 epochs of no improvement on the val set. During fine-tuning, we use 90%/10% weighted combination of the fine-tuning loss and the MSE loss.
We compare two different forms of fine-tuning, shown in Figure 2.
"task loss" refers to fine-tuning the model ğe using the decision-focusee learning task loss from [19]. As explained in Section 5.2, we define the decision for an input x as
(x)) == arg min f(o (z), 2),
=E2
and the task loss incurred is fv, &o (x)). Thus, "task loss" fine-tuning refers to using fly, (x)) the training loss function.
"conformal risk training" refers to using the gradient (9). In this battery storage problem, we
(x)) Ty while seek control the CVaR tail risk of the financial term L(0, A) X()p(X) 2guit (X))
minimizing the task loss ((0, X) = J(Y, Aeo X)). We seek to solve
min ifx,y)>Pi(YY, Adg (X))] s.t. CVaR(x.y) [A(=]) (X) - 2g (X)) YJ a. XEA
During each minibatch of training, we pick the t that allows for the largest A. In other words, we pick X(0) by solving the convex optimization problem
x(0) arg max max tEll
s.t. he A) S a, B(Amin) St a.
(12)
This problem is convex because h, is jointly convex in (A, t) by Proposition 3. The gradient
dA(0)) is then given by Theorem 3(i) with the ΦovaR' disutility function.
term dØ
36

---

Computational efficiency Our conformal risk training procedure imposes negligible computational overhead compared to fine-tuning using the task loss. For each minibatch, the main overhead incurred is from solving and differentiating through (12), which we found in practice to require on average <50 milliseconds of wall-clock time per minibatch using cvxpylayers [1].
E
Conformal risk training subsumes conformal training
In this section, we illustrate how our method, conformal risk training, includes conformal training (ConfTr) from Stutz al. [47] special case.
Consider the multi-class classification problem with classes y = .......KK. For a nonconformity score function 89 X x y + R parameterized by 0, conformal prediction [49, 45, 4] forms prediction set
Co(X;A)= {k E y so( A.) 1
The overall goal posed by Stutz al. [47] is to optimize inefficiency (i.e., minimize the average size of the conformal prediction set) subject to minimum coverage rate. The minimum coverage constraint can be expressed using the loss function L(0, A) == 1 [s , Y) > 1 1 A]. which is left-continuous and nondecreasing in X. Since [(, X) Pr(so X, Y) > - =-
Co(X; (o(x;)) A)), the
constraint (6, A)] S a equivalent to Pr(Y E Co(X; A)) 2 1-a.
To optimize inefficiency, Stutz et al. [47] proposes using cost function that is the sum of soft assignment of classes:
Σ a (1 (1-
K
(1--))-80(
so (X,
((0,)) == max 0,
T
where 7' is temperature hyperparameter. Thus, this problem an instance of the end-to-end optimal risk control problem
min E[](O,A]] s.t. E[L(0,A)] a. AE AEA
Given a dataset D = {(X,, Y.) drawn exchangeably from P, let L; == [[sa (X, Y.) >
1 1- Aj. Clearly, every is bounded by B = Following the conformal risk control procedure, and noting that smaller values of À yield lower objective values, we may pick
1
so X1 M) 1- 21 sa}
x(0) = sup AER
1+
N
+1
i=1
1
sup
A AER
1+
1 1sg $1 1 s1)
sa]
N
i=1
N
(13)
sup
AER
Σ
188 X Y/) $1 $1
{ -
N
N
sup
AER
Σ
Y2) 51 < 1)((1a))
1 ([N
where $0,() denotes the j-th smallest value of the set S = {se (X,, Y)) Ufoo). By Proposition 1, x(0) satisfies Pr(Y Co(X; x(0))) 2 a
We note that whereas Stutz al. [47] approximates dA(0) de using differentiable approximate ranking and sorting operations, the gradient can be computed exactly [26, 51, 39]. That is, if s (X,, Y2) is differentiable w.r.t. for all i 1, 1,..., N and if sg, (+(((+)))1-a))) is unique amongst the calibration scores, then by [51, Theorem 2]
dA(0)
(X (Xo Y (o)),, if 2 N+1
d8
d0
0,
otherwise
where à: [(N + 1)(1 -
a)) -)) and a [N] - [N] denotes the permutation that sorts the scores in ascending order, such that so (() Yo()) se(Xo Yo()) for all i < j.
37


Reviews:
[{"review_id": "622CZHnE2M", "reviewer": "Reviewer_UbDa", "summary": "The paper presents two primary contributions. First, it extends the Conformal Risk Control (CRC) framework beyond expected loss to a broader class of Optimized Certainty-Equivalent (OCE) risks, which includes tail-risk measures like Conditional Value-at-Risk (CVaR). This represents a  leap forward, as the OCE class includes common tail-risk measures like the CVaR. This work provides the community with a practical methodology for creating models with provable guarantees on worst-case scenarios, moving beyond simple averages. The core technical insight is demonstrating that any OCE risk can be bounded via a monotone transformation of the loss, thereby preserving the essential monotonicity property that underpins the conformal guarantee. Proposition 1 prepares the extension for OCE risks, and improves more generally previous CRC by considering a lambda dependent bound. The main result is then theorem 2 that considers CVaR among OCE, and it proposes algorithm 1 for building the set of feasible lambda (controlling the risk). It discusses briefly the feasibility of the risk control, as well as the role of t. It introduces an assumption 5 where L_i’s and B can be either increasing or decreasing. \n\nSecond, it introduces Conformal Risk Training (CRT), an end-to-end training methodology that embeds the risk control procedure differentiably into the model's fitting loop, and generalizes previous attempts in conformal training who inspired partly the paper. The stated goal is to achieve provable risk guarantees while mitigating the degradation in performance often seen with post-hoc calibration methods (say the choice of lambda to get the guarantee on the False Positive Rate implies a drop on the False Negative Rate). Consequently the an end-to-end optimization method propose an interesting way to deal with the inherent tension between risk control and model performance. The model is optimized not only for accuracy but also for its \"amenability\" to risk control, by using Lagrangian relaxation, where the constraint on the risk is added in the global criterion. It learns to produce predictions that can satisfy the risk constraint with minimal performance degradation. Theorem 3 gives conditions for that specific approach for selecting the proper risk; the formal version is given in appendix where parametric families of risk and bound are given (piece constant in lambda), in order to be able to compute the gradient.\n\nA short Empirical Validation & Key Results are given:  for  validation the CRT framework's efficacy through two real-world applications that highlight the  dual benefits of safety and performance.\n* Controlling False Negative Rate in Tumor Segmentation: CRT is used to control the False Negative Rate (FNR) at a user-specified level α. Standard post-hoc CRC also achieves this safety guarantee, by incurring a high False Positive Rate (FPR) (such as 80%). The CRT model, in contrast, successfully resolves this trade-off, by reducing the FPR by reducing significantly and  maintaining the FNR guarantee at different risk levels.\n* Controlling CVaR in Battery Storage Operation: In a financial task where an agent optimizes battery charging cycles based on price forecasts, CRT is used to control the CVaR of monetary losses. CRT can successfully bounds this tail risk at the target level and it also yields a higher average profit compared to the post-hoc CRC baseline. This result showcases its ability to manage downside risk while actively improving upside performance.", "questions": "Additional Questions to address : \n1. In addition to the questions asked in the S&W, I would to have a clearer explanation of the computation of the gradient in the example 2, and the importance of the assumption of having all values f_theta(X_i)_j  to be unique. What happens if not the case ? And If they are very close? How does it influence the quality or existence of the derivative?\n2. It is said in assumption that we can add increasing or decreasing losses: is it a generalization? Is it really useful? \n3. What about using the standard cross entropy loss in image segmentation problem,  instead of specificity minimizing specificity.\n4. Can you provide sensitivity analysis on t, and give a complete methodology for doing the estimation, taking into account all the datasets needed train, final calibration (intermediate split)… to see how much a standard training (to see how the standard split train test validation is changed).\n5. Compare with other approaches that give a guarantee on the loss (like probabilistic for instance).", "limitations": "Partly. I really appreciate the direction the paper proposed, and the technics used that have nice theoretical properties and guarantees. \nNevertheless, I think that the paper does not face with enough depth and rigor the limitations or real impact of the paper : comparison with LTT, RCPS or other risk control approach in the paper; the choice of hyperparameter t, or the importance of the quality of B, the feasiblity problem, the feasibility of the computation of the gradient, and the stability / computational aspects of the gradient. \nI am ready to increase my score if the authors provide a more rigorous, organised and deeper treatment of these points.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\nThe paper is well written and opens the way to the very promising direction of building models that offer a finite horizon controlled of the risk by design. Indeed, while standard Conformal Prediction comes into play only for evaluating or correcting for the uncertainty of a model, hence it could be an important enabler of reliable and trustworthy AI. \nThe two examples presented and discussed in the paper motivate well the development of Conformal Training by their important and the practical impact it can have. \nThe technics and proofs seem ok, and the approach for computing the gradient seems efficient in some cases, with an interesting and promising direction based on bilevel optimization, although numerous theoretical and practical problems could happen, and be discussed.\n\nWeaknesses and critical analysis \n1. Real Impact of OCE and Novelty of CVaR Control somehow Overstated.\nThe entire framework, both for OCE risk control and CRT, is built upon the foundational assumption of a monotone loss function L(lambda) in the hyperparameter lambda. While a relaxation is noted for OCE and CVaR under specific conditions, this does not resolve the core limitation for the general framework. While CVaR or other OCE may allow to consider new applications, I wonder if it is really a big extension for dealing nonmonotonous loss function. Other approach such as Learn Then Test, Angelopoulos et al, AoAS 2025  or RCPS Angelopoulos et al could be used. In addition, the control of CVaR and other distortion measures is also addressed in Chen et al, cited in the paper. While the paper does extend traditional CRC to CVaR and potential othe risks,  this contribution is rather incremental and not the only way of doing it. A better discussion on the various possibilities of controlling of CVaR would be useful to understand the impact of the contribution. As well as the ability to address the distortion risk measures. Would it be possible to claim theorem 2 for general OCE risk instead of only for CVaR, is there any reason that precludes a general formulation and adding a corollary for CVaR.  \n2. The OCE Formulation Introduces an Unanalyzed Complexity with the new hyperparameter t from the variational definition of the risk measure. t to be selected without access to the calibration data. The paper's proposed solution—tuning t on a separate held-out set—is a practical heuristic but lacks theoretical grounding, as there is no analysis of the method's sensitivity to this choice. How does a sub-optimal t affect the tightness of the risk bound and, consequently, the model's ultimate performance? This introduces a new, unanalyzed layer of complexity to the calibration process.\n\n3. As a side and related effect is the feasibility of the constraint. It seems that there is an interplay between t, the constraint alpha, and the bound used B(lambda), as we can see in theorem 2, or in theorem 3. It is a theoretically and practically important point that deserves an explicit and detailed treatment discussion in the paper. Could you discuss in a coherent way the problem of existence, and possibly how it is related to the existence of a gradient.\n\n4. The Practicality of Conformal Risk Training is Limited by Gradient Computation.\nThe CRT method's central mechanism is to make use of the implicit value theorem by considering a bilevel optimisation, and propagate through the risk-control parameter lambda(theta). The paper shows this is feasible for highly-constrained settings with Piecewise Constant Losses or strict convexity / monotony : The gradient exists because the optimal lambda(theta) snaps to a specific model output value.\nIn addition every batch must be split in a calibration and prediction, but the size and the role of the split is not detailed and it becomes a bit blurry. The complexity and subtleties of its computation are hidden, as in appendix problem of stabilities and variance reduction p. 27 line 1043 do require averaging. Could you dedicate a specific part to the existence of the gradient and the numerical computation of the gradient. The ability to compute or approximate properly the gradient narrows the the applicability of CRT.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing Conformal Risk Training as a promising method for controlling risk by design. We respond to each point raised by the reviewer.\n\n**Comparison against related methods**\n\nThe review highlights 3 related works on risk control: Learn then Test (LTT), Risk Controlling Prediction Sets (RCPS), and the Distortion Risk Control method by Chen et al. (henceforth referred to as \"Chen\"). All 3 methods provide *high-probability* bounds on risk, as opposed to the stronger *certain (marginal)* bound that our Conformal OCE Risk Control (CORC) method achieves. As shown in Appendix B of the original Conformal Risk Control (CRC) paper, the marginal bounds from conformal methods are significantly more statistically efficient compared to LTT/RCPS.\n\nWe further discuss each of these methods in order:\n- LTT: LTT is primarily used to control the expected loss of non-monotone loss functions. While the generic LTT framework in principle supports a general risk measure $R$, one must be able to compute p-values for null hypotheses of the form $H_\\lambda:\\ R(\\lambda)>\\alpha$. The Hoeffding–Bentkus inequality p-values described in the LTT paper only apply to expected losses; while a recent work (“Quantile Learn then Test”, Farzaneh et al. 2024) proposes a p-value for $R=\\text{VaR}$ (value-at-risk), to the best of our knowledge, we are unaware of p-values for $R=\\text{CVaR}$. We note that a conservative approach would be to use LTT to control the expectation of $\\tilde{L}_t(\\lambda)=t+\\phi_{\\text{CVaR}^\\delta}(L(\\lambda)-t)$, which upper-bounds $\\text{CVaR}^\\delta[L(\\lambda)]$. If the reviewer would like to see a comparison between CORC and this approach to LTT, we can try to implement this comparison.\n- RCPS: RCPS is used to control risk resulting from set-valued predictors. For our battery storage CVaR example, it is unclear to us how to formulate the problem using set-valued predictors. Furthermore, RCPS requires the loss functions to be monotone in $\\lambda$ (with monotonicity in the same direction). Thus, we do not believe RCPS to be suitable for controlling CVaR for our battery storage example, where the loss functions can be either monotone increasing or decreasing.\n- Chen: The main result from the Chen method is an *asymptotic* (as opposed to finite-sample) high-probability distortion risk control bound. The Chen paper does introduce a finite-sample variant, which is more conservative. If the reviewer would like to see a comparison between CORC and the more conservative Chen method, we can try to implement this comparison.\n\nWe will include a more thorough discussion of these three related works in the camera-ready revision.\n\n**Generalizing Theorem 2 to all OCE risk measures**\n\nTheorem 2 does not apply more generally to all OCE risk measures. For example, the expected loss is an OCE risk measure. As the original CRC paper makes clear, monotonicity of the loss function (where all loss functions are monotone in the same direction) is necessary for controlling the expected loss.\n\n**Sensitivity of hyperparameter $t$**\n\nTuning the hyperparameter $t$ to tighten the OCE risk bound is indeed important. In the Proof of Theorem 1 (Appendix C.2), $t$ shows up in two inequalities:\n\n$$\nR[L_{N+1}(\\hat\\lambda)]\n\\leq\\mathbb{E}\\left[t+\\phi(L_{N+1}(\\hat\\lambda)-t)\\right]\n=\\mathbb{E}[\\tilde{L}_{N+1,t}(\\hat\\lambda)]\n\\leq \\alpha.\n$$\n\nThe 1st inequality is tight at the value $t$ that minimizes the variational OCE expression. For example, when $R=\\text{CVaR}^\\delta$, $t=\\text{VaR}^\\delta[L(\\hat\\lambda)]$ makes the 1st inequality tight. On the other hand, theoretically analyzing the best choice of $t$ to tighten the 2nd inequality is difficult in general. We welcome suggestions, but for now, we leave this to future work.\n\nIn the 2 tables below, we show empirical sensitivity of the average task loss and empirical CVaR to the choice of $t$ for post-hoc Conformal CVaR Control in the battery storage task. Here, $\\Delta t$ is the (absolute) amount by which we change $t$ from the value tuned on the training set. Evidently, the value of $t$ tuned on the training set is not always optimal, but the resulting task loss tends to be close (within 1 stddev) of the task loss from the optimal $t$.\n\nDue to space limitations, we can only show a few choices of $\\Delta t$, but we will include a proper figure in the camera-ready.\n\nTask loss\n| Δt | α=2, δ=.9 | α=2, δ=.95 | α=2, δ=.99 | α=5, δ=.9 | α=5, δ=.95 | α=5, δ=.99 | α=10, δ=.9 | α=10, δ=.95 | α=10, δ=.99 |\n|-|-|-|-|-|-|-|-|-|-|\n| -5 | 0±0 | 0±0 | 0±0 | 0±0 | 0±0 | 0±0 | -8.4±18 | 0±0 | -2±3.2 |\n| -1 | 0±0 | 0±0 | -0.39±0.64 | 0±0 | -2.9±6.1 | -3.7±1.7 | -8.4±18 | -6.2±13 | -8.4±3 |\n| 0 | 0±0 | -1.5±2.7 | -1.8±0.51 | -4.2±13 | -3.6±6.7 | -4.5±1.3 | -13±20 | -7.3±13 | -9±2.6 |\n| 1 | 0±0 | -2.2±3 | -0.95±0.7 | -8.4±18 | -5.3±7.3 | -4.5±0.76 | -13±20 | -10±14 | -9.3±2.1 |\n| 5 | -1.9±6 | -0.69±1.5 | 0±0 | -8.4±18 | -5.1±5.9 | 0±0 | -13±20 | -13±15 | -4.8±3.5 |\n\nCVaR\n| Δt | α=2, δ=.9 | α=2, δ=.95 | α=2, δ=.99 | α=5, δ=.9 | α=5, δ=.95 | α=5, δ=.99 | α=10, δ=.9 | α=10, δ=.95 | α=10, δ=.99 |\n|-|-|-|-|-|-|-|-|-|-|\n| -5 | 0±0 | 0±0 | 0±0 | 0±0 | 0±0 | 0±0 | 1±2.3 | 0±0 | 1.3±2.1 |\n| -1 | 0±0 | 0±0 | 0.27±0.43 | 0±0 | 0.87±1.8 | 2.7±1.1 | 1±2.3 | 1.9±4 | 6.1±1.6 |\n| 0 | 0±0 | 0.46±0.81 | 1.3±0.24 | 0.39±1.2 | 1.1±2 | 3.3±0.59 | 1.5±2.5 | 2.3±4.1 | 6.7±1.2 |\n| 1 | 0±0 | 0.68±0.9 | 0.77±0.61 | 1±2.3 | 1.6±2.2 | 3.4±0.53 | 1.5±2.5 | 3.2±4.4 | 6.9±1 |\n| 5 | 0.17±0.53 | 0.31±0.68 | 0±0 | 1±2.3 | 1.7±1.9 | 0±0 | 1.5±2.5 | 4.2±4.6 | 3.8±3 |\n\nRegarding the interplay among $t$, $\\alpha$, $B$, it is true that a bad choice of $t$ may lead to $\\hat\\Lambda\\_t$ being empty. As noted in Thm 1, if $\\hat\\Lambda\\_t$ is empty (so $\\sup\\hat\\Lambda\\_t=-\\infty$), we set $\\hat\\lambda=\\lambda\\_{\\min}$.\n\n**Gradient computation**\n\nFirst, we would like to emphasize that the post-hoc Conformal OCE Risk Control method we introduce does not require any gradient computation. We acknowledge that Conformal Risk Training is only applicable when the gradient $\\frac{d\\lambda}{d\\theta}$ can be computed (or approximated), and Theorem 3 is our best attempt to obtain sufficient conditions for the existence of the gradient.\n\nRegarding the size and role of batch splitting, we recommend that the size of the batch $N$ be large enough such that $\\tilde{h}\\_t(\\lambda\\_{\\min})\\leq\\alpha$, so that the gradient $\\frac{d\\lambda}{d\\theta}$ is non-zero. If the batch size is smaller, the model will still train with gradients $\\frac{\\partial\\ell}{\\partial\\theta}(\\theta,\\lambda\\_{\\min})$, but this may result in reduced training efficiency. We will make a note of this in the camera-ready.\n\nRegarding the assumption that all values $f_\\theta(X_i)_j$ are unique in Example 2, this assumption is necessary for an *exact* gradient to exist. This is similar to how the exact gradient for a max-pooling function $g(\\mathbf{x})=\\max_i x_i$ (e.g., in convolutional neural nets) only exists when the entries of the vector $\\mathbf{x}$ are unique. When there are multiplicities, the exact gradient is technically undefined.\n\nWhen there are multiplicities, or when there are multiple model outputs with very similar values, the gradient sensitivity can be large. This is the motivation and intuition behind our choice to average the model's gradients over the $M$ pixels with the most similar output values (see Appendix D.1). This averaging reduces variance in the gradient estimation, as noted by Hong (2009) and Noorani et al. (2025). We would be happy to move this discussion to the main text in the camera-ready revision.\n\n**Standard cross entropy loss in image segmentation**\n\nOur understanding of the reviewer's question is, what if we set the training objective $\\ell(\\theta,\\lambda)$ to be the cross-entropy (CE) loss? We claim that this is equivalent to simply training the model with CE loss (without conformal risk training), and then applying post-hoc CORC.\n\nBecause CE loss does not depend on any threshold $\\lambda$, $\\ell(\\theta,\\lambda)=\\ell(\\theta)$. Thus, the risk constraint can be dropped, since picking $\\lambda=\\lambda_{\\min}$ will suffice (by Assumption 4). That is,\n\n$$\n\\min_{\\theta,\\lambda}\\mathbb{E}[\\ell(\\theta)]\\text{ s.t. }R(\\theta,\\lambda)\\leq\\alpha\n\\quad\\equiv\\quad\n\\min_\\theta\\mathbb{E}[\\ell(\\theta)].\n$$\n\nWe can make a note of this in our camera-ready revision to highlight that Conformal Risk Training generalizes empirical risk minimization (ERM) to the setting where the objective $\\ell$ depends on a risk control parameter $\\lambda$ that is constrained to satisfy a risk constraint. When $\\ell$ does not depend on $\\lambda$, then Conformal Risk Training reduces to standard ERM.\n\nAs shown in Figure 1, further training the pre-trained image segmentation model with CE loss and then applying post-hoc CRC (shown in orange) does not improve model performance over directly applying post-hoc CRC to the pre-trained model (shown in blue).\n\nTo further emphasize how Conformal Risk Training (CRT) with the specificity (FPR) objective differs from ERM with CE loss, the following table shows the average CE loss on the test set for 3 different training settings (mean±std over 10 random seeds). Notably, at a given risk level $\\alpha$, CRT achieves lower FPR but incurs higher CE loss. This highlights the decision-focused nature of CRT.\n\n| | pretrain | cross-entropy | CRT α=.01 | CRT α=.05 | CRT α=.10 |\n|-|-|-|-|-|-|\n| CE loss* | 2.8 ± .1 | 2.9 ± .2 | 11.9 ± .4 | 5.8 ± 2.5 | 7.8 ± 6.1 |\n| FNR @ α=.01 | .009 ± .003 | .008 ± .003 | .008 ± .005 |-|-|\n| FPR @ α=.01 | .841 ± .014 | .811 ± .049 | .465 ± .057 |-|-|\n| FNR @ α=.05 | .048 ± .008 | .050 ± .014 |-| .049 ± .014 |-|\n| FPR @ α=.05 | .524 ± .022 | .580 ± .106 |-| .385 ± .054 |-|\n| FNR @ α=.1 | .101 ± .019 | .101 ± .017 |-|-| .106 ± .021 |\n| FPR @ α=.1 | .256 ± .035 | .293 ± .092 |-|-| .152 ± .041 |\n\n**In this table, \"CE loss\" refers to the weighted cross-entropy loss described in the PraNet (Fan et al., 2020) paper.*"}, {"review_id": "kK5NM6Dxop", "reviewer": "Reviewer_yYUJ", "summary": "This work introduces Conformal Risk Training, an extension of the traditional Conformal Risk Control framework to a broader class of risk measures, including CVaR, entropic risk etc. Their end-to-end approach integrates risk control directly into the training process by differentiating through the risk-controlling decision rule. The authors provide theoretical guarantees for risk control by applying monotone transformations of the loss function and develop techniques for computing gradients in common bi-level optimization settings.", "questions": "1. I think a discussion on the tightness of the CORC bounds might be helpful. Especially, under what conditions might these guarantees become overly conservative?\n2. what happens when the loss functions $L_i(\\lambda)$ is not globally monotonic? For example, could one consider piecewise monotonic, quasimonotonic, or approximate monotonic losses and still retain valid risk control?\n3. While differentiability of the inner solution $\\lambda(\\theta)$ is addressed under specific conditions, can the authors formally analyze its stability with respect to perturbations in $\\theta$? \n4. Can the CORC framework be extended to non exchangeable data, such as time series settings?\n5. I believe this work has strong connections to Robust Optimization, especially with the recently evolving end to end robust optimization methods. I think a discussion along this direction can help connect it to a broader scope of works. \n6. Can the authors provide insight into when end-to-end conformal risk training yields better efficiency or generalization compared to post-hoc CRC? Are there conditions under which the added complexity of bilevel training is provably beneficial?", "limitations": "as discussed above", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1. The paper extends CRC beyond expected loss to risk measures like CVaR addressing an important open question. Introduces a novel bi-level optimization framework that differentiates through the conformal risk control procedure, enabling joint learning of model parameters and risk-sensitive thresholds.\n2. The authors provides finite sample, distribution free risk guarantees for a broad class of risks under monotonicity assumptions. They derive closed form gradient expressions under common conditions.\n3. The framework appears broadly applicable to any bounded, monotone loss, which makes it suitable for a wide range of decision making tasks beyond the two domains tested.\n4. Theoretical guarantees in the paper seem to depend on the assumption that loss functions are monotone and exchangeable. These assumptions may not hold in many practical settings, especially in tasks involving structured or high-dimensional outputs.\n4. The framework introduces an extra hyper parameter t, whose selection requires a hold out set and may impact practical performance if not tuned well.\n5.  Solving nested optimization problems (especially with CVaR constraints) can be computationally expensive or sensitive to initialization.\n6. While baseline comparisons are made, no direct comparison is made with other non conformal risk sensitive training approaches (e.g., DRO, robust optimization).\n7. The manuscript doesn't make connections with the rapidly growing literature on decision focused learning and contextual stochastic optimization. in particular, it omits discussion of the recent EJOR survey by Sadana et al. on contextual optimization and the JAIR survey by Mandi et al. on Decision Focused Learning, both of which place bilevel risk-aware training in a broader taxonomy.", "was_revised": true, "final_justification": "I am maintaining my score. The authors addressed the main concerns, and while some broader connections and extensions remain open, that does not diminish the contributions of the paper. Overall, this is a solid and timely contribution to risk aware learning with strong theoretical and empirical results.", "rebuttal": "We thank the reviewer for recognizing Conformal Risk Training as a promising method for controlling risk by design. We respond to each of the points raised by the reviewer.\n\n**Monotone assumption**\n\nThe reviewer correctly points out that in some real-world risk control settings, the monotonicity assumption does not apply, especially in tasks involving structured or high-dimensional outputs. For example, the Learn then Test (Angelopoulos et al., 2025) paper gives multiple examples of non-monotone losses and develops a method for controlling their expected loss with high probability. However, as we (and many other papers in the conformal literature) demonstrate, the monotonicity assumption nonetheless encompasses a large number of real-world loss functions.\n\nUnfortunately, relaxing the global monotonicity assumption on the loss functions $L_i$ remains challenging. As shown by the original Conformal Risk Control (CRC) paper (see \"Proof of Proposition 2\" in their Appendix E), even functions $L_i$ that are _monotone in expectation_ cannot be controlled using Conformal Risk Control methods. For a loss $L$ that is approximately monotone, the original CRC paper proposes the idea of \"monotonizing non-monotone risks\"—_i.e._ computing the tightest monotone upper-bound of $L$. This idea can be directly translated to our Conformal OCE Risk Control (CORC) method, though we do not explore this idea further.\n\n**Exchangeability assumption**\n\nWhile our work uses the standard assumption that the loss functions $L_i$ are exchangeable, this assumption can be relaxed following standard conformal prediction arguments. For example, the CRC paper demonstrates how to apply CRC under covariate shift by using importance weighting, and it is straightforward to adapt that methodology to our CORC method. We can include a discussion of this in our camera-ready paper if the reviewer would like.\n\nSpecifically for time series, split conformal methods such as what we use are generally not directly applicable. The online variants of conformal methods (e.g., adaptive conformal inference (Gibbs and Candes, 2021)) generally rely on very different proof techniques compared to the exchangeable data setting. Thus, we believe that developing an online variant of CORC would be a valuable direction of future work.\n\n**Sensitivity of hyperparameter $t$, and tightness of CORC bound**\n\nThe reviewer keenly points out the importance of tuning the hyperparameter $t$ to tighten the OCE risk bound. As seen in the Proof of Theorem 1 (Appendix C.2), $t$ shows up in two inequalities:\n\n$$\nR[L_{N+1}(\\hat\\lambda)]\n\\leq \\mathbb{E}\\left[ t + \\phi(L_{N+1}(\\hat\\lambda) - t) \\right]\n= \\mathbb{E}[\\tilde{L}_{N+1,t}(\\hat\\lambda)]\n\\leq \\alpha.\n$$\n\nThe first inequality is tight at the value $t$ that minimizes the variational OCE expression. For example, when $R = \\text{CVaR}^\\delta$, $t=\\text{VaR}^\\delta[L(\\hat\\lambda)]$ makes the first inequality tight. On the other hand, theoretically analyzing the best choice of $t$ to tighten the 2nd inequality is difficult in general. We welcome suggestions, but for now, we leave this to future work.\n\nIn the 2 tables below, we show the empirical sensitivity of the average task loss and the empirical CVaR on the choice of $t$ for post-hoc Conformal CVaR Control in the battery storage task. Here, $\\Delta t$ is the (absolute) amount by which we change $t$ from the value tuned on the training set. Evidently, the value of $t$ tuned on the training set is not always optimal, but the resulting task loss tends to be close (within 1 stddev) of the task loss from the optimal $t$.\n\nDue to space limitations, we can only show a few choices of $\\Delta t$, but we will include a proper figure in the camera-ready.\n\n_Table 1_: Task loss\n| Δt | α=2, δ=0.9 | α=2, δ=0.95 | α=2, δ=0.99 | α=5, δ=0.9 | α=5, δ=0.95 | α=5, δ=0.99 | α=10, δ=0.9 | α=10, δ=0.95 | α=10, δ=0.99 |\n|-|-|-|-|-|-|-|-|-|-|\n| -5 | 0 ± 0 | 0 ± 0 | 0 ± 0 | 0 ± 0 | 0 ± 0 | 0 ± 0 | -8.4 ± 18 | 0 ± 0 | -2 ± 3.2 |\n| -1 | 0 ± 0 | 0 ± 0 | -0.39 ± 0.64 | 0 ± 0 | -2.9 ± 6.1 | -3.7 ± 1.7 | -8.4 ± 18 | -6.2 ± 13 | -8.4 ± 3 |\n| 0 | 0 ± 0 | -1.5 ± 2.7 | -1.8 ± 0.51 | -4.2 ± 13 | -3.6 ± 6.7 | -4.5 ± 1.3 | -13 ± 20 | -7.3 ± 13 | -9 ± 2.6 |\n| 1 | 0 ± 0 | -2.2 ± 3 | -0.95 ± 0.7 | -8.4 ± 18 | -5.3 ± 7.3 | -4.5 ± 0.76 | -13 ± 20 | -10 ± 14 | -9.3 ± 2.1 |\n| 5 | -1.9 ± 6 | -0.69 ± 1.5 | 0 ± 0 | -8.4 ± 18 | -5.1 ± 5.9 | 0 ± 0 | -13 ± 20 | -13 ± 15 | -4.8 ± 3.5 |\n\n_Table 2_: CVaR\n| Δt | α=2, δ=0.9 | α=2, δ=0.95 | α=2, δ=0.99 | α=5, δ=0.9 | α=5, δ=0.95 | α=5, δ=0.99 | α=10, δ=0.9 | α=10, δ=0.95 | α=10, δ=0.99 |\n|-|-|-|-|-|-|-|-|-|-|\n| -5 | 0 ± 0 | 0 ± 0 | 0 ± 0 | 0 ± 0 | 0 ± 0 | 0 ± 0 | 1 ± 2.3 | 0 ± 0 | 1.3 ± 2.1 |\n| -1 | 0 ± 0 | 0 ± 0 | 0.27 ± 0.43 | 0 ± 0 | 0.87 ± 1.8 | 2.7 ± 1.1 | 1 ± 2.3 | 1.9 ± 4 | 6.1 ± 1.6 |\n| 0 | 0 ± 0 | 0.46 ± 0.81 | 1.3 ± 0.24 | 0.39 ± 1.2 | 1.1 ± 2 | 3.3 ± 0.59 | 1.5 ± 2.5 | 2.3 ± 4.1 | 6.7 ± 1.2 |\n| 1 | 0 ± 0 | 0.68 ± 0.9 | 0.77 ± 0.61 | 1 ± 2.3 | 1.6 ± 2.2 | 3.4 ± 0.53 | 1.5 ± 2.5 | 3.2 ± 4.4 | 6.9 ± 1 |\n| 5 | 0.17 ± 0.53 | 0.31 ± 0.68 | 0 ± 0 | 1 ± 2.3 | 1.7 ± 1.9 | 0 ± 0 | 1.5 ± 2.5 | 4.2 ± 4.6 | 3.8 ± 3 |\n\n\n**Nested optimization**\n\nWe believe the reviewer may have a misunderstanding of our method. We never directly solve any optimization problem that involves a CVaR constraint. Instead, our method replaces the CVaR constraint with the sufficient criterion from Theorem 2.\n\nFor Conformal Risk Training, while it is true that we have to calculate $\\lambda(\\theta)$ at every minibatch, we note that this computation is generally very efficient via the bisection-search method shown in Algorithm 1.\n\n**Comparisons with other risk-sensitive training approaches (e.g., DRO, robust optimization)**\n\nWe were indeed inspired by other risk-sensitive training approaches, especially the conditional robust optimization (CRO) methods by Yeh et al. (2024) and Chenreddy et al. (2024). However, our Conformal CVaR Risk Control method fundamentally solves a different problem than robust optimization. Whereas end-to-end robust optimization (RO) aims to learn the _easiest_, say, 90%-confidence uncertainty set with respect to which to make decisions, our Conformal CVaR Risk Control method aims to control risk for the _hardest_ 10% of outcomes. In other words, RO and DRO methods aim to be robust to the most likely outcomes, whereas our approach focuses on tail risks which encompass the most unlikely, and costly, outcomes. Without strong distributional assumptions on the loss function, RO and DRO methods do not provide any tail risk guarantees.\n\n**Literature connections to DFL and Contextual Stochastic Optimization**\n\nOur work is very much inspired by the broader decision-focused learning (DFL) literature. Due to the page limit in our submission, we only cited DFL methods that used conformal prediction for uncertainty quantification (e.g., Yeh et al. 2024, and He et al. 2025), and we did not have space in our submission to discuss connections with the broader DFL literature. In the camera-ready revision, we will definitely discuss and cite the Sadana and Mandi survey papers.\n\n**Stability of gradient**\n\nHere, we briefly discuss the stability of the gradient for the two cases described in Theorem 3. For case (i), where $B$ and $L_i$ are piecewise constant in $\\lambda$, the exact gradient may exhibit high variance. Thus, as we note in Appendix D.1, for the image segmentation example we average the gradient over pixels with similar model outputs. This variance reduction technique was first discussed by Hong 2009 and Noorani et al. (2024) for the simpler conformal prediction settting where the risk parameter $\\lambda$ is chosen as an empirical quantile. In contrast, for the CORC setting (whether Theorem 3 (i) or (ii)), the risk parameter $\\lambda$ is chosen as the solution to an optimization objective. Analyzing stability then requires understanding the sensitivity of the solution of the optimization problem to its parameters. Characterizing this stability is therefore challenging, and we agree this is a valuable direction for additional research.\n\n**Efficiency and generalization of Conformal Risk Training vs. post-hoc CRC**\n\nLike the original Conformal Training method, the benefit of Conformal Risk Training is largely empirical. In general, we would expect Conformal Risk Training to yield larger improvements when the Conformal Risk Training objective $\\ell(\\theta, \\lambda)$ (e.g., specificity) is substantially different from the model pretraining objective (e.g., cross-entropy loss).\n\n**Formatting concerns**\n\nWe apologize that the reviewer finds Algorithm 2 unclear. If the reviewer could please elaborate on which specific aspect of Algorithm 2 should be improved, we are happy to make changes. As for the figures, we will take the reviewer's suggestion into account when preparing camera-ready figures."}, {"review_id": "LuulUXPV31", "reviewer": "Reviewer_wLXq", "summary": "This paper is exploring post-hoc risk control, for a broader class or risk measures. They particularly generalize the CRC method for the cases beyond expected loss, like Cvar. They provide precise theoretical arguments for the validity of their method. Furthermore, they extend conformal training ideas to their risk control setup, by differentiating through post-hoc risk control. This way, they enhance the efficiency of the whole decision making system, while controlling the desired risk. Their arguments are also sufficiently supported by experiments.", "questions": "Posed earlier.", "limitations": "I think the assumptions made should be discussed, particularly with the help of some explicit examples of loss functions.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- The conjunction of ML and reliable decision making is a timely and very important research direction. This paper studies a somewhat neglected but very important topic of \"how can we use ML predictions while controlling tail sensitive downstream risks\", and the findings of the paper can contribute positively to the growing understanding of the community.\n\n- I would suggest to add some explicit examples for CRC loss function. For instance, it is not obvious how the notion of \"action\" or \"decision\" would show up in this formulation, as at the end of the day a decision maker wants to make a decision. \n- The assumption 4 (the same comment applies to assumption 3) needs more clarifications. The authors claim their contribution to be in controlling Cvar, however under assumption 4, controlling Cvar is trivial by just picking $\\lambda= \\lambda_{\\rm min}$.  Hence, perhaps the goal under assumption 4 is to pick the \"largest $\\lambda$ possible\" while controlling Cvar. This is not clearly stated, and it is not the direct subject of study in the paper (although it looks like the algorithm does try to satisfy such an objective).\n- In light of these, a more general concern emerges. Why would a 1-dimensional characterization of the decision space ($\\lambda$) makes sense at all when trying to control Cvar. Although CRC also assumes a 1-dimensional characterization, however, at least some of the use cases of CRC, like coverage of sets or FDR control, do have some fundamental 1-dimensional characteristics. For the case of FDR control neyman-pearson lemma shows the existence of a 1-dimensional rule, and for prediction sets [1, 2] explore similar results. \n\nThese being said, I believe the paper exceeds the acceptance threshold, and i am willing to increase my score, if the authors take concrete actions to resolve/clarify these concerns in the revised manuscript.\n\n[1] Least Ambiguous Set-Valued Classifiers with Bounded Error Levels\n[2] Decision Theoretic Foundations for Conformal Prediction: Optimal Uncertainty Quantification for Risk-Averse Agents", "was_revised": true, "final_justification": "I thank the authors for their response. I also have read the other reviewers comments, and I believe the paper has clear and interesting results that is of particular interest of the community. Hence, I raise my score and vote for acceptance.", "rebuttal": "We thank the reviewer for recognizing Conformal Risk Training as a promising method for controlling tail risk by design. We respond to each point raised by the reviewer.\n\n**More examples of CRC loss functions**\n\nOur paper highlights three explicit examples of CRC loss functions: false negative rate, battery storage operations loss, and miscoverage loss.\n\nIn the false negative rate loss function (see Example 1), the _decision_ refers to classifying a pixel as either tumorous (1) or benign (0), based on the model's predicted probability $f_\\theta(x)_j \\in [0,1]$ for pixel $j$. This decision is determined by the threshold $\\lambda$, and the decision loss is measured by the false negative rate $L(\\theta, \\lambda)$ given in Example 1.\n\nIn the battery storage optimization example (see Section 5.2), the _decision_ refers to the vector $z := (z^\\text{in}, z^\\text{out}, z^\\text{net}) \\in \\mathbb{R}^{72}$ which describes how much to charge or discharge a battery for each hour of the day. The decision is scaled by the parameter $\\lambda$ to reduce CVaR tail risk to the desired level.\n\nFurthermore, in Appendix E, we show how standard conformal prediction is a special case of conformal risk control that aims to control the miscoverage loss. Here, the _decision_ refers to constructing a prediction set $C_\\theta(X; \\lambda)$, whose size depends on the risk control parameter $\\lambda$.\n\nThe original Conformal Risk Control (CRC) paper (Angelopoulos et al., 2023), the Risk-Controlling Prediction Sets (RCPS) paper (Bates et al., 2021), and many other works provide various examples of monotone CRC losses, just to name a few:\n- financial risk from portfolio optimization\n- false discovery rate in multilabel classification\n- ranking loss\n- projective distance\n\nWe would be happy to provide a few more examples of these monotone CRC losses in our camera-ready revision to better motivate our work.\n\n**Clarifying assumptions 3 & 4**\n\nWe apologize for the lack of clarity around assumptions 3 & 4. The reviewer correctly points out that under these assumptions, picking $\\lambda = \\lambda_{\\min}$ always achieves OCE risk control of a random loss function $L(\\lambda)$. This is why we use the objective $\\ell(\\theta, \\lambda)$ to determine an _optimal_ $\\lambda$. Previous works such as Conformal Decision Theory paper (Lekeufack et al., 2024) only consider $\\ell$ that are strictly decreasing in $\\lambda$; in that case, they are indeed concerned with choosing the _largest_ $\\lambda$ such that risk control (of expected loss) is achieved. Our framework is more general by allowing $\\ell$ that are either monotone or convex in $\\lambda$, in which case the optimal $\\lambda$ (with respect to $\\ell$) may not necessarily be the largest.\n\nWe will revise our paper to clarify this point.\n\n\n**1-d characterization of decision space**\n\nAs the reviewer points out, the original CRC formulation also assumes a 1-d characterization. Set coverage, FDR control, FNR control, etc. all have fundamental 1-d characteristics.\n\nFor controlling CVaR in the battery storage problem, we are motivated by the \"decision-scaling\" rule: $z\\_\\theta(x, \\lambda) = \\lambda \\cdot \\hat{z}\\_\\theta(x)$. In this setting, $\\hat{z}\\_\\theta(x)$ represents the decision that some base decision-model suggests. We scale this decision by $\\lambda$, where $\\lambda$ intuitively quantifies how much we trust the base decision-model. If we fully trust the model, we set $\\lambda = 1$ and perfectly follow the base decision-model. If we don't trust the model at all, we set $\\lambda = 0$, resulting in a do-nothing decision: $\\hat{z}\\_\\theta(x) = 0$. The amount of trust (which we call \"aggressiveness\" in our paper) is naturally characterized by a 1-d parameter. We will revise our paper to more clearly explain this 1-d characterization.\n\nNonetheless, we recognize that not all decisions can be easily boiled down to a 1-d parameter. Notably, the paper \"How to Trust Your Diffusion Model\" (Teneggi et al., 2023), provides an example of multi-dimensional $\\lambda$ in the RCPS setting. We believe that investigating similar generalizations of our CORC method to accommodate multi-dimensional $\\lambda$ would be an interesting direction of future work."}, {"review_id": "jc6wTANvzo", "reviewer": "Reviewer_mggh", "summary": "The paper has two key contributions:\n(i) a generalization of Conformal Risk Control for general Optimized Certainty Equivalent (OCE) risks, which enables the control of risks beyond average risks: e.g., CVaR and Entropic risks. The paper pays particular attention to the CVaR case.\n(ii) A conformal training scheme, which allows for the propagation of gradients through the calibration process of the proposed conformal OCE risk control procedure (which subsumes the standard Conformal Risk Control / Vanilla Conformal procedures). This is formally derived, a distinct feature in comparison to previous works.\nBoth schemes seem to have favorable empirical results.", "questions": "I'd appreciate it if the authors could clarify my questions in the 'Strengths&Weaknesses' section regarding the proof of Theorem 3.", "limitations": "Limitations were appropriately discussed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "- The generalization of Conformal Risk Control for Optimized Certainty Equivalent risks is quite nice, and the ability to work with more general risks such as CVaR is highly desirable. The only cost seems to be (besides boundedness assumptions) the introduction of the $t$ hyperparameter, which the authors propose to be selected using a separate data split. How sensitive is the choice of $t$? Is there some guidance for how to select it a priori?\n\n- On Theorem 3 (regarding the conformal training scheme):\n    - I strongly advise the authors to write the closed-form gradients in the proposition of the theorem, at the very least in the supplementary material, rather than hiding them in the proofs.\n    - This is, as far as I am aware, the most formal treatment of conformal training to date, for which I applaud the authors. I just have some (hopefully) minor comments:\n        - On the proof of case (i):\n            - The writing of the proof is a bit tough to follow, in particular due to simultaneous notational overload and notational abuse. Perhaps it can be more clarified?\n            - Line 960: it is claimed that because $(\\varphi, c\\_j, g\\_j)$ are differentiable almost everywhere, so is $\\tilde{c}_j$. But as far as I can tell this involves a use of the chain rule, which need not apply for almost-everywhere differentiability. Did I miss something? Or maybe $\\varphi$ should be assumed to be differentiable everywhere? (Also the role of the $g_j$ seems delicate here, and I'm not entirely certain that differentiability is indeed preserved.)\n        - On the proof of case (ii):\n            - Lines 986-988 claim that we can replace $\\ell_i$ with $\\lambda$ (or $-\\lambda$), but I'm not sure this is correct; could the authors clarify why this is so?\n            - The idea of the proof seems to then be to reduce the problem to a standard bilevel optimization (via a sort of duality argument). Is my interpretation correct?\n    - The theorem establishes that the gradient exists almost everywhere, and somewhat states how to compute them in the proof. However, is it guaranteed that the gradients, where defined, are at least non-zero? For example, a combinatorial loss (e.g., AUC, Accuracy) is also differentiable almost everywhere, but has nil gradients where it is defined (and thus its gradients are useless). I *think* that generally the gradients in the proposed approach will generally be useful, but that there may be some situations under which the gradients become nil.\n\n- Finally, I am somewhat bothered by the title of the paper. As mentioned, the paper has two key contributions, these being (i) the new calibration procedure for OCE risks, and (ii) the conformal training approach.\nContribution (i) seems quite solid, and I think would have even stood as a paper on its own.\nContribution (ii) is also nice, but essentially spans a single theoretical result, and is reliant on the formulation of (i) (though it does seem like it could be restated for the special case of the more usual risks).\nOverall, the paper's content reads as the primary contribution being (i), with (ii) being a nice consequence.\nHowever, the title refers solely to (ii), making no mention at all to (i) or Optimized Certainty Equivalents; one would think that it is a mere adaptation of conformal training onto conformal risk control (of average risks). It's more than that!\nI realise this is a bit nitpicky on my part, but it feels like a genuine mismatch between the title and content.", "was_revised": true, "final_justification": "The authors reasonably answered my concerns. I was unfortunately unable to carefully review the new version of the theorem, as it is very lengthy and technical, but their comments beforehand inspire some confidence. I generally lean towards acceptance: the paper is on a relevant topic, and I think their contributions -- both on controlling OCE risks as well as deriving gradients for conformal training -- are of significant interest to the conformal prediction community. I do, however, keep my score to a borderline accept due to (i) the new theorem, which I was unable to fully check, and (ii) the title, which I believe to have a slight mismatch with the paper.", "rebuttal": "We thank the reviewer for recognizing our Conformal OCE Risk Control method as a desirable generalization of Conformal Risk Control to work with more general risks such as CVaR. We also thank the reviewer for recognizing our formal treatment of conformal training. We respond to each of the points raised by the reviewer.\n\n**Sensitivity to $t$ hyperparameter**\n\nAs the reviewer points out, Conformal OCE Risk Control is dependent on picking a good hyperparameter $t$ to tighten the OCE risk bound. From a practical / implementation perspective, we recommend selecting $t$ using a separate data split.\n\nFrom a more theoretical perspective, we can see that $t$ shows up in two inequalities in the Proof of Theorem 1 (Appendix C.2):\n\n$$\nR[L_{N+1}(\\hat\\lambda)]\n\\leq \\mathbb{E}\\left[ t + \\phi(L_{N+1}(\\hat\\lambda) - t) \\right]\n= \\mathbb{E}[\\tilde{L}_{N+1,t}(\\hat\\lambda)]\n\\leq \\alpha.\n$$\n\nThe first inequality is tight at the value $t$ that minimizes the variational OCE expression. For example, when $R = \\text{CVaR}^\\delta$, $t=\\text{VaR}^\\delta[L(\\hat\\lambda)]$ makes the first inequality tight. On the other hand, theoretically analyzing the best choice of $t$ to tighten the 2nd inequality is difficult in general. We welcome suggestions, but for now, we leave this to future work.\n\nIn the 2 tables below, we show the empirical sensitivity of the average task loss and the empirical CVaR on the choice of $t$ for post-hoc Conformal CVaR Control in the battery storage task. Here, $\\Delta t$ is the (absolute) amount by which we change $t$ from the value tuned on the training set. Evidently, the value of $t$ tuned on the training set is not always optimal, but the resulting task loss tends to be close (within 1 stddev) of the task loss from the optimal $t$.\n\nDue to space limitations, we can only show a few choices of $\\Delta t$, but we will include a proper figure in the camera-ready.\n\n_Table 1_: Task loss\n| Δt | α=2, δ=0.9 | α=2, δ=0.95 | α=2, δ=0.99 | α=5, δ=0.9 | α=5, δ=0.95 | α=5, δ=0.99 | α=10, δ=0.9 | α=10, δ=0.95 | α=10, δ=0.99 |\n|-|-|-|-|-|-|-|-|-|-|\n| -5 | 0 ± 0 | 0 ± 0 | 0 ± 0 | 0 ± 0 | 0 ± 0 | 0 ± 0 | -8.4 ± 18 | 0 ± 0 | -2 ± 3.2 |\n| -1 | 0 ± 0 | 0 ± 0 | -0.39 ± 0.64 | 0 ± 0 | -2.9 ± 6.1 | -3.7 ± 1.7 | -8.4 ± 18 | -6.2 ± 13 | -8.4 ± 3 |\n| 0 | 0 ± 0 | -1.5 ± 2.7 | -1.8 ± 0.51 | -4.2 ± 13 | -3.6 ± 6.7 | -4.5 ± 1.3 | -13 ± 20 | -7.3 ± 13 | -9 ± 2.6 |\n| 1 | 0 ± 0 | -2.2 ± 3 | -0.95 ± 0.7 | -8.4 ± 18 | -5.3 ± 7.3 | -4.5 ± 0.76 | -13 ± 20 | -10 ± 14 | -9.3 ± 2.1 |\n| 5 | -1.9 ± 6 | -0.69 ± 1.5 | 0 ± 0 | -8.4 ± 18 | -5.1 ± 5.9 | 0 ± 0 | -13 ± 20 | -13 ± 15 | -4.8 ± 3.5 |\n\n_Table 2_: CVaR\n| Δt | α=2, δ=0.9 | α=2, δ=0.95 | α=2, δ=0.99 | α=5, δ=0.9 | α=5, δ=0.95 | α=5, δ=0.99 | α=10, δ=0.9 | α=10, δ=0.95 | α=10, δ=0.99 |\n|-|-|-|-|-|-|-|-|-|-|\n| -5 | 0 ± 0 | 0 ± 0 | 0 ± 0 | 0 ± 0 | 0 ± 0 | 0 ± 0 | 1 ± 2.3 | 0 ± 0 | 1.3 ± 2.1 |\n| -1 | 0 ± 0 | 0 ± 0 | 0.27 ± 0.43 | 0 ± 0 | 0.87 ± 1.8 | 2.7 ± 1.1 | 1 ± 2.3 | 1.9 ± 4 | 6.1 ± 1.6 |\n| 0 | 0 ± 0 | 0.46 ± 0.81 | 1.3 ± 0.24 | 0.39 ± 1.2 | 1.1 ± 2 | 3.3 ± 0.59 | 1.5 ± 2.5 | 2.3 ± 4.1 | 6.7 ± 1.2 |\n| 1 | 0 ± 0 | 0.68 ± 0.9 | 0.77 ± 0.61 | 1 ± 2.3 | 1.6 ± 2.2 | 3.4 ± 0.53 | 1.5 ± 2.5 | 3.2 ± 4.4 | 6.9 ± 1 |\n| 5 | 0.17 ± 0.53 | 0.31 ± 0.68 | 0 ± 0 | 1 ± 2.3 | 1.7 ± 1.9 | 0 ± 0 | 1.5 ± 2.5 | 4.2 ± 4.6 | 3.8 ± 3 |\n\n**Theorem 3**\n\nWe thank the reviewer for a very detailed reading of our proof, and we apologize for the heavy notation. If the reviewer can point to specific lines that are unclear, we welcome feedback to improve the clarity of the proof.\n\nWe also deeply appreciate the reviewer for identifying our regrettable mistake in applying the chain rule to the composition of differentiable-almost-everywhere functions. We noticed that we actually made the same mistake in both case (i) and case (ii). The mistake in case (i) is as the reviewer pointed out. The mistake in case (ii) lies in our claim that $\\tilde{L}_{i,t}$ is continuously differentiable in $\\theta$.\n\nTo correct this mistake, one option is indeed to assume that $\\phi$ is differentiable almost everywhere, but then this excludes CVaR disutility function $\\phi_{\\text{CVaR}^\\delta}(x) = \\frac{1}{1-\\delta} [x]_+$ which is not differentiable at the origin.\n\nInstead, we will update Theorem 3 to make the more generic assumption that $\\tilde{L}_{i,t}$ is differentiable almost everywhere in $(\\theta, \\lambda)$. And for case (i), we will directly assume that $\\tilde{L}\\_{i,t}$ has the form\n\n$$\n\\tilde{L}\\_{i,t}(\\theta,\\lambda) = c\\_{i,0}(\\theta) + \\sum\\_{j \\in \\mathbb{N}} c\\_{i,j}(\\theta) \\cdot \\mathbf{1}[g\\_{i,j}(\\theta) < \\lambda].\n$$\n\nThese changes make our theoretical result less elegant, but do not fundamentally change the applicability of the theory to CVaR and related risk measures.\n\nWe also note that our mistake does not affect our empirical experiments. Because the disutility function is only non-differentiable at the origin, the likelihood in practice that $c_0(\\theta) - t + \\sum\\_{j=1}^k c\\_j(\\theta) = 0$ is minimal, if not negligible.\n\n**Theorem 3 case (ii)**\n\nWe apologize for the confusion about replacing $\\ell_i$ with $\\lambda$. What we mean is that if $\\ell\\_i(\\theta, \\lambda)$ are strictly increasing in $\\lambda$, then\n\n$$\\begin{aligned}\n& \\arg\\min\\_\\lambda \\frac{1}{N} \\sum\\_{i=1} \\ell\\_i(\\theta, \\lambda)\n\\quad\\text{s.t.}\\ \\\n\\tilde{h}\\_t(\\theta, \\lambda) \\leq \\alpha \\\\\\\\\n&= \\ \\arg\\min\\_\\lambda \\frac{1}{N} \\sum\\_{i=1} \\lambda\n\\quad\\text{s.t.}\\ \\\n\\tilde{h}\\_t(\\theta, \\lambda) \\leq \\alpha.\n\\end{aligned}$$\n\nWe will clarify this in the camera-ready revision.\n\nAlso, the interpretation that we are reducing the problem to a standard bilevel optimization is correct. However, we are not using any duality argument.\n\n\n**Non-zero gradients**\n\nThe reviewer is correct that Theorem 3 does not guarantee the gradients to be non-zero. For example, if $\\ell$ and $L$ depend on a neural network that outputs a constant value, then clearly $\\lambda(\\theta)$ is constant, and therefore the gradient is zero. Thus, it is up to the user to define $\\ell$ and $L$ such that the gradient is non-zero.\n\n**Paper title**\n\nWe thank the reviewer for appreciating both of our contributions, and we are open to suggestions to improve the paper title (if permitted by the NeurIPS conference). We considered including \"optimized certainty-equivalent\" in the title, but we found that it made the title longer than we liked.\n\nSome options that we considered include:\n\n- Conformal Risk Training: End-to-end optimization of conformal optimized certainty-equivalent risk control\n- Conformal Risk Training: Learning to control optimized certainty-equivalent risks\n- Conformal Risk Training: Differentiable control of optimized certainty-equivalent risks"}, {"review_id": "Mit4H0RQhk", "reviewer": "Reviewer_A5AC", "summary": "This paper addresses two key limitations of standard Conformal Risk Control (CRC) methods [Angelopoulos et al.]- 1) while CRC provides finite-sample guarantees on expected loss, many real-world high-stakes applications require control over tail risks (such as Conditional Value-at-Risk (CVaR)), 2) CRC is typically applied post hoc to a pretrained model, which may degrade performance due to the lack of feedback between risk calibration and model learning.\n\nTo address the first limitation, the authors introduce a new post-hoc conformal risk control framework, called Conformal OCE Risk Control (CORC). This framework extends the original CRC method to a broader family of risk measures known as Optimized Certainty Equivalent (OCE) risks. These include not only the standard expected loss (recovering CRC) but also more tail-sensitive metrics like CVaR. \n\nTo address the second limitation, the authors propose a new training strategy called conformal risk training, which incorporates feedback from CORC into the model’s training process. This approach jointly optimizes both the model parameters and the risk parameter by minimizing performance loss while ensuring the risk stays within the target upper bound. It is conceptually similar to the conformal training method introduced by David (cite), but extends it to the conformal risk control setting.\n\nThe proposed conformalized risk training method is evaluated on two high-stakes tasks—tumor image segmentation and battery energy storage optimization. The results demonstrate that the method provides valid risk control while also improving efficiency and utility.\n\n[Angelopoulos et al.] A. N. Angelopoulos, S. Bates, A. Fisch, L. Lei, and T. Schuster. Conformal Risk Control, Apr. 2023", "questions": "1. In your experiments involving conformal risk training, did you apply an additional layer of CRC or CORC after model fine-tuning? Or were the reported results directly from the trained model? \n2. Have you derived a lower bound on the test-time risk, similar to the approach taken in the original CRC paper?\n3. Do you have any results comparing CRC and CORC purely as post-hoc methods (i.e., using the same pretrained models without conformal risk training)?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1. Clarity and presentation: This paper is very well-written and well-structured. The authors clearly present their problem statement and articulate their motivations effectively. They also provide an easy-to-follow description of the methodology along with solid theoretical justification. I particularly appreciate the inclusion of intuitive examples that help clarify the technical details throughout the paper.\n2. Novelty and Significance: This paper addresses an important challenge in high-stakes applications—controlling tail risks—and fills a notable gap in the conformal prediction literature. It extends the existing conformal risk control framework in a principled manner to accommodate a broad class of Optimized Certainty Equivalent (OCE) risk functions. Additionally, the paper examines specific instances within the OCE family, such as Conditional Value-at-Risk (CVaR), and demonstrates that the standard non-decreasing assumption can be relaxed to monotonicity in this case. To the best of my knowledge, this is also the first work to incorporate conformal risk control into the conformal training paradigm.\n3. Soundness: The authors provide rigorous theoretical justification for the validity of their method and demonstrate improved efficiency through numerical experiments.\n\nWeaknesses: \n1. Please correct me if I have overlooked anything, but the empirical evaluation appears limited. The method is tested on only two datasets, each evaluated under a limited set of configurations (e.g., varying the target risk level $\\alpha$ or tail risk level $\\delta$. The work would be further strengthened by evaluating the proposed method on a more diverse set of classification tasks. Additionally, it may be helpful to explore the impact of varying calibration set sizes, as the proposed approach requires an additional data split to ensure validity. This introduces a trade-off between reduced data efficiency and the benefits of the proposed method. Investigating the performance in low-calibration regimes could provide valuable insights into the method's robustness and practical applicability. It would be particularly interesting to explore the potential benefits of the method when the calibration sample size is small.", "was_revised": true, "final_justification": "This paper addresses an important methodological gap that is valuable to the conformal prediction community. The authors provide both theoretical justifications and empirical evidence demonstrating the effectiveness of their proposed method. In addition, their rebuttal offered clear and informative responses that satisfactorily resolved my concerns. Based on these considerations, I recommend acceptance of this paper.", "rebuttal": "We appreciate the reviewer for recognizing the novelty and significance of our Conformal Risk Training method for learning to control the broad class of optimized certainty-equivalent risks. We respond to each point raised by the reviewer.\n\n**Empirical evaluation**\n\nWe have taken note of the reviewer's desire to see evaluation on more datasets. While we did not have time to run more experiments during this 1-week rebuttal period, we are actively looking into other datasets to test our method.\n\nThe reviewer also suggested evaluating the effect of varying the calibration set size on tail risk control. Below are 2 tables that show the average task loss and the CVaR on the battery storage problem as the calibration set size changes for post-hoc conformal CVaR Control. Values are given as mean ± std over 10 random seeds for choosing the calibration set.\n\nAs the tables show, a larger calibration set is generally better, both in terms of reducing the task loss and achieving a tighter CVaR bound closer to $\\alpha$ (_i.e._, not being overly conservative). The reason a larger calibration set size $N$ helps is because the term $\\frac{1}{N+1} \\tilde{B}_t(\\lambda)$ appears in $\\tilde{h}_t(\\lambda)$. Increasing $N$ reduces the effect of the upper bound $B$ and enables choosing larger (less conservative) $\\hat\\lambda$. However, there are diminishing gains as $N$ increases, which we clearly observe when $\\delta = 0.9$.\n\n\nTable 1: Average task loss\n\n| calib size | α=2, δ=.9 | α=2, δ=.95 | α=2, δ=.99 | α=5, δ=.9 | α=5, δ=.95 | α=5, δ=.99 | α=10, δ=.9 | α=10, δ=.95 | α=10, δ=.99 |\n|-|-|-|-|-|-|-|-|-|-|\n| 100 | 0 ± 0 | -0.64 ± 1.3 | -0.8 ± 0.073 | 0 ± 0 | -1.6 ± 3.4 | -2 ± 0.18 | -7.7 ± 16 | -3.2 ± 6.7 | -4 ± 0.37 |\n| 300 | 0 ± 0 | -1.1 ± 2.3 | -1.4 ± 0.26 | -4.2 ± 13 | -2.6 ± 5.7 | -3.4 ± 0.66 | -13 ± 20 | -7.2 ± 12 | -6.8 ± 1.3 |\n| 613 (default) | 0 ± 0 | -1.5 ± 2.7 | -1.8 ± 0.51 | -4.2 ± 13 | -3.6 ± 6.7 | -4.5 ± 1.3 | -13 ± 20 | -7.3 ± 13 | -9 ± 2.6 |\n\nTable 2: CVaR\n\n| calib size | α=2, δ=.9 | α=2, δ=.95 | α=2, δ=.99 | α=5, δ=.9 | α=5, δ=.95 | α=5, δ=.99 | α=10, δ=.9 | α=10, δ=.95 | α=10, δ=.99 |\n|-|-|-|-|-|-|-|-|-|-|\n| 100 | 0 ± 0 | 0.19 ± 0.41 | 0.61 ± 0.11 | 0 ± 0 | 0.48 ± 1 | 1.5 ± 0.28 | 0.96 ± 2.1 | 0.97 ± 2.1 | 3 ± 0.57 |\n| 300 | 0 ± 0 | 0.31 ± 0.66 | 1 ± 0.2 | 0.39 ± 1.2 | 0.79 ± 1.7 | 2.6 ± 0.49 | 1.5 ± 2.5 | 2.1 ± 3.5 | 5.1 ± 0.98 |\n| 613 (default) | 0 ± 0 | 0.46 ± 0.81 | 1.3 ± 0.24 | 0.39 ± 1.2 | 1.1 ± 2 | 3.3 ± 0.59 | 1.5 ± 2.5 | 2.3 ± 4.1 | 6.7 ± 1.2 |\n\n\n**Applying CRC / CORC after fine-tuning**\n\nIn all of our results, we always apply CRC or CORC after model training with a separate held-out validation set in order to formally guarantee risk control.\n\n**Lower bound on test-time risk**\n\nWe have not derived a lower bound on the test-time risk. If we apply the same derivation from the original CRC paper to OCE risks, and we assume that $B$ is constant, we end up with\n\n$$\n\\mathbb{E}[\\tilde{L}\\_{N+1,t}(\\hat\\lambda)]\n\\geq \\mathbb{E}\\left[ \\frac{1}{N+1} \\sum\\_{i=1}^{N+1} \\tilde{L}\\_{i,t}(\\hat\\lambda'') \\right]\n\\geq \\alpha - \\frac{2 \\tilde{B}\\_t}{N+1}\n$$\n\nwhere\n\n$$\n\\hat\\lambda''\n:= \\inf\\left\\\\{ \\lambda \\in \\Lambda \\ \\middle|\\ \\frac{1}{N+1} \\left( \\tilde{B}\\_t + \\sum\\_{i=1}^{N+1} \\tilde{L}\\_{i,t}(\\hat\\lambda) \\right) \\leq \\alpha \\right\\\\}.\n$$\n\nHowever, we have $R[L\\_{N+1}(\\lambda)] \\leq \\mathbb{E}[\\tilde{L}\\_{N+1,t}(\\lambda)]$, and the inequality is generally not tight unless $t$ is chosen to be the minimizer of the variational form of the OCE risk. Therefore, we cannot directly compare $R[L\\_{N+1}(\\hat\\lambda)]$ with $\\alpha - \\frac{2 \\tilde{B}\\_t}{N+1}$.\n\n\n**Comparing CRC and CORC as purely post-hoc methods**\n\nWe are not 100% sure what the reviewer means with their question, so we will try to answer the question in 2 different ways.\n\nIf the reviewer is asking about comparing post-hoc CRC against post-hoc CORC, we would like to point out that if the risk measure is the expected loss, then CRC and CORC are exactly equivalent. On the other hand, if the risk measure is not the expected loss, then CRC is undefined. Therefore, it does not make sense to compare post-hoc CRC against post-hoc CORC in this setting.\n\nIf the reviewer is instead asking to see comparisons of post-hoc CRC vs. Conformal Risk Training (with expected loss), or comparisons of post-hoc CORC vs. Conformal Risk Training (with CVaR risk), we note that this comparison is the focus of our experimental section, and we apologize if our experimental section did not make this clear---we will clarify this accordingly in the camera-ready paper."}]

Author rebuttal:

Meta-review: This paper makes two contributions: (i) extending CRC to Optimized Certainty-Equivalent (OCE) risks, and (ii) proposing a method for differentiating through CRC-based optimization objectives for model training.  The reviewers are unanimously in favor of acceptance.  Most of the reviewers' critical comments / questions were about details such as hyperparameters and limitations that also apply to CRC (use of calibration sets and restriction to monotonic losses).
Decision: Accept (poster)