Conformal Risk Control
Authors: Anastasios Nikolas Angelopoulos
Stephen Bates
Adam Fisch
Lihua Lei
Tal Schuster
Venue: ICLR 2024 spotlight (2024)
Keywords: conformal prediction
uncertainty quantification
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: Generalizing the conformal guarantee to control risks (expectations of losses).

Abstract: We extend conformal prediction to control the expected value of any monotone loss function. The algorithm generalizes split conformal prediction together with its coverage guarantee. Like conformal prediction, the conformal risk control procedure is tight up to an $\mathcal{O}(1/n)$ factor. We also introduce extensions of the idea to distribution shift, quantile risk control, multiple and adversarial risk control, and expectations of U-statistics. Worked examples from computer vision and natural language processing demonstrate the usage of our algorithm to bound the false negative rate, graph distance, and token-level F1-score.

Published as a conference paper at ICLR 2024
CONFORMAL RISK CONTROL
Anastasios N. Angelopoolos'' Stephen Bates", Adam Fisch", Lihua Lei², Tal Schuster
UC Berkeley
MIT
Stanford 'Google Research
ABSTRACT
We extend conformal prediction to control the expected value of any monotone loss function. The algorithm generalizes split conformal prediction together with its coverage guarantee. Like conformal prediction, the conformal risk control procedure tight up to an (11/n) factor. We also introduce extensions of the idea to distribution shift, quantile risk control, multiple and adversarial risk control, and expectations of U-statistics. Worked examples from computer vision and natural language processing demonstrate the usage of our algorithm to bound the false negative rate, graph distance, and token-level F1-score.
INTRODUCTION
We seek to endow some pre-trained machine learning model with guarantees on its performance as to ensure its safe deployment. Suppose have base model that is a function mapping inputs x X' to values in some other space, such as a probability distribution over classes. Our job is to design procedure that post-processes the output of f to give it statistical safety property.
Split conformal prediction (Vovk al. 2005 Papadopoulos al., 2002), which we will refer sim- ply as "conformal prediction", has been useful in areas such as computer vision (Angelopoulos et al.) 2021b) and natural language processing (Fisch et al., (221)) to provide such a guarantee. By measur- response pairs, con- formal prediction post-processes the model to construct prediction sets that bound the miscoverage,
ing the model's performance on a calibration dataset { {X,, Y)) ;=1 of feature-response
P (Yn+11 & C(Xn+1)) S
(1)
where (Xn+1, Yu+1) is new test point, a user-specified error rate (e.g., 10%), and a function of the model and calibration data that outputs a prediction set. Note that C is formed using the first n data points, and the probability in ( is over the randomness all n + I data points (i.c., the draw of both the calibration points I,..., and the test point 1).
In this work, ror is not simply miscoverage. In particular, our main result is that generalization of conformal prediction provides guarantees of the form
extend conformal prediction to prediction tasks where the natural notion of
E (CCC((X
S Sa,
for any bounded loss function & that shrinks as Cx(Xn+1) grows, and À is an input parameter that controls the growth of Cx CX(xx+1). We call this conformal risk control. Note that (2) recovers the conformal miscoverage guarantee in (T) when using the miscoverage loss, (X,x)))))) Yn+1)
1 (Yr+1 Cx(Xx++)}, However, our algorithm also extends conformal prediction situations where other loss functions, such as the false negative rate (FNR), are more appropriate.
As an example, consider multilabel classification, where the Y; C {1, K} are sets comprising subset of K classes. Creating massive; instead, given a trained multilabel classifier f : X' + [0, 1]K we want to output sets that include large fraction of the true classes Y.. To that end, we post-process the model's raw outputs into the set of classes with sufficiently high scores, Cx(x) {k: f(X); 2 1 A), where the main parameter of the algorithm À E [0, 1] threshold. Note that as the threshold À grows, we include more classes in Cx it becomes more conservative. In this case, conformal risk control finds threshold value Á that controls the fraction of missed classes, i.e., the expected value
that contain all the classes may be too conservative if K is

---

Published as a conference paper at ICLR 2024
of (X)+),)) 1 [Y (Xx++\)))/) Setting 0.1 would ensure that our algorithm produces sets C2 (Xn+1) containing 2 90% of the true classes in Yn+ 1 on average.
1.1 ALGORITHM AND PREVIEW OF MAIN RESULTS
Formally, we will consider post-processing the predictions of the model f to create function C.). The function has a parameter that encodes its conservativeness: larger À values yield more conservative outputs (e.g., larger prediction sets). To measure the quality of Cx.
consider loss function ((C)(z).)) E (-o0, B] for some B < cc. We require this loss to be non-increasing in X. Our goal is to choose Å based on the observed data ()) so that risk control as in ② holds.
We now rewrite this same task in a more notationally convenient and abstract form. Consider an exchangeable collection of -on-increas,ng, bounded, random functions L3 : A + (-00, B), i .,..., 1, where A is the space of all inputs (c.g.. 'thresholds') to the function L, Throughout the paper, we assume Amax == sup A E A, so that [;(Amax) is well-defined and satisfies (Amaxx) for any by design (i.e., a is achievable). We seek to the first functions choose a value of the parameter, Å, that the risk on the unseen function controlled:
E G[itta (2) s a.
(3)
We are primarily motivated by the case where Li(A) - E(Cx(X.)) Y.), in which case the guarantee
in (3) coincides with risk control as in (2)
..... + I, (A))/n. Given any desired risk Now we describe the algorithm. Let Rz (A) = (I1(A) +...
level upper bound (-00, B), define
n Rr (A) { A n m+1
a n+ S a { inf {A: RZ)() S B-a
À = inf
(4)
Since Rt (A) is monotone, we can efficiently search for Å using binary search to arbitrary precision. When the is empty, we define Á Amax- Our proposed conformal risk control algorithm to deploy Á on the forthcoming test point. Our main result is that this algorithm satisfies ( Intuitively, we can see that this algorithm reduces to searching for a value of À that results in slightly conservative empirical risk-the gets less conservative when the difference between the worst-case risk (B) and the desired risk (a) is smaller, or the calibration set size (n) is larger.
Moreover, when the L; are i..dd from continuous distribution, satisfies a tight lower bound saying is not too conservative,
show that the algorithm
E Ln+ (2) 2
2B
n +1
We show the reduction from conformal risk control to conformal prediction Appendix A Fur- thermore, if the risk is non-monotone, then this algorithm does not control the risk; we discuss this in Section 2.3] Finally, we provide both practical examples using real-world data and several theoretical extensions of our procedure in Sections 3 and respectively.
1.2 RELATED WORK
Conformal prediction developed by Vladimir Vovk and collaborators beginning in the late 1990s (Vovk et al.] [999 [ and has recently become popular uncertainty estimation tool in the machine learning community, due its favorable model-agnostic, distribution-fres simile-sapple guarantees. See Angelopoulos & Bates 2021) for modern introduction to the area or Shafer & Vovk (2008) for more classical alternative. As previously discussed, in this paper we primarily build on splir conformal prediction (Papadopoulos et al. 2002): statistical properties of this algo- rithm including the coverage upper bound were studied in Lei et al. (2018). Recently there have been many extensions of the conformal algorithm, mainly targeting deviations from exchangeabil-
ity (Tibshirani et al. 2019 Gibbs & Candes 2021 Barber et al. 2022 Fannjiang et al., 2022) and
Note that any unbounded loss can be transformed to a bounded loss. For example, any unbounded loss É(X) on the positive reals can be transformed by taking the inverse tangent, i.e., ('()) arctan (((A)). As long the transformation is monotone, loss of the original loss corresponds exactly to a loss of a' on the new loss; controlling this transformed risk may be enough in practice.
2

---

Published as a conference paper at ICLR 2024
improved conditional coverage (Barber al. 2020 Romano et al.
2019 Guan 2020: Romano
cet al. 2020; Angelopoulos et al., 202 Most relevant to us is recent work on risk control in high
probability [ovk 2012; Bates al.) 2021 Angelopoulos al. 202 and its applications Park
et al. 2020; Fisch et al. 2022 Schuster et al. 2021 2022] Sankaranarayanan et al. 2022 An- gelopoulos al. 2022a inter alia). However, while Bates al. (2021) and Angelopoulos et al. (2021a) operate in similar mathematical settings and we reuse much of their notation, the algorithm presented herein differs greatly. It far more sample-efficient, simpler, and provides a guarantee in expectation versus a guarantee in probability. Our algorithm is entirely different- no existing algorithm gives guarantees in expectation for risk control and its validity proof is mathematically unrelated to these previous works. See Appendix B for a detailed discussion and comparison of these algorithms, including experiments.
To further elaborate on the difference between our work and the broader existing literature, first con- sider conformal prediction. The purpose of conformal prediction is provide coverage guarantees of the form [T. The guarantee available through conformal risk control, (3) strictly subsumes that of conformal prediction; it is generally impossible recast risk control as coverage control. As second question, one might ask whether (3) can be achieved through standard statistical machinery, such uniform concentration inequalities. Though it is possible to integrate uniform concentration inequality to get bound in expectation, this strategy tends to be excessively loose both in theory and practice (see, e.g., the bound of [Anthony Shawe- Shawe-Taylor (1993). The technique herein avoids these complications; it is simpler than concentration-base approaches, practical to implement, and tight up a factor 1/n, which is comparatively faster than concentration would allow.
2
THEORY
In this section, we establish the core theoretical properties of conformal risk control. All proofs, unless otherwise specified, are deferred to Appendix
2.1 RISK CONTROL
We first show that the proposed algorithm leads to risk control when the loss is monotone.
Theorem 1. Consider sequence of exchangeable random loss functions, {L, (A)) where L
+ R which are non-ince increasine X, right right-continuous, and for Amax = sup A E A, satisfy
Tz()mmax) S a, sup T,(A) S B < Do almost surely.
(5)
Then
B[Ln+1(A]] a.
Proof. Let Rx+1(A)= = (LI(A) + .... Ln+1 (A)) and \' = inf { A: Ro (A) S a
Since inf Lx(A) Li(Amax) S <a, X well-defined almost surely. Since Lm+1(A) B, we know n+T (2) + (x) s - - RA)A) a+1
(A) = n
+ n Thus,
B
n
R (A)
n+1 ca (A)
n
This implies \' when the LHS holds for some X E A. When the LHS is above a for all X E A, by definition, Amax X. Thus, X < Á almost surely. Since (;(A) is non-increasing in X, B[Tn+t (2) < E Timti (8
(6)
Let E be the multiset loss functions {LI...., Lntthh,,, Then X a function E, or, equiva- lently, X' is constant conditional on E. Additionally, Ln+1 (A) ~ Uniform((Ly ((L...,, exchangeability. These facts combined with the right-continuity of L, imply
Ln+1 I.,
by
a+1
[m++1 |
Σ
L,(X' sa
n
The proof is completed by the law of total expectation and 6
3

---

Published as a conference paper at ICLR 2024
2.2 A TIGHT RISK LOWER BOUND
Next we show that the conformal risk control procedure is tight up factor 2B/ (n that cannot be improved in general. The proof will rely on a form of continuity that generalizes the assumption of continuous
conformity scores used for the standard conformal proof. Define the jump func- tion, which quantifies the size of the discontinuity in a Tight-continuous input function I at point X, J[, A) 1o+ o+ 1(A - e) - I(A). the probability that L, has discontinuity any pre-specified is ([, X) 0) = O, then the conformal risk control procedure is not too conservative.
Theorem 2. In the setting of Theorem
! further assume that the L, are i.i.d. L, 2 0, and for any
X, P (J(L,, > 0) Then
2B
E [ 2 n +
This bound is tight for general monotone loss functions, we show next.
Proposition 1. In the setting of Theorem for any > 0, there exists a loss function and a E (0, 1) such that
E Lntti (2) a 2B
n+1
Since take arbitrarily close to zero, the factor 2B/(n 1) in Theorem is required. Con- formal prediction both the algorithm and the guarantee iss exactly equivalent to conformal risk control when the loss function is indicator, including the tighter lower bound (see Appendix Ap.
2.3 CONTROLLING GENERAL LOSS FUNCTIONS
We next show that the conformal risk control algorithm does not control the risk if the L; are not assumed to be monotone. In particular, (3) does not hold. We show this by example.
Proposition 2. For any e, there exists a non-monotone loss function such that
1 (2) B-c.
Notice that for any desired level a, the expectation in B can be arbitrarily close to B. Since the function values here are in [0, B), this means that even for bounded random variables, risk control can be violated by arbitrary amount-unless further assumptions placed the I4- However, the algorithms developed may still be appropriate for near-monotone loss functions. Simply 'monotonizing' all loss functions L; and running conformal risk control will guarantee ( but this strategy will only be powerful (i.e., not conservative) if the loss is near-monotone. For concreteness, we describe this procedure below corollary of Theorem !
Corollary 1. Allow (A) be any (possibly non-monotone) funonotoon)e of satisfying 5 Take
Í,(A) = sup L(X), Rx(A) 1 Σ L.()) and Ā = inf =
n Rx()) +
S a
X2
n+1
n+
Then,
E Lnta (2) sa. sa.
the loss function already monotone, then Ả reduces to Å. We propose further algorithm for picking À in Appendix C that provides asymptotic risk-control guarantee for non-monotone loss functions. However, this algorithm again is only powerful when the risk E[Ln+1(A) (X)] is near- monotone and reduces the standard conformal risk control algorithm when the loss is monotone.
EXAMPLES
To demonstrate the flexibility and empirical effectiveness of the proposed algorithm, we apply it to four tasks across computer vision and natural language processing. All four loss functions binary, monotone losses bounded by 1. They are commonly used within their respective application domains. Our results validate that the procedure bounds the risk as desired and gives useful outputs
4

---

Published as a conference paper at ICLR 2024
30
denakt
20
10-4
10
10
0.05
0.07
0.08
0.20
0.11
0.12
0.13
1.
20
fraction of
Figure 1: FNR control in tumor segmentation. The top figure shows examples of procedure with correct pixels in white, false positives in blue, and false negatives in red. The bottom plots report FNR and set size over 1000 independent random data splits. The dashed gray line marks a.
tork cake
penson tennis racket
suitcase
person
penson
traffic light
tie
dining
handbag
80
0.25
denstty
0:20
0:35
0.10
0.65
0.00
0.09
0.30
0.11
0.12
6
30 12
risk
size
Figure 2: FNR control on MS COCO. The top figure shows examples of our procedure with correct classes in black, false positives in blue, and false negatives red. The bottom plots report FNR and set size over 1000 independent random data splits. The dashed gray line marks a.
to the end-user. We note that the choices of Cx used herein are only for the purposes of illustration; any nested family of sets will work. For each example use case, for representative (details provided for each task) we provide both qualitative results and quantitative histograms of the risk and set sizes over 1000 random data splits that demonstrate valid risk control (i.e., with mean S o).
3.1 FNR CONTROL IN TUMOR SEGMENTATION
In tumor segmentation, our input is a d x d image and our label is a set of pixels Y, e o (((1, 1), (1, ..., (d, d))), with ş the power set. We an image segmentation model f X'
[0, 1 dxd outputting probability for each pixel and measure loss as the fraction of false negatives,
L FNR (A) = Y CX(())
where (x,)) = tv: f(Xi), 2 21 1- A}.
[Y
The expected value of L. FNR is the FNR. Since L,NRR is monotone, so is the FNR. Thus, the technique in Section 2.1 pick Å by 4 that controls the FNR on new point, which guarantees:
E
(.)
FFNN n+1 (3) S a.
For evaluating the proposed procedure we pool data from several online open-source gut polyp segmentation datasets: Kvasir, Hyper-Kvasir, CVC-ColonDB, CVC-ClinicDB, and ETIS Larib.. We choose PraNet (Fan et al. 2020) control with the 781 remaining validation data points. We report results with a = 0.1 in Figure The mean and standard deviation of the risk over 1000 trials are 0.0987 and 0.01 14, respectively.
base model f and used = 1000, and evaluated risk
5

---

Published as a conference paper at ICLR 2024
soup bowl dish frol pot
Loafer shop Seridll
poncho
condle
dish
dloak clock
tamp jack-d Hanterm
nuteinent
stew
SOLB
meat loaf dish burrito
table lamp lamp candle
spollight lamp cendle
soup bowl dish consomme
por 935
sot po0
20y
density
10-1
10-2
0.047
0.048
0.049
0,050
0.053
0.052
0.053
risk
height
Figure 3: Control of graph distance on hierarchical ImageNet. The top figure shows examples of our procedure with correct classes in black, false positives in blue, and false negatives in red. The bottom plots report our minimum hicrarchical distance loss and set size over 1000 independent random data splits. The dashed gray line marks a.
3.2 FNR CONTROL IN MULTILABEL CLASSIFICATION
In the multilabel classification setting, our input X; is an image and our label is a set of classes Y, C (1,.... K} for some number of classes K. Using multiclass classification model X -
[0, we form prediction sets and calculate the number of false positives exactly as in (7). By
Theorem I picking À as in (4) again yields the FNR-control guarantee in (8). We evaluate on the Microsoft Common Common Objects Context (MS COCO) dataset (Lin al. 2014), large-scale 80-class multiclass classification task commonly used in computer vision. We choose a TResNet (Ridnik) al. 2020) as our model and used n 4000, and evaluated risk control with 1000 validation data points. We report results with a = 0.1 in Figure 2 The mean and standard deviation of the risk over 1000 trials are 0.0996 and 0.0052, respectively. The results indicate that the risk is almost exactly controlled, the spread is not too wide, and the set sizes are reasonable, not overly inflated.
3.3 CONTROL OF GRAPH DISTANCE IN HIERARCHICAL IMAGE CLASSIFICATION
In the K-class hierarchical classification setting, input X; is image and label is leaf node Y; E {1.... K) on tree with nodes V and edges E. Using a single-class classification model
f X + A we calibrate loss in graph distance between the interior node select and the closest ancestor of the true class. For any I E X, let ((x) = arg max f(x)2 be the class with the highest estimated probability. Further, let d : V V Z be the function that returns the length of the shortest path between two nodes, let A: V + 2 be the function that returns the ancestors of its argument, and let P V + 2 be the function that returns the of leaf nodes that are descendants of its argument. We also let 2(v,2)) EkEP(e) (x)) be the sum of scores of leaves descended from v. Further, define hierarchical distance
dn(v,, u) inf {d(a, w)}. aEA(v)
For a set of nodes CA 2V, we then define the set-valued loss L Graph (A) inf (dH(v,))}/D, where Cx(x) sXC)(X.)
n
P(a).
(aEAA((<>)) : -(a,))2-11 A}
This loss returns zero if y is a child of any element in Cx. and otherwise returns the minimum distance between any element of C2 and any ancestor of y, scaled by the depth D. Thus, it is a monotone loss function and can be controlled by choosing as in (4h to achieve the guarantee
L n+1
Graph
(3) S a.

---

Published as a conference paper at ICLR 2024
[1 august 1966, 1965, 11 january 2006, ...)
Answors 1965,
F1
(obort jesus, david, kelth grecn....
who sold the story of the prodiga son?
WIKIPEDIA The Foe Eagalgule
Knowers = (Jesus Christ
F1
denrety
10
0.26 0.28
0.30 0.32
0.34
20 22
24 26
28
30
risk
set size
Figure 4: Fi-score control on Natural Questions. with fully correct answers in green, partially correct answers in blue, and false positives in gray. Note that answers that are technically correct may still be down-graded if they do not match the reference. We treat this part of the randomness in the task. The bottom plots report the F1 risk and average set size over 1000 independent random data splits. The dashed gray line marks a.
The top figure shows examples of our procedure
We use the ImageNet dataset (Dng et al. 2009), which comes with an existing label hierarchy, WordNet, of maximum depth D
We choose a ResNet152 (He et al. 2016) for and 0.05 in Figure
30000, and evaluate risk with the remaining 20000. We report results with The mean and standard deviation of the risk over 1000 trials are 0.0499 and 0.001 I, respectively. The results indicate that the risk is almost exactly controlled, and that the adaptively chosen resolution of the prediction appropriately encodes the model uncertainty (it is almost always a leaf node).
3.4 F1-SCORE CONTROL IN OPEN-DOMAN QUESTION ANSWERING
In the open-domain question answering setting, our input X; is a question and our label Y; is a set of (possibly non-unique) correct answers. For example, the input
X "Where was Barack Obama Born?"
could have the answer set
Yn+1 = ("Hawaii", "Honolulu, Hawaii", "Kapo" olani Medical Center")
Formally, here we treat all questions and answers as being composed of sequences (up to size m) of tokens in a vocabulary V-i.e., assuming valid answers, we have X; Z and Y; E z², where Z : ym Using open-domain question answering model that individually scores candidate output answers f Z x + R, we calibrate the best bakedd F1-score of the prediction set, taken over all pairs of predictions and answers:
(A) 1 - max (Fi(a, c ax E.). where Cx = (v E V f(X,, 2 A}. We define the Fi-score following popular QA evaluation metries (Rajpurkar et al. 2016), where we treat predictions and ground truth answers as bags of tokens and compute the geometric average of their precision and recall (while ignoring punctuation and articles {"a"", "an", "the")). Since L as defined in this way, is monotone and upper bounded by 1, it can be controlled by choosing Î as in Section 2.1 to achieve the following guarantee:
E
L <a
We use the Natural Questions (NQ) dataset (K (kiiakkwwkki et al. 2019), a popular open-domain ques- tion answering baseline, to evaluate our method. We use the splits distributed as part of the Dense Passage Retrieval (DPR) package (Karpukhin et al. 2020). Our base model the DPR Retriever- Reader model (Karpukhin et al. 2020). which retrieves passages from Wikipedia that might contain the answer to the given query, and then uses reader model extract text sub-spans from the retrieved passages that serve as candidate answers. Instead of enumerating all possible answers to a given question, we retrieve the top several hundred candidate answers, extracted from the top 100 passages. We use 2500 calibration points, and evaluate risk control with the remaining 1110. We use a = 0.3 (chosen empirically as the lowest F1 score which reliably results in approximately correct answers by manual validation) in Figure 4 The mean and standard deviation of the risk over 1000 trials are 0.2996 and 0.0150, respectively. The results indicate that the risk is almost exactly controlled, and that the reasonably sized, scaling appropriately with question difficulty.
7

---

Published as a conference paper at ICLR 2024
4 EXTENSIONS
We now discuss several extensions of conformal risk control to different settings and risks.
4.1 RISK CONTROL UNDER DISTRIBUTIONAL SHIFT
Under a distribution shift, the goal in (3) can be redefined as
Y))..... Puatn (X==1,X++1)> T1ut1 (2) aa
(9)
Assuming that Puest is absolutely continuous with respect to Pirain and defining w(x,y) dPrent (a-31) the weighted objective ( can rewritten as d Pirnin (ar.3)
M1...) (1. (on-11 Yutt ( Lm+ (2) S a.
(10)
When w is known and bounded, we can apply our procedure on the loss function In+ 1 (A) = w(Xn+= 1 mm- (A), which is non-decreasing, bounded, and right-continuous in whenever La+1 is. Thus, Theorem guarantees that the resulting Ả satisfies (TO). For example, in the covariate shift setting, w(X Yata) w(Xn+1) - (Xpe)(X
d'Penin (X. In this case, we can
achieve risk control even when is unbounded. In particular, assuming Le E [0, B), for any potential value r of the covariate, we define
A(x) inf x E w0 (X,)) + w(x) saj.
(nf(A
E w(1)) + w(x)
Proposition 3. In the setting Theorem ! with N as above,
(X....)...... (x...... (K [Z
(A(Xx+1))) I a
This is an exact generalization of the procedure of Tibshirani al. (2019) beyond indicator losses. As proposed therein, when unlabeled data in the test domain is available, w can be estimated by the probabilistic classification algorithm; this gives good practical results (see also our experiment in Appendix D). For arbitrary distribution shifts, we give total variation bound analogous to that of (Barber et al. (202)) for independent data (see their Section 4.1), though the proof different.
Here we will use the notation Z, = (X,, Yi). and A(ZI...., ,nn to refer to that chosen in (4)
Proposition 4. Let Z conditions in Theorem
(Z....., Zn+1) be sequence of random variables. Then, under the TVZZ, Zn+1). If further the assumptions
E
(A) a < BE
of Theorem hold, E
(X)
2 a--a B
2 n+I + E TV(Z;, ZV+1))
4.2 QUANTILE RISK CONTROL
Snell et al. (2022) generalizes Bates al. (2021) to control the quantile of a monotone loss function conditional (X,, Y.) with probability 1 over the calibration dataset for any user-specified tolerance parameter In some applications, may sufficient to control the unconditional quantile
of the loss function, which alleviates the burden of the user to choose the tolerance parameter 5.
For any random variable X, let Quantile (X) inf P(X S x) 2 3). Analogous to (3), want to find based on (X,, Y such that
Quantile; (Inti(As)) sa.
(11)
By definition, Quantile, (Im+1(As)) S a = E I1 ((mmmniAA))
> a consequence, quantile risk control is equivalent to expected risk control Í,()) (L.(A) > Let
< - B. As with loss function
Let Åg inf { A + E 1 (L.(A) a)
D+T I 1 8
Proposition 5. the setting of Theorem 1 with À as above, II achieved.
is unclear whether the wider class of quantile-ber based risks considered by Snell et al. (002) (e.g.
the CVaR) can be controlled unconditionally.
8

---

Published as a conference paper at ICLR 2024
4.3 CONTROLLING MULTIPLE RISKS
Let () be family of loss functions indexed by 7 I for some domain that may have infinitely many elements. A researcher may want to control ([);():)) at level a(7). Equivalently, need to find an À based on (X,, Y) such that
LAddn) a()) - 1.
sup E
(12)
Though the above worst-case risk is not an expectation, it can still be controlled. Towards this end, we define = supyer Xy, where Xy inf : n+1 Σ.= L,(>;)) +
B S a(>)).
Proposition 6. In the setting of Theorem / with Å as above, 12
satisfied.
4.4 ADVERSARIAL RISKS
We next show how control risks defined by adversarial perturbations. We adopt the same notation
as Section 4.3 Bates et al. (2021) (Section 6.3) discusses the adversarial risk where I parametrizes
a class of perturbations researcher may want to find an À based on (X,, Xi) such that
Xx+1, e-g.. [x(>; L(X; + 7, Yi) and {r lhls S c). A
Ejsupyer L((A:)) S a.
(13)
This can be recast as conformal risk control problem by taking L.(A) supyer L,(X:)). Then, the following choice of X leads to risk control: Ā = inf(X: n+1 Ei= La(A) + n+1 S a).
Proposition 7. In the setting of Theorem ! with \ as above, 13 is satisfied.
4.5 U-RISK CONTROL
For ranking and metric learning. Bates al. (2021) considered loss functions that depend on two test points. In general, for any and subset S C {I...., n + k} with |S] = k. let (A) loss function. Our goal is to find ÅS based on (X,, Y such that
[Lintt,..ntt)) (2x) Sa. a.
(14)
We call the LHS U-risk since, for any fixed dς, it the expectation of an order-k U-statistic. natural extension, we can define
d inf {x:
kin!! (x++))) Esc(L....JISII= Ls(A) B
(n!)²
sa)
(15)
(n+k))(m=a))
Again, we detine du = Amax when the RHS is empty. Then we can prove the following result. Proposition 8. Assume that (2( is non-increasing X, right-continuous, and Ls (Amax) I
a, supA Ls(A) & < oo almost surely. Then (14) is achieved with X as above.
CONCLUSION
This generalization of conformal prediction broadens its scope to new applications, as shown in Section 3 Still, two primary limitations of our technique remain: firstly, the requirement monotone loss is difficult to lift. Secondly, extensions to exchaannaeee data require knowledge about the form of the shift. This issue affects most statistical methods, including standard conformal prediction, and ours is different in this regard. Finally, the mathematical tools developed in Sections[]ann may be of independent technical interest, as they provide a new, and more general, language for studying conformal prediction, along with new results about its validity.
REPRODUCIBILITY STATEMENT
Code to reproduce our examples is available at http://ihhuuu.c.caamggggeee httpss//ggShhbbbcommmm/ngooooooo
con fonformmll - -risk

---

Published as a conference paper at ICLR 2024
REFERENCES
Anastasios N Angelopoulos, Stephen Bates, Emmanuel Candès, Michael Jordan, and Lihua Lei. Learn then Test: Calibrating predictive algorithms to achieve risk control. arXiv preprint arXiv:21 2021a.
Anastasios N Angelopoulos, Amit Pal Kohli, Stephen Bates, Michael Jordan, Jitendra Malik, Thayer Alshaabi, Srigokul Upadhyayula, and Yaniv Romano. Image-to-image regression with distribution-free uncertainty quantification and applications in imaging. In International Confer- ence on Machine Learning, pp. 717-730. PMLR, 2022a.
Anastasios N Angelopoulos, Karl Krauth, Stephen Bates, Yixin Wang, and Michael I Jor- dan. Recommendation systems with distribution-free reliability guarantees. arXiv preprint arXiv:2207.01669, 2022b.
Anastasios Nikolas Angelopoulos and Stephen Bates. A gentle introduction to conformal predic- tion and distribution-fii uncertainty quantification, 2021. URL htttttttp//aastt wwtp
0107.07511
Anastasios Nikolas Angelopoulos, Stephen Bates, Jitendra Malik, and Michael I Jordan. Uncertainty sets for image classifiers using conformal prediction. In International Conference on Learning Representations (ICLR), 2021b. URL htttpa://operrevieee wtw.:pperrrreeeeeeeooooorrnnnan (inerumi
DDM9
Martin Anthony and John Shawe- Taylor. A result of vapnik with applications. Discrete Applied Mathematics, 47(3):207-217, 1993.
Rina Barber, Emmanuel Candès, Aaditya Ramdas, and Ryan Tibshirani. The limits of distribution- free conditional predictive inference. Information and Inference, 10, 08 2020. doi: 10.1093/ imaiai/iaaa017.
Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. arXiv preprint arXiv:2202.134/5 2022.
Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I. Jordan. Distribution-free, risk-controlling prediction sets. Journal of the ACM, 68(6), September 2021. doi: 10. 10.1145/3478535 URL hhtttt::///dl
ISSN 0004-5411.
Vidmantas Bentkus. On Hoeffding's inequalities. The Annals of Probability, 32(2):1650 1673, 2004. doi: 10. 101121400011779900000000060
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 248-244222 225-2555222 2009. doi:
Deng-Pine Fan, Ge-Peng Ji, Tao Zhou, Geng Chen, Huazhu Fu, Jianbing Shen, and Ling Shao. Pranet: Parallel reverse attention network for polyp segmentation. In International Conference Medical Image Computing and Computer-Assisted Intervention. pp. 263 273-273, 2020. doi: 26.
10.
Clara Fannjiang, Stephen Bates, Anastasios N Angelopoulos, Jennifer Listgarten, and Michael I Jordan. Conformal prediction for the design problem. arXiv preprins arXiv:2202.03613, 2022.
Adam Fisch, Tal Schuster, Tommi S. Jaakkola, and Regina Barzilay. Efficient conformal predic- tion via cascaded inference with expanded admission. In International Conference on Learning Representations, 2021. URL pecttp::/oppnreeee
net fttt:/forrmmiiidtttoooeeee
Adam Fisch, Tal Schuster, Tommi Jaakkola, and Regina Barzilay. Conformal prediction sets with limited false positives. arXiv preprint arXiv:2202.076600 2022.
Alexandre Froda. Sur la distribution des propriétés de voisinage des fonctions de variables réelles. Bulletin mathématique de la Société Roumaine des Sciences, 32(2):105-20) 1931.
10

---

Published as a conference paper at ICLR 2024
Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. Ad- vances Neural Information Processing Systems, 34:1660-1677 2021.
Leying Guan. Conformal prediction with localization. arXiv: 190.08890 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi: 10.11099CPP.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, November 2020. Association for Computational Linguisties.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-4666 2019.
Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J. Tibshirani. and Larry Wasserman. Distribution- free predictive inference for regression. Journal of the American Statistical Association, 113 1594)1109-1 2018.
Tsung Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft COCO: Common objects context. In European conference on computer vision, pp. 740-755. Springer, 2014. doi: 10. 100009978-33311-10022 48.
Harris Papadopoulos, Kostas Proedrou, Vladimir Vovk, and Alex Gammerman. Inductive confi- dence machines for regression. In Machine Learning: European Conference on Machine Learn- ing. pp. 345-356, 2002. doi: https://doi.org/10 wttp:///r//.1.10/3355/////0 10000005540-66555555 1.29.
Sangdon Park, Osbert Bastani, Nikolai Matni, and Insup Lee. PAC confidence sets for deep neural networks via calibrated prediction. In International Conference on Learning Representations (ICLR), 2020. UtLpattps://ppeeeeeeee
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, November 2016.
Tal Ridnik, Hussam Lawen, Asaf Noy, and Itamar Friedman. TResNet: High performance GPU- dedicated architecture. arXiv:2003. 13030000...3,, 2020.
Yaniv Romano, Evan Patterson, and Emmanuel Candès. gression. In Advances Neural Information Processing Systems, volume 32, pp. 3553. 2019. URL
Conformalized quantile
cttpeurpss..eppppp.eeeeee wttpc//pppppe/1///11eeee. Paper. Pawebb--apppppddp...
Yaniv Romano, Matteo Sesia, and Emmanuel Candès. Classification with valid and adaptive coverage. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 3581-3591. Curran Asso- ciates, Inc., 2020. URL https: http::::/pproccedigg...ees neuris 244edd7e85dcB 602676 (0000//615dd/05555555------- http://s....rrrppsscpcapp//////a cc//papprr/220//flee
Swami Sankaranarayanan,
Anastasios Angelopoulos, Stephen Bates, Yaniv Romano, and Phillip Isola. Semantic uncertainty intervals for disentangled latent spaces. arXiv preprint 2.Xiv:2207.10074 2022.
Tal Schuster, Adam Fisch, Tommi Jaakkola, and Regina Barzilay. Consistent accelerated inference via confident adaptive transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2021.

---

Published as a conference paper at ICLR 2024
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani. Dara Bahri, Vinh Q Tran. Yi Tay, and Donald Metzler. Confident adaptive language modeling. arXiv preprint arXiv:220.0076./ 2022.
Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(Mar)3371-42111 2008.
Jake C Snell, Thomas Zollo. Zhun Deng, Toniann Pitassi, and Richard Zemel. Quantile risk con- trol: A flexible framework for bounding the probability of high-loss predictions. arXiv preprint arXiv:2212.13622, 2022.
Ryan Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal predic- tion under covariate shift. In Advances in Neural Information Processing Systems, volume 32, pp. 2530-2540. 2019. URL http:::::pppoceeeeteeee nwwriis.s....ipspp httpsscccppppppeeee cc/papper/2119//ile) 8fb2lee7a222 Sttaaaeee222212222a555a571522222
ppdf
Aad W Van der Vaart. Asymptotic statistics, volume Cambridge university press, 2000.
Vladimir Vovk. Conditional validity of inductive conformal predictors. In Proceedings of the
Asian Conference on Machine Learning, volume 25, pp. 475-490, 2012. URL http://
proceedings wttp./p.ee//////ooooos.///
Vladimir Vovk, Alexander Gammerman, and Craig Saunders. Machine-learning applications of algorithmic randomness. In International Conference on Machine Learning, pp. 444-453, 1999.
Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in Random World. Springer, New York, NY, USA, 2005.
Ian Waudhy-Smith and Aaditya Ramdas. Variance-adaptive confidence sequences by betting. arXiv preprinttt 1XXi:20010006686 2020.
A CONFORMAL PREDICTION REDUCES TO RISK CONTROL
Conformal prediction be thought of controlling the expectation of indicator loss function. Recall that the risk upper bound (2) specializes to the conformal coverage guarantee in ( when the loss function is the indicator of miscoverage event. The conformal risk control procedure specializes to conformal prediction under this loss function as well. However, the risk lower bound in Theorem has a slightly worse constant than the usual conformal guarantee. We now describe these correspondences.
First, show the equivalence of the algorithms. In conformal prediction, have conformal scores sX,,, Y2) for some score function $ X x y + R. Based on this score function, we create prediction sets for the test point Xn+1 as C₁ (Xx+1) = {y : s(Xn+1, y) I Å}. where Á is the conformal quantile, a parameter that set based on the calibration data. particular, conformal prediction chooses À to the
[(n + 1)(1 - a)//n sample quantile of {s(X,, Y:) formulate this in the language of risk control, we consider miscoverage loss L,
(A) LV
Y, $ (x.))
(s,,,, Y) > X}. Direct calculation of from procedure to conformal prediction:
then shows the equivalence of the proposed
Sa 1
= inf A: n
(6
sa}=
a #1
(n + 1) a)1
inf
1 (s(,,, Y2) 1}
n
Ti
conformal prediction algorithu
Next, we discuss how the risk lower bound relates to its conformal prediction equivalent. In the setting of conformal prediction, Lei al. (2018) proves that P(Y,+1 & C (Xn+1) a 1/(n + 1) when the conformal score function follows continuous distribution. Theorem 2 recovers this
guarantee with slightly worse constant: P(Y.41 & (x(x)))) 2 a-2/(no 2/nn+nnn 1). First, note that
12

---

Published as a conference paper at ICLR 2024
assumption in Theorem about the distribution of discontinuities specializes to the continuity of the score function when the miscoverage loss is used: P (L, Cvv ..) -
> = + P(s(X,, Y))
X) = O. However, the bound for the conformal case is better than the bound for the general case in Theorem by factor of two, which cannot be improved according to Proposition This difference is an interesting oddity of the binary loss, but is not practically important.
B COMPARISON WITH RCPS/LTT
The setting of conformal risk control resembles the setting of previous works in high-probability risk control Bates et al. 2021); Angelopoulos et al. (2021a) (RCPS/LTT); however, the algorithms, guarantees, and proof strategies are entirely different.
To summarize, the main differences are as follows:
1. Conformal risk control is substantially more sample-efficient than RCPS/LTT. On the scale of the risk, RCPS/LTT are far more conservative, converging a vn rate, while conformal risk control converges rate.
2. RCPS/LTT are high probability bounds of the form
(X,, Y) S <a) 2 1
Conformal risk control gives an expectation bound the form
C(
a.
The former bound provides high-probability control over the sampling of the calibration data, which can be attractive if one needs to be below with high probability, as opposed centered a in the latter. However, the former bound is more difficult interpret for nonstatisticians and requires the selection of two parameters-a a and 5-which can introduce complexity.
3. RCPS/LTT substantially more complicated procedures than conformal risk control- especially LTT, which requires careful discretization and bespoke multiple testing correc- tion. This careful construction is required because the LTT procedure allows for the high- probability control of non non-monotone risks, while conformal risk control only applies to monotone ones.
4. The theory of conformal risk control is basic advancement the theory of conformal prediction and exchangeability- based arguments, extending them to broader mathemati- cal setting. Mcanwhile, LTT/RCPS use well-established concentration bounds, which are mathematically unrelated to conformal prediction.
To underscore these points, we perform careful evaluation against these procedures on the polyp segmentation dataset in Figure 5] We compare against two versions of LTT/RCPS: Baseline has 5 0.5, Baseline 2 integrates the tail bound LTT/RCPS to achieve bound expectation. The former strategy is not statistically valid for the goal of expectation control, and the latter strategy is essentially only possible using fixed-width bounds such as Hoeffding's inequality, and is described in the below Appendix B.I We use the same risk level a 0.1 for all procedures and make plots with n = (25, 50, 100, 200, 300).
The takeaways are as follows: The statistical efficiency gains of conformal risk control are massive. On the scale of the risk gap, [B[ - al, conformal risk control is 50% closer to the desired risk than Baseline 1, and orders magnitude closer than Baseline 2. This expected given the theo- retical convergence rate is quadratically faster than both approaches. The computational efficiency of conformal risk control substantially better than all baselines. For example, 14x faster than Baseline I and 19% faster than Baseline 2. It is also substantially easier to implement than Baseline 1, only requiring lines code. Bascline 2 is equally easy to implement, although far less flexible.
B.1 RECOVERING EXPECTATION BOUNDS FROM RCPS
We introduce a technique for deriving bounds in expectation using RCPS Bates et al. 202TVLTTT Angelopoulos al. (021a). This technique was described in the original papers;
13

---

a. Published as a conference paper at ICLR 2024
Insegrase Tall Bound (Hoefiding)
0.10
RCPS =0.51
20
15
0.04
10
0.02
Integraze
100
150
200
250
200
250
Figure 5: Comparison of RCPS/LTT with conformal risk control on the polyp dataset.
we only derive it here for the purpose of the above comparison (i.e., to show that this approach should essentially never used for monotone losses).
Consider a pointwise upper-bound on the empirical risk, as constructed RCPS Bates et al. (2021)
P(R(A) (A) > <) < 6(c,A),
for all [0, B). The next proposition says that if the bound has equal width everywhere, like in the case of Hoeffding's s inequality, then it can be integrated to control the risk. Unfortunately. this strategy is not possible for general concentration inequalities, such that of [Bentkus) (2004) or WWaddy--mmie & Ramdas (2020). as it would require knowing 8(c, X*) (for an oracle X defined in the proof below).
Proposition B.1. Let [2(A) be right-continuous, monotone, i..d functions in 00, B) satisfying
P(J(L,, X) > 0) = Furthermore, define 8+ (+)))= sup,, 8(c, X) and
A == inf X: R(A)) 5 aa)
where o' satisfies a" S a--Jo -a- BB 8t (e)de. Then,
R(Å) S a.
Proof. Let x- = inf R(A) a + e). By monotonicity and the definition of Å,
x2x x Rx(A** S a
But also, by the bounded jump assumption, R(X*) a Therefore, on the event Á > Xr, have that
R(**) (X*) 2 - - cc=c.
(16)
Since the event 16) happens with probability at most 8+ (e), we have that P(1 2 X²) < (+),
so
PP(R(A) 2 +c) 8555500 (e).
Finally, we have that
1 [[((A)]
P(((()) > r)dr
$ a +
((R(A) > a- + tedde
B-a
S a +
8 (c)dee
This can be applied Hoeffding's inequality, giving the aforementioned Baseline 2.
14

---

Published as a conference paper at ICLR 2024
Corollary 2. In the setting of Proposition B.1 let a- = a
yaB enf( v
Then, R(\) S a.
2v2nn
Proof. Applying Hoeftding's inequality have 8 (+) e 2ne2 and then
de a / 2n de
VT B erl( (V2n//////// B) a
2v2n
as required.
C MONOTONIZING NON-MONOTONE RISKS
We next show that the proposed algorithm leads to asymptotic risk control for non-monotone risk functions when applied monotonized version of the empirical risk. We set the monotonized empirical risk to be
(A) sup (t), 12A
then define
= inf A: R7 (A) S a)
Theorem C.1. Let the Lę(A) be right-continuous, Liddd, bounded (both above and below) functions satisfying Then,
lim E [ (27)
a.
n-000
Theorem particular, taking
C.1 implies that analogous procedure also controls the risk asymptotically. In
Àt =t=inn inf (A. (A)
B n n a
also results in asymptotic risk control (to see this, plug Xt into Theorem C.1 and see that the risk level bounded above by ) Note that in the case of a monotone loss function, Ät = Å.
However, the counterexample Proposition 2 does not apply to λt, and currently unknown whether this procedure does or does not provide finite-sample risk control.
D EXPERIMENTS ON COVARIATE SHIFT
To validate the covariate shift extension from Section 4 regression under covariate shift. The model is follows:
we perform an experiment on a synthetic
X; (NO, 1)
Y; 2X, + (0000,
Xest: 2(0,2)
Yeest Dest + (O, 0.5 (est))
The model used for prediction was standard linear regression pre-trained on 1000 data points independent and identically distributed with the calibration data. We estimated the likelihood ratio using a logistic regression model by solving classification problem to classify between all the available trainingccalibaation covariates and batch of 1000 unlabeled test datapoints from the shifted distribution. The probits from the logistic regression were then transformed likelihood ratios via We took 100 and 0.1, controlling the clipped projective
the function h(x) x/1
distance of y onto the set C:
Ely, C* min(proj B),
where = 2. The set construction is the standard
Cx(x) (x) =
Figure shows results.
15

---

Published as a conference paper at ICLR 2024
14
method
standard
12
covariate shift (estimated)
Count
6-
2--
0.20
risk
Figure 6: Risk control results synthetic covariate shift dataset. Running standard conformal risk control does not control the risk, while running the covariate shift algorithm from Section does-even though the likelihood ratios are estimated by a logistic regression model. We also com- puted the risk with the true likelihood ratios, and it overlaps entirely with the histogram for estimated likelihood ratios, so we do show here.
E PROOFS
The proof of Theorem uses the following lemma on the approximate continuity of the empirical risk.
Lemma (Jump Lemma), In the setling Theorem ie.,
2
any jumps the empirical risk are bounded,
8.8. B
sup
J
R
X
Proof of Jump Lemma, Lemmal the jump is
By boundedness, the maximum contribution of any single point
so
B n
:YE J(Rn: 2)
YA: J(L,, and J(L,, A) for some + j.
Call D. = {A J(Li, A) > 0} the sets of discontinuities in L.. Since L, is bounded monotone, D, has countably many points. The union bound then implies that
B Ti
P aA: JBB,, 1)
Σ
P(D.nD, #0)
Rewriting each term of the right-hand side using tower property and law of total probability gives
IP (D, + 0) = E (P,
0 D. +
D,)]
Σ (AAED; D.) = E Σ P(AED)) AED
<
AED,
Where the second inequality because the union of the events À E Dj is the entire sample space, but they disjoint, and the third equality is due the independence between D, and Dj- Rewriting in terms of the jump function and applying the assumption P (J A) > 0) = 0.
E Σ P(A E D.)
= E Σ PJJ X) > 0) = 0.
AED
AED
16

---

Published as a conference paper at ICLR 2024
Chaining the above inequalities yields IP
(E) J 2) 0) 00,80 0,
XE J(Rn
B = -.
Proof of Theorem 2 If Li (Amax) 2 a- 23/ (n+ 1), then + + (Å)]
(--2B/n+11,, Through-
out the rest of the proof, we assume that Li (Amax) < a - -2BB 1 + 1). Define the quantity
B
x== = inf A: (x+ n - 0
Since Li(Amax) <a- aa2BB/(n
(n + +) <a a B/(n + 1), X" exists almost surely. Deterministically. n+1 R₂(2) I Re+1(M) (A), which yields A s X². Again since (,(A) is non-increasing in X,
E [[ (X") s E [ (2)
By exchangeability and the fact that \" is symmetric function LI...., Latt. Ln+1 (x*) =E [x*)
For the remainder of the proof we focus on lower-bounding Rut1 (X'). We begin with the following identity:
= 1
+ n+l
B
(Rati(8")
+ n 1
B a).
Rearranging the identity,
R1+1 (X")
n+1 + (Rntil"")
B
+ n + 1 a).
B
n
Using the Jump Lemma bound (Rm+1 (X')) + n+1
B
-a)
below by m+ gives
B
2B
Ru+i (X') 2
n + 1
Finally, chaining together the above inequalities,
( 2 E Rn=i(("))
2B 2a n=
Proof of Proposition 7 Without loss generality, assume B = 1. Fix any > 0. Consider the following loss functions, which satisfy the conditions in Theorem
A E [0, 2z) A 12 Wt) else
L((A)
i.d. ~
where 1s E M. the Z, i.ddd Uniform/ 0.5). the W, k+1-6' Then, by the definition of Å, we know
i.dd Uniform(0.3, for i E {1, + 1] and
R (2) <
(17)
If n > ++ R(A) Z k+1
whenever X 5 2. Thus, we must have > 2. Since le is an
integer and by (17). we know that {i E (1...., m} (a() > 03 s ((k: 1)(k c'/kk S k. This immediately implies that
A W W(n-++1),
17

---

Published as a conference paper at ICLR 2024
where () denotes the j-th order statistic. Notice that for all À 1.
h
R(A) = (A)]
k+1 2(1 2),
so R())
I+1 2(1 W(n. R+))) Let U())) be the k-th smallest order statistic of ii.d. uniform
random variables on (0, 1). Then, by symmetry and rescaling, 2(1 W (nn-++1) a U(e)
k 3 k +
R(*)
U(a)
where denotes the stochastic dominance. well-known that U(K) Beta(k, n + k) and hence
k
B[R(A)] < k+1
n+1
Thus,
- B [R())
h +
k
1
(2
c)k +
+1
(n 1) 1)
12 n
k +
For any given > eet = 2/2 and k = - 11. Then
(2 d')l + 1 è'
2-6
k+1
implying that
[[((i)]
2 2-6 2
+
Proof of Proposition Without loss of generality, assume B = 1. Assume Å takes values in [o, 1] and a E (1/(n + 1),1). 1). Let p E 1), be any positive integer, and (((A) i.idd right- continuous piecewise constant (random) functions with
L.(N/N) 0, L.10/),, Li((N - 1)/N)) i.d.d Ber(p).
By definition, Â is independent of Ln41- Thus, for any j = 0, 0,.,...,N- - 1,
{ 2 3/N} Ber(p), {Lnti(2) i 13 80.
Then,
E Ln+1 (2) = =p. =p-P(A+))
Note that
11
min 1
Σ
n
L(j/N) S a 1
Jeto....NNI.
n + 1
n+1
Since 1/(n+ 1).
P(Å # 1) = - P(A = 1) = 1-P P
for all j. we have
Σ
L.GiNN) > n+1
n2
1=1
N
1
Σ
11
p* (1-11-77 (1-
k
k= [m++1)a]
BinoCDF n, [(n + 1)aa] - 1)) N
1
As a result,
[(n E Ln+ (8)) = p 1 1- BinoCDF [(a +(n+ 1)ae - -1))**
18

---

Published as a conference paper at ICLR 2024
Now let N be sufficiently large such that
(1 (1, [(n + 1)a] -1)) > p.
Then
In+1 (2) p?
For any a > O, we can take close enough to 1 to render the claim false.
Proof of Theorem C.1 Define the monotonized population risk as
RT(A) sup L (t)
<2>
Note that the independence of Lx+1 and Åt implics that for all
T (At)
- E (TTT((A))
Since Rt is bounded, monotone, and one-dimensional, a generalization of the Glivenko-Cantelli Theorem given in Lemma givee that uniformly over X,
lim sup RRn (A) R(A) #
n A
As a result,
lim sup [A) Rd (A) # 0. no0
which implies that
lim ({t) R
By definition, (it) S ar almost surely and thus this directly implies
sup R (At) S a.s..
n-
Finally, since for all Ret (i1) < B, by Fatou's 's lemma,
lim E Ln+ (2t)
< limsup E Rd (At)
< E limsup RC (8h) a. n-030
2=000
Proposition 3 Let
a++ w ((.)). Ei=l (Xx))
X inf
S
Since inf (A( S a, X exists almost surely. Using the same argument as in the proof of Theorem we can show that X < \(Xn+1). Since Ln+1 (A) is non-increasing in X,
E| (Ā(X < El
Let E be the multiset of loss functions {(X1, Yi....., (Xa+1, Yxt1).. Then X is function of E, or, equivalently, X is a constant conditional on E. Lemma 3 of Tibshirani et al. (201)) implies that
w w(X)) D w(X,)
n+1
w w(X;) SL.
Kntt,Yytil Y
)(x,))
Ln+1 E
w(X,)
j=1
where 8, denotes the Dirac measure Together with the hight-continun of Li, the above result implies
B
[L In+ (8)
2i=1 w w(X))L,(A') S a.
Σ w(X))
The proof is then completed by the law of total expectation.
19

---

Published as a conference paper at ICLR 2024
Lemma 2 SelvenkkoCaantlll for Monotone Functions), For ii.dd, right-continuous, monotone loss functions Li...., Lns we have that
lim sup [R R(A)) _ 0. 7-000
We generalize the proof of the Glivenko-Cantelli Theorem (see, e.g., Van der Vaart Let the notation R(²-) denote the limit of R from the left-hand side approaching X. By the strong law of large numbers, (A) + R(A) and (A-) 3 R(A-) for all A. Fix <> O. It is well-known that monotone function cannot have more than countable number of discontinuities (Froda 1931). Thus, there exists a finite partition A(e) = {-oc I X I ... S R(A--1) E. (There can only be finite number of jumps greater
Lemma 2000
A S )o), such that R(,,- than in size, and these can be included in M(e))) The uniform convergence on the finite set A(e) is immediate by the law of large numbers; thus, we have that lim SUPxEA(r) [A) R(A) aut 0.
For any À E Ac). suppose E 11)) Then Rr (A) - R()) I (2)) R(Aj-1) < Rm, (A₂) - R(>;) + Similarly, R., (A) - R(A)= 2 (
R2 ()--1) ---1))-R)-j1
Combining two pieces, we conclude that supx (A)
(A) - R((A) - su))EA(e) RLn (A) R(A) + e.
As result, lima- Supx (A) R(A) S c. Taking e + completes the proof.
Proposition f Let
Define the vector Z' (Z1, .... Un Zn+1), where Z ii.d L(Zn+1) for all [n].
a
TV 4).
By sublinearity,
TV(Z Se.
(18)
It is a standard fact that
18 implies
sup [[((Z)] B[f(Z')| SeA
where F, = [ 2 + 0]] Let e: Z x A + [0, B] be a bounded loss function. Furthermore,
let g(2) E(zn+1: 7(21, 2n)). Since g Z) E B).
Big(Z) E[g(Z')]1 < Bc.
Furthermore, since Zi..... Za+1 are exchangeable, can apply Theorems and 2 to B[g(Z")), recovering
2B
S Elg(Zz)) aa
n +1
A final step of triangle inequality implies the result:
2B
Be Be <a Be.
n+1
Proposition 5 It is left to prove that (x()) satisfies the conditions of Theorem It is clear that (,()) < and (A) is non-increasing in À when L₂(A) is. Since Lę(A) is non-increasing and right-continuous, for any sequence Xm + A,,
L,(Am) 1 L,(A) - ((Ls(Am) > af - (( > a}. Thus, 14(A) is right-continuous. Finally, Tx(Amaxx) S a implies (x(2max) = 0 S -
Proposition o Examining the form of Å, for each 7 E P, we have
E [(A))) < [(), s a(r).
[(x)) a(>)
Thus, dividing both sides by a()) and taking the supremum, we get that sup,er E and the worst-case risk is controlled.
< 1,
20

---

Published as a conference paper at ICLR 2024
Because (, 7) is bounded and monotone in À for all choices of y it is also true that (A) is bounded and monotone. Furthermore, the pointwise supremum of right-continuous functions is also right-continuous. Therefore, the Lę satisfy the assumptions of Thheormm
Proposition 7
Proposition g
Let
}
k!m! (n + k)!
N/2 inf A
Σ
Ls(A) S {
Since L.s (ssx) S X/ exists almost surely. Since (s(A) B, we have
k!m!
Σ (s(A)
(n k))!
k!n!
$ (n k)!
Σ
Ls(A) + B.
Σ
SC sc
kln!
k!n!
Σ
Ls(A) +
Σ
(n + k)!
(n + k)!
k!n!
Σ
Ls(A) +
(1- (1- (n + k))l(n k:)!
(n!)²
(n + k)! (C()...)),($1
Since (A) is non-increasing in X, we conclude that \/. S dn if the right-hand side of (15) is not empty; otherwise, by definition, X/- Amax Ån. Thus, X% < du almost surely. Let E be the multiset of loss functions {Ls S C [1,.... fl.....n+kk] k+ = k}. Using the same argument in the end of the proof of Theorem [ and the right-continuity of Ls, we can show that
k!m!
E Li (X/)
Σ SCH1...nt4,,1811
Ls(A) a.
(n k)!
The proof is then completed by the law of iterated expectation.
21


Reviews:
[{"review_id": "vU2BWllA6F", "reviewer": "Reviewer_tJUM", "summary": "The paper extends CP coverage guarantees to risk upper bounds. Under the assumption that the risk function is monotone, the authors derive a finite-sample and distribution-free bound on the risk expectation. Extensions and applications of the idea are also provided.", "questions": "- The algorithm's main parameter, $\\lambda$, is introduced in an example. It would be better to have a formal definition of it.\n- A motivation is mentioned at the end of page 2: \"it is generally impossible to recast risk control as coverage control\". The statement is not proven or supported by examples. \n- Why does $R_n$ carrry an index? Are $R_1, ..., R_{n-1}$ used anywehere? This seems similar to [1], where the index refers to a specific value of $\\lambda$.\n- Can $R(\\lambda)$ be interpreted as a parameterized conformity score and $\\lambda$ as the quantile of the corresponding empirical distribution?\n- [1] addresses the case of non-monotonic risks. How is this compatible with Proposition 2?\n- Does the method apply to the outputs of a regression model?\n- Are the extensions of Section 4 straightforward consequences of Theorem 1 and existing CP theory? \n\n[1]\n\"Learn then Test: Calibrating predictive algorithms to achieve risk control\".", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Extending the finite-sample and distribution-free coverage guarantees of CP to more general risk minimization problems may have a great practical impact. The experiments section contains a nice series of practical applications. I appreciate the authors included a full proof in the main text.", "weaknesses": "The relevance and novelty of the theoretical parts may be stated better in the introduction. The authors should \n- clarify why obtaining expectation-based bounds is more challenging than applying existing risk-control algorithms, e.g. the\nhypothesis testing strategy of [1], and \n- specify what is different and what is taken from other works, e.g. some of the definitions are similar to [1], where the setup is slightly different.\n\nUnder the monotonicity assumption, Theorem 1 seems to be a straightforward reformulation of the standard validity proof for CP prediction intervals. Indeed, Section 4.2 outlines a much simpler formulation of the proposed algorithm. The authors may consider moving and discussing that section before Theorem 1 and adding a practical example where Theorem 1 is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9JbTGvQThc", "reviewer": "Reviewer_TBcZ", "summary": "The paper extends conformal prediction to conformal risk control for any monotone loss function, which can be applied to different tasks more generally, such as token-level F1 scores. They also consider distribution shifts quantified by TV distribution distance and provide evaluations to validate the results.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The research question is important for many perspectives, such as trustworthiness ML.\n2. The paper is comprehensive and includes multiple perspectives besides the main result, including distribution shifts and discussions of different types of losses.", "weaknesses": "1. I suggest adding detailed discussions of differences to related work [1,2].\n\n[1] Learn then test: Calibrating predictive algorithms to achieve risk control. arXiv preprint arXiv:2110.01052, 2021\n\n[2]  Distribution-free, risk-controlling prediction sets. Journal of the ACM (JACM), 68(6):1–34, 2021\n\n2. The preview (sec 1.1) is clear, but theorem 1 is not well presented. $\\lambda_{\\text{max}}$ and $L_i$ are not defined or referred to in the context.\n\n3. It is interesting to discuss whether we can have inverse relations of theorem 1? Specifically, given a parameter $\\lambda$, can we inversely compute $\\alpha$ to control the risk of $\\lambda$?\n\nI'll raise the scores once the problems are well discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zp7qxrwqnq", "reviewer": "Reviewer_nYbV", "summary": "This work generalizes the conformal prediction guarantees for coverage to controlling the risk of general monotone (or near monotone) losses, validating the proposed method on various real-life examples.", "questions": "What are limitations and future directions? It would be helpful to discuss these in the conclusion as well.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I think this is an exciting work with many strengths\n- Method: generalize the CP method to arbitrary monotone loss functions, with extensions & modifications in distributional shift, controlling risk of multiple tasks, adversarial risks, etc.\n- Theory: provide the guarantee for the proposed method, and demonstrating the need for monotone functions (Proposition 2) and how to monotonie loss functions (Corollary 1) for the same guarantee\n- Experiment: provide extensive and useful illustration of the method in various tasks.", "weaknesses": "I think the only weaknesses lie in the experimental comparison.\n1. In all examples presented in section 3, the authors only show the satisfactory performance of the proposed method. No comparison against other baselines is provided. If this is due to a lack of existing methods for similar tasks, it would be helpful to highlight this, explain why this is the case, and include doing so as future directions.\n2. For tasks mentioned in the extensions, it would be helpful to also illustrate the effectiveness of the proposed method on a subset of them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b6iBTvRl7J", "reviewer": "Reviewer_pT4W", "summary": "The paper offers a generalization of split conformal prediction, from its standard coverage-type guarantees to producing (threshold-based) set-valued predictions which bound, in expectation, arbitrary monotonic (in the threshold selected) loss functions (i.e. risks). The guarantees are very similar to conformal prediction (up to knowing an upper bound on the risk function), and the method flexibly generalizes some other important extensions of conformal prediction, such as e.g. the distribution shift conformal modeling of Tibshirani et al.", "questions": "As stated above, I would like to see an experimental comparison to other methods offering guarantees on generalized risks; in particular Learn-then-Test.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper considers a productive generalization of conformal prediction, and easily extends the same proof technique as in vanilla conformal to handle relevant monotonic risks that are not coverage indicators. This leads to a host of both worked-out, and potential, applications to provably bounding useful risks (such as e.g. the running example of FNR) in various settings.\n\nThe useful extensions, to quantile control, multiple risks, and covariate shift among others, are also quite easy to derive from this generalized framework.\n\nThe experiments are (for the most part) cleanly formulated and executed, confirming that relevant risks are easily bounded in practice as predicted from the theory. \n\nThe paper is overall pleasantly written, and surveys all its contributions in a transparent manner.\n\nAs a result, I believe this paper would be a nice contribution to the conference, for both its theoretical soundness and simplicity as well as for its potential to be used in applications as diverse as split conformal prediction itself.", "weaknesses": "1. A relatively significant point is that, while theoretical bounds here are given in expectation, as compared to existing PAC-type guarantees on general risks developed in other recent related papers, but in practice it would still make a lot of sense to establish good empirical performance of conformal risk prediction relative to those --- and in particular, to Learn-then-Test, which appears both very tractable just as the proposed framework here, but also handles arbitrary losses, not just ones that are monotonic, and thus seems to offer better flexibility in practical settings (without having to resort to discussions of near-monotonicity like the present manuscript does). Since the displayed metrics about the conformal risk control method are agnostic to whether in-expectation or PAC-style guarantees are given, it should be easy to make this comparison quite direct, from a practical standpoint.\n\n2. As a very minor point, the experiment on F1 score control in open-domain question answering, in contrast to other experiments in this paper, seems less useful and set up in a somewhat stylized way that doesn't seem to aim to capture any of the linguistic difficulties in tackling open domain question-answering --- which makes the plotted performance graph seem not very interesting. The F1 score is computed over a rather simplistic bag-of-tokens similarity between predictions and answers, and is not justified by the authors as the risk measure of choice here; moreover, a shift to alpha = 0.3 happens with the scant explanation that this is the best empirical choice such that almost all answers are typically correct. Recognizing that this is just a toy example to support the validity of the proposed method, I would still have liked to see a somewhat deeper discussion of what monotonic risks one could/should use when dealing with natural language; this would be very useful in the context of recent developments in LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zmItawzpxd", "reviewer": "Reviewer_fQpV", "summary": "This paper extends conformal prediction to control the expected value of any monotone loss function. This type of guarantee called conformal risk control can be seen as a generalization of CP and includes CP as a particular case. Authors prove that for bounded loss, the conformal risk procedure provides a bound for any given $\\alpha$ and this bound is tight up to a $\\mathcal{O}(1/n)$ factor. The paper also explores different settings/problems such as distribution shift, quantile risk control, adversarial risk control, and expectations of U-statistics. Finally, experiments highlight the soundness of their proposals.", "questions": "Under the distribution shift setting, the weights need to be estimated. How this can be done? and what is the impact on the control of the risk when $w$ is replaced by an estimate?\n\nIs there a link between quantile risk control and training-conditional guarantee in standard conformal prediction?\n\nCan you give a real situation where we want to control risks defined by adversarial perturbations?\n\nThe assumption that $P(J(L_i, \\lambda) > 0) = 0$ appears to be the CP equivalent of the continuous score assumption. Is this true?\n\nIt is said in the paper that in \"monotonizing\" a loss will only be powerful if the loss is near-monotone. Although this seems intuitive, do you have any numerical evidence for this? (or/and a basic example).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is a clever and promising idea to generalize conformal prediction guarantees.\n\nThis problem is of particular interest to the conformal prediction community.\n\nThe paper is very well written and easy to follow. This is a real pleasure to read it.", "weaknesses": "It seems to me that the paper is more a collection of lots of theoretical results (c.f. Section 4. \"Extensions\": distribution shift, quantile risk, adversarial risk..)  than a real important contribution to a particular problem. This somewhat blurs the main contribution of the paper, which might have benefited from presenting only the risk control in a standard setting, but with a much more in-depth theoretical analysis.\n\nThe experimental results are not analyzed at all. In addition, the extensions in section 4 have not been the subject of any experiments.\n\nNo limits are indicated in the conclusion. In my opinion, comments on the limitations of the proposed approach are lacking.\n\nMinor:\n\nThe proof of Theorem 1 should be in the appendix with all the other proofs.\n\n\"In this case, conformal risk control finds...the expected value is...\" , hat on $\\lambda$ is missing in the right part of the equality.\n\nTypos: Examining (??),", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9H7we1noff", "reviewer": "Reviewer_GsY4", "summary": "This paper generalizes the tools of the Conformal Prediction framework to achieve prediction sets satisfying general criteria, including the traditional coverage criteria. They provide an adaptation to Split Conformal Prediction that achieves this new \"risk\" metric. They demonstrate empirically that such a method indeed holds similar risk guarantees to the known coverage guarantee over a host of real datasets.", "questions": "Please see above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The intuition of the paper is strong. Developing a method to generalize Conformal Prediction to different loss functions is a clear improvement on existing literature, and the contribution is clear.\n2. The experiments are very well done. The experiments over several loss functions are clear and convincing. \n3. The paper's motivation is clear, and the impact of such work is immediately evident. \n4. The extensions are strong and of good breadth. This framework is evidently useful for providing guarantees under different risk settings, not just the initial canonical setting of risk presented first. \n\nOverall, I vote for acceptance of this paper. I have several questions on computability and the writing, but these can be fixed and explained for the camera-ready version.", "weaknesses": "1. Some motivating examples of cases where such conformal risk control would be desirable over the traditional coverage guarantee. One can think of cases where F1-score is more appropriate to measure than coverage. It would be great to mention these examples in the introduction so that the reader understands more the motivation of this generalization. \n2. Why is a bounded loss function reasonable? What examples of loss functions violate this or satisfy this? Obviously, a coverage guarantee satisfies this, but what other metrics satisfy this? \n3. The $\\Lambda$ operator is quite confusing. You do not explicitly define it and still say $L_i$ is a map from $\\Lambda$ to another set. You should have defined $\\Lambda$ before. You also say $\\sup \\Lambda \\in \\Lambda$. Is $\\Lambda$ the set of possible $\\lambda$, so the set of real numbers?\n4. In the buildup of section 1.1, I find the intuitive explanation of the method and why it works lacking. I believe a strong reason of why conformal prediction is so popular is its intuitiveness. I find these definitions unintuitive and the provided explanation lacking. I would spend more time describing the intuition of the constants and claims defined. \n5. One of my main concerns is the computability of $\\hat{\\lambda}$. This seems highly nontrivial to calculate. This seems like the most important point of the paper. Could you describe how to compute this? Moreover, does this formalization make it easier to compute than other similar methods doing conformal risk control? \n6. Does the guarantee of $\\alpha -\\frac{2B}{n+1}$ in Section 1.1 match that of conformal prediction? In that case, the known guarantee is $\\geq \\alpha$, which is loose by $\\frac{2}{n+1}$. Does this method recover the tightest guarantees of Split Conformal Predictions when the loss function is set to be miscoverage?\n7. In the related works section, I find the differences between existing and current work confusing. You mention the guarantee in this paper is in expectation. Why is this better than existing works? Do existing works do worst-case analysis or high probability analysis? The technical similarity between different methods is not a sufficient difference in my opinion. In the second paragraph, I miss what the existing conformal risk works do. Is it possible that existing conformal risk works do not directly generalize conformal predictions or are often less tight? I believe more details on the differences between existing conformal risk papers and this paper are needed. \n8. I find the theoretical section again void of any intuition presented. I would highly encourage the authors to provide more intuition on the proof as to the meaning of each step in section 2.1. I found it very difficult to parse. Moreover, the notation in the proof of Theorem 1 is difficult to read. I believe a little cleaning up of this proof would be helpful. An example is that using $\\hat{\\lambda}'$ and $\\hat{\\lambda}$ are poor notation choices and create a lack of clarity. \n9. The assumption in Theorem 2, i.e. $\\mathbb{P}(J(L_i, \\lambda) > 0) = 0$ is provided without sufficient explanation. What set of loss functions satisfies such a definition and why? Which loss functions do not satisfy this assumption? Providing this analysis will help give intuition as to what Theorem 2 shows and when it works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)