Boosted Conformal Prediction Intervals
Authors: Ran Xie
Rina Foygel Barber
Emmanuel Candes
Venue: NeurIPS 2024 poster (2024)
Keywords: Conformal Prediction
Uncertainty Quantification
(Other) Statistical Learning
Primary area: other
TL;DR: We introduce a boosted conformal procedure designed to tailor conformalized prediction intervals toward specific desired properties, such as enhanced conditional coverage or reduced interval length.

Abstract: This paper introduces a boosted conformal procedure designed to tailor conformalized prediction intervals toward specific desired properties, such as enhanced conditional coverage or reduced interval length. We employ machine learning techniques, notably gradient boosting, to systematically improve upon a predefined conformity score function. This process is guided by carefully constructed loss functions that measure the deviation of prediction intervals from the targeted properties. The procedure operates post-training, relying solely on model predictions and without modifying the trained model (e.g., the deep network).  Systematic experiments demonstrate that starting from conventional conformal methods, our boosted procedure achieves substantial improvements in reducing interval length and decreasing deviation from target conditional coverage.

Boosted Conformal Prediction Intervals
Ran Xie Department of Statistics Stanford University ranxie@stanford.c
Rina Foygel Barber
Department of Statistics
University of Chicago
rina@uchicago.com edu
Emmanuel J. Candès
Department of Statisties
Department of Mathematics Stanford University candes&s wareessttaoor....ooommmm edu
Abstract
This paper introduces a boosted conformal procedure designed tailor confor- malized prediction intervals toward specific desired properties, such as enhanced conditional coverage or reduced interval length. We employ machine learning techniques, notably gradient boosting, to systematically improve upon prede- fined conformity score function. This process guided by carefully constructed loss functions that measure the deviation of prediction intervals from the targeted properties. The procedure operates post-training, relying solely on model pre- dictions and without modifying the trained model (e.g.. the deep network). Sys- tematic experiments demonstrate that starting from conventional conformal meth- ods, our boosted procedure achieves substantial improvements in reducing interval length and decreasing deviation from target conditional coverage.
Introduction
Black-box machine learning algorithms have been increasingly employed to inform decision-making in sensitive applications. For instance, deep convolutional neural networks have been applied to di- agnose skin cancer (14), and AlphaFold has been utilized in the development of malaria vaccines [2 25): here, scientists have employed AlphaFold predict the structure of key protein in the malaria parasite, facilitating the identification of potential binding sites for antibodies that could prevent the transmission of the parasite (25). These instances highlight the critical need for under- standing prediction accuracy, and one popular approach to quantify the uncertainty associated with general predictions relies on the construction of prediction sets guaranteed contain the target label or response with high probability. Ideally, we would like the coverage to be valid conditional on the values taken by the features of the predictive model (e.g., patient demographics).
Conformal prediction (3] stands out as flexible calibration procedure that provides a wrapper around any black-box prediction model to produce valid prediction intervals. Imagine we have a data set {(X,, } and test point (X241,Y.41) drawn exchangeably from an unknown, ar- bitrary distribution P (e.g. the pairs (X,,Y) may be i.i.d.). Taking the data set and the observed features X,+1 as inputs, conformal prediction forms prediction interval C, (Xn-1) for Y with valid marginal coverage, i.e. such that IP(Y,+1 E C. (X,++)))) by the user ahead of time. This is achieved by means represents a data point while ff represents any aspects of the distribution that instance, the score may be given by the magnitude of the prediction error by - f(xx), where p(x) represents the model prediction of the expected outcome, in which case ff is simply f. Roughly,
0.95 or any nominal level specified conformity score F(x, yi f), where (ar, y)
have estimated. For
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Training
Boosting
Calibration
Testing
conformity score
boosted score
evaluate scores (ε,,
conformalized prediction intervals
(.-)
Figure/ I: Illustration of the boosted conformal prediction procedure. We introduce boosting stage between training and calibration, where we boost rounds on the conformity score function E(:,)) and obtain the boosted score (-.-).... The number of boosting rounds selected via cross validation. A detailed description of the procedure is presented in Algorithm
would include in the prediction interval if EX y; f' does not take atypical value when Selecting an appropriate conformity score akin to choosing test statistic in statistical testing, where two statistics may yield the same Type error rate yet differ substantially in other aspects of performance.
compared with (E(X,, Yú; f),, 10. ;,....;n.
One central issue is that while the conformal procedure guarantees marginal coverage, does not extend similar guarantees to other desirable inferential properties without additional assumptions. In response, researchers have introduced a variety of conformity scores, including the locally adaptive (Local) conformity score [16], the conformalized quantile regression (CQR) conformity score (18),
and variants, CQR-m [2]] and CQR-r (3). Among these, CQR has often demonstrated superior empirical performance in terms of both interval length and conditional coverage (18).
This paper introduces a boosting procedure aimed at enhancing an arbitrary score function! By employing machine learning techniques, namely, gradient boosting, our objective to modify the Local or CQR score functions (or other baselines) reduce the average length of prediction intervals or improve conditional coverage while maintaining marginal coverage. While this paper focuses primarily on length and conditional coverage, our methods can be tuned optimize other criteria; we elaborate on this in Section
Our boosted conformal procedure searches within a family of generalized scores for a score achiev- ing a low value of loss function adapted to the task hand. Specifically, to evaluate the conditional coverage of prediction intervals, we build a loss function that maximizes deviation from the target coverage rate in the leaves of shallow contrast tree (21). Searching within a strategically designed family of score functions, rather than directly retraining or fine-tuning the fitted model under the task-specific loss function, yields greater flexibility and avoids the costs associated with retraining or fine-tuning. Further, this boosting process is executed post-model training, requiring only the model predictions and direct access the training algorithm.
Source code for implementing the boosted conformal procedure is available online https: github. com/ran-Ta xie www.e//bosoeddeeeeeeeeeeeeeeeeee conf ormal Details regarding the acquisition and preprocessing of the real datasets also provided in the GitHub repository.
The split conformal procedure
We begin by outlining the key steps of the split conformal procedure applied to family ((X,, k of exchangeable samples (e.g., i.i.d.).
Training. Randomly partition [ni] into training set 11 and calibration set 12. On the training set, train a model by means of an algorithm A to produce a conformity score function E(.,;; :; f). The structure of this score function is predetermined, whereas the model f is learned from A. An example a conformity score E(x, y; f) = ly f(x)), where ((xx) is learned regression function that is here simply j..
Calibration. Evaluate the function :) on each instance in the calibration set and obtain scores {E.)ee [7 with each E, EX,, Y; f). The (1 a)th empirical quantile
1 implementation of the boosted conformal procedure (BoostedCP) is available online at https:// github. Com/ran-sn www/boxsteedcoooeteoooooooee
"The term 'score' will henceforth refer the conformity score unless stated otherwise.

---

(E, of the score, Q1,a(E,12) 12). is calculated as
QI 12) = (n =) aa,
where Z follows the distribution [7, | (d + E)..). and 8 is point mass
Testing. For a new observation X output the conformalized prediction interval
(x(xn)1)) {y E R B(X s (E,)((E11)),
(1)
If ties between {Ei her, occur with probability zero, it holds that
1 S P(2+1 E Cx(Xn+1) & a
(2)
12 +1
see [16]. By introducing additional randomization during the calibration step, the prediction interval can be tuned to obey IP (Yn++1 E Cx(xn+)) = =1 a, see [4]. This adjustment is not critical here and we omit the details.
Locally adaptive conformal prediction (Local for short) [66 introduces score function that aims to make conformal prediction adapt to situations where the spread of the distribution of Y varies sig- nificantly with the observed features X. On the training set, run an algorithm A fit two functions po(-) and oo(-), where No (o() estimates the conditional mean E X]. and ox()) ) the dispersion around the conditional mean, frequently chosen as the conditional mean absolute deviation (MAD), E|Jy - po (X)) X]. With f = (No. Tu), the locally adaptive (Local) score function is:
F(r, y: f) = by - #o(x)\/oo(x))
For new observation Xn+1, the conformalized prediction interval I takes on the simplified ex- pression [No(Xn+1) Qi- (Xx Ho (Xx+))) + Q E, (X
Conformalized quantile regression (CQR) [17] also aims to adapt to heteroskedasticity by cal- ibrating conditional quantiles, which often results shorter prediction intervals. Apply quantile regression to produce a pair of estimated quantiles (1x//(xz,,, (1-a/2(ar), where (x(X) is the esti-
mated fith quantile of the conditional distribution of Y. The CQR score function is defined as
E(x, yi f) (2x y. (x))
where f (4a/2. a1-a For a new observation X x,+ following | yields the prediction interval [da/2(Xn+1) Q1-a (E, T2), (x + Qi-a(F, T2)]
(5)
Generalized conformity score families. To construct Local conformity score, we estimate two functions (o(-) and (o()) to plug into B Since these components are constructed without looking performance downstream, it is reasonable to imagine that other choices may enjoy enhanced properties. How then should we systematically select μu--) and a(-)? To address this, we define a generalized Local score family J containing all potential score functions of the form
F: (E(+ss)) E(z, vi f) ly ((=))/((z)) > 0},
(6)
where f = (), For each E(.++)) F, the conformalized prediction interval is given by (x(x) Q1-a (F, T2) (X), (XX + (F,
(7)
Turning to CQR, one notable limitation is the uniform adjustment of prediction intervals by the constant factor Q1-a(E,14 I2), as shown (5). This approach is suboptimal the presence of het- eroskedasticity. as it applies an identical correction to prediction intervals of varying widths for each x= z. Thus, simply updating the fitted quantiles (â.. q/))) and plugging them into the original score function would be inadequate, as the structure of the original score imposes significant limita- tions on the effectiveness of conformalized prediction intervals. To address this, several variants in- cluding CQR-m [22] and CQR-r (23) have been proposed. Focusing on CQR-r, it employs flexible score function, defined as (xx yi f) = max(qu/2(22 y. y.y- (=)))((()))))))))))))/))))))>>)) ((1 (/1-a///////aa ///2///2()))))
with f = (Ω./2 /2, 9 a1- - qa Following conformalized prediction intervals become
(x) - a(X)Q I2), á - (x) +0())=0X--_
-
(8)
3

---

where ė = -1-0/22- /2. Intuitively, the adjusted score function allows prediction bands to adjust in proportion to their width, instead of adding constant shift as in CQR. However, despite the intuitive appeal of adjusted scores as a seemingly more reasonable "allocation" of the conformal correction, empirical studies reveal that they do not result narrower prediction intervals when compared to CQR (23). This phenomenon is largely due to the uniform direction of the conformal adjustment, represented by Q1-a (E, 12), across all observations. In particular, if Q Q1-a(E)) 12) < O, indicating that the true target y predominantly lies within the estimated quantile range [9a/2> Q1 -a/2], there is
a uniform narrowing of the predicted interval across all samples.
In light of these insights, we propose a novel score family, H. designed to augment the flexibility of the conformity score functions:
H == (E(. :f) E(z, y: f) = max {11(2) - y. - (2(2)) /a(x), (z())))(())))))))))))) S (2()) > 0). (9)
where f (11, M2, o). which leads to conformalized prediction intervals of the form
(X) o (XX) (2 + (acaax(>oXXXX)) aX))) I2)].
(10)
Notably, H. includes the Local, CQR, and CQR-r scores as special cases.
3 Boosted conformal procedure
is clear from above that model is trained to produce conformity score E(,, :: f); e.g.. we may learn a regression function p(-) to plug it into score function ly ((x)). To overcome the limitation of working with an arbitrarily selected score function, we introduce boosting step before calibration, see Figure] In a nutshell, we use gradient boosting to iteratively improve upon predefined score E(,, : f) now denoted as (0() (-, +), where the superscript indicates the Oth iteration.
To achieve this, we construct a task-specific loss function l, which takes a dataset 'D and a score function :(.:;f) as inputs, and outputs (E f); D) measuring how closely the conformalized prediction interval aligns with the analyst's ' objective. This loss function f is designed to be differen- tiable with respect each of the model components produced by the training algorithm. Importantly, it does not require knowledge of the gradient of f(x) with respect to x. In the example above, taking the labels as fixed, this means that for each feature I; E D. i 1...., n. if we set in p(zz)), then the loss {EE )): D) is function of {ūr 1 and the derivative D) D)/0j, is well defined. In Sections 5.1 and 6. we present examples of such derivatives.
Each boosting iteration updates the score function sequentially, employing a gradient boosting al- gorithm such as XGBoost [12] or LightGBM [15]. These algorithms accept as input dataset D, base score function : a custom loss function gradients of l with respect to f (denoted V ff)), and a number of boosting rounds т. We may write the boosting procedure as
(E (.)))., E(()),))))))())))) (1, GradientBoosting(2. F(ss
(11)
This yields a boosted score function (..), which then used for calibration and for constructing prediction intervals. The number 7 is calculated using k-fold cross- validatin on the training dataset, selecting from potential values up to a predefined maximum (e.g., 500). We partition the dataset into A folds and for each j 1,...,k, k. we hold out fold for sub-calibration and the remaining A 1 folds for sub-training. We apply T rounds of gradient boosting (IT) on the sub-training data, generating T + candidate score functions E (..)...., E
(..). Each score function is then evaluated on sub-calibration data, using the loss function é to compute losses at all epochs, i.c., for each fold j = 1,...k,
1T = (old)
fold
Last, T is selected as the round that minimizes the average loss across all & folds:
arg minogigr E4
(12)
This cross-validation step simulates the calibration step in conformal prediction and effectively prevents the overfitting of the score function.
Figure 2
Since boosting conducted on the training data, the boosted procedure satisfies the same marginal coverage guarantee as the split conformal procedure, as formalized below.

---

k-fold Cross Validation
B (oma-)
Boost Boost(fold loss max-round)
(+)
fold
best-round
max-round
boost rounds
(omazoround)
(fold Ej (((mas-roodd)
L +/4 4
Figure 2: Schematic drawing showing the selection of the number of boosting rounds via cross- validation. Left: we hold out fold j. and use the remaining
folds to generate candidate scores F 1 O...., max-round. The performance of each score is evaluated on fold using the loss function L. Right: best-round minimizes the average loss across all folds. A detailed description of the procedure presented in Algorithm 1
Proposition 3.1. Let {(X,, =) be the held out calibration set, and (Xm+1, Ym+1) be pair of new observation. If the m + 1 samples are exchangeable, and ties between (X, Y;) ) occur with probability zero, the confromalized prediction interval 7 computed from score function (.,)) satisfies the coverage guarantee (2)
Searching within generalized conformity score families. To update the Local score function
we search within the generalized score family F (6) First, we initialize μ Pu and
After completing iterations of boosting on the training set, we obtain the boosted score function E y) ly ((7))) (=))/<<(())) Notably, we can update any score function within J.. For instance, to update E'x, y; f) ly ((x)), we simply initialize (0()) á. and take o(0) be the constant function equal to one. Similarly, to update the CQR score function 4 we search within the
score family H (9p. First, we initialize triple μr1 (0) 9a/2- 12 d1 01-a/2- ((0) -1-0/2 4a//2-
After boosting rounds, obtain the boosted score function EXT) (zx y) max(ui) (x) y. y n2 (T) (z))/o()
Algorithm Boosting stage
Input:
Training data (X,, Y)) RR R, 1,..., n; base conformity score function ((O)),)) (::))
Loss function f; target mis-coverage level a E 10
Number & of volos-valii folds; maximum boosting rounds T
Procedure:
Randomly divide (1...., n into folds for j <- 1 to A: do
Set fold j as sub-calibration set, and the remaining 1 folds as sub-training set On the sub-training
call GradientBoosting 11 to obtain candidate scores (E 1T
On the sub-calibration set, evaluate _ {(E end for
O....,T
Set boosting rounds 7 <- arg min; 1 Σズ ( in 12
&
On the training set, call GradienlBoosting 11 to obtain boosted functions (E())
Output:
Boosted conformity score function (T) (.,-)
Related Works
Adapting the classical conformal procedure to improve properties of the conformalized intervals has been one the primary focuses of recent literature. Noteworthy contributions -ineisdinnnngggg CF- GNN 28 and ConTr 226 aapproach this problem by introducing modifications to the training stage
5

---

of the procedure. As outlined in Section a model is trained to produce score function E(: f).. The model f usually depends on a set of model parameters, e.g.. neural network parameters 0. Denote the trained model f by So. CF-GNN and Con'Tr retrain or fine-tune the model by using a carefully constructed loss function, which may aim to produce narrower prediction intervals or prediction sets of reduced cardinality in classification problems. This process generates a new set of model parameters 0'. The new model før is then plugged into the same predefined conformity score function-namely CQR (28) or the adaptive prediction set score (APS) 266-tt to produce
There are two primary limitations. First, the score function imposes constraints on the properties of conformalized intervals as explained in Section 2 Our approach introduces more flexibility by constructing family of generalized score functions that is superset of (B(+,-io)) 0 E }). where is the parameter space of the training model. This family strategically designed to contain an oracle conformity score ideally suited to the task at hand, e.g., achieving exact conditional coverage. Second, current methodologics necessitate fine-tuning or retraining models from scratch, requiring both access to the training model and significant computational resources. In contrast, our boosted conformal method operates directly on model predictions and circumvents these issucs.
Conditional coverage of conformalized prediction intervals has also attracted significant interest, characterized by efforts to establish theoretical guarantees and achieve numerical improvements. Prior work established an impossibility result [8] 20]. which states that exact conditional coverage in finite samples cannot be guaranteed without making assumptions about the data distribution. Sub- sequently, Gibbs et al. (27) developed a modified conformal procedure that guarantees conditional coverage for predefined protected sub-gro sub-groups, i.e. subsets of the feature space. Our approach dif- fers from the previous works by introducing a numerical method directly aimed at improving the conditional coverage, IP E Cx(X) v), across all potential values of x.
5 Boosting for conditional coverage
Maintaining valid marginal coverage, our goal is to produce a prediction interval C, obeying
P( C, (X++ 2) R 1
(13)
for all possible values of x. To this end, we present loss function that quantifies the conditional coverage rate of any prediction interval. Requiring merely dataset D and prediction interval (-) as inputs, also serves as an effective evaluation metrie, which may be of independent interest.
5.1 A measure for deviation from target conditional coverage
From now on, we let E be the score function E Set D = {(X., Y:) and denote by Cn the conformalized prediction interval constructed from E. We shall assess the deviation of C,()) from the target conditional coverage by means of Contrast Trees (2T). As background, contrast tree iteratively identifies splits within the feature space X' in a greedy fashion, aiming maximize absolute within-grout deviations from the target conditional coverage rate (1 aa). For a subset R of the data point indices [n], let Dn (X,, YjJjen The absolute guohun deviation is computed
as
d (.C,(.): = | - 'E E (x,)) - (1-a)
(14)
The overall empirical maximum deviation is then defined as
(M (E:D) maxigmg (C.():DA...)
(15)
where R U... U RMM is partition of [n], which itself depends on E and D. Specifically, it is computed by running contrast tree for M iterations. At each iteration, the algorithm not only seeks to isolate regions with large deviations but also discourages splits where any subset R,, is too small.
To update score functions via gradient boosting as described in (TI. we would need differentiable approximation of the maximum deviation. To this end, we construct approximations for the follow- ing three components of the loss function. With an abuse of notation, in subsequent discussions, we shall employ the same notations to denote these differentiable approximations.
Approximation for the prediction interval C., in (44 the prediction interval formu- lated as (7) for the generalized Local score, and as 10 for the generalized CQR score.

---

Denote the upper and lower limits of C.(.) (.) by u(-) and ((-). We approximate the empirical quantile (1-a(E,1 12) in u(-) and 1(-) with smooth quantile estimator Qi-a. Given scalars {zi) Qi is constructed as:
(16) where (.-)) represents the dot product. Here, HD(r) [W..,.... Wr.. is the weight vector corresponding the Harrel-Davis distribution-free empirical quantile estimator [T], and s(z) is differentiable ordering {² arranged in the ascending order. In practice, the derivative of s(z) with respect to each 2; is given by the package devel- oped in [19]. This approach is smooth approximation of the Harrel-Davis quantile estimator Qi constructed as linear combination of the order statistics, Q (HD(r), (=())) = 1 (()) where W takes the value ())-)))+))))((+))))))) (()-0))++)).a(++11))) ((--))(((((((/)))))) and (a,(()) represents the incomplete beta function.
(i-a(1=)==) == (HD(r),
2. Approximation for absolute deviation d; (14): the indicator function in [4] be approx-
imated by the product of two sigmoid functions,
I(Y, E Cx()))) = ((u(X,) Y, - 0) & Sr (u(X) - Y,),,, (Xx)))
where Ti is a parameter, trading off smoothness and quality of the approximation. The sigmoid function S,, (ar) defined as S (1) (1 =-)-1
3. Approximation for maximum deviation: we employ log-sum-exp function 2 to derive the differentiable approximation of f as
L (E; D) := log Em= exp (T2dm (C,, Dm)).
(17)
where T2 is a parameter, serving the same purpose as T1.
Here, we demonstrate calculating the derivative of the smooth approximation each component of the generalized Local score, expanding it as follows:
with respect to
M
la (E;D) 72 -1 log
exp
12
Rm
-1 'EjER,, Sr -
(Y, -(x,))()))))))))))))))))) - (1-a))
m=1
where
(u(,)) - Y2) = (1+ exp
x,))
Sn Sn - -((x)) (1 + exp [- 11 ((
- |Y,-u//oy As result, for each feature X, within D, we can evaluate a (E; /0p; and Ø (E;D) /00, via the chain rule.
with μ4 = p(X.), 01 = (X.). E, |Y,
5.2 Boosting score functions for conditional coverage
Since the empirical maximum deviation lM (15) is non-differentiable, we opt for the diffcrentiable approximation during the gradient boosting step select the number boosting rounds as in step conformalized prediction interval on the test set.
Nonetheless, utilize the original EM to 12) and to evaluate the conditional coverage of the
5.2.1 Theoretical guarantees
The oracle score function achieving conditional coverage as defined generalized score families.
(33 belongs to both proposed
Proposition 5.1 (Asymptotic expressiveness). Let (X,, Y, be i.i.d. with continuous joint prob-
ability density distribution. Under the split conformal procedure. for any target coverage rate
as n. + oc, there exists (p",a*) a²) and (Hî p5, o²) such that the corresponding generalized Local 6
and CQR (9) score functions recover conditional coverage at rate
defined in (73.
It goes without saying that there is no reason to assume that the optimal μr" corresponds to the conditional mean, median or any quantile of Y given X, or that the optimal o" corresponds to the standard deviation the mean absolute deviation of Y given X, as in the original Local score (3). That said, our greedy strategy has no guarantee on global optimality and this is why the choice of the starting point-wiie woither it is the Local CQR score function-piae -paans role in the performance.
7

---

Local
Boosted Local
Node 8
5%
11%
Node
59
5%
10%
5%
Node
50%
10%
74% 5%
15%
25%
25%
13%
10%
Cisssical Split Conformal Procadurs
Boosted Contermal Procadumm
Figure 3: Comparison of test set conditional coverage evaluated on the dataset meps- 19; (a) shows the classical Local-type conformal procedure and (b) our boosted Local-type conformal procedure. The target miscoverage rate is set to 10% (red). Miscoverage rate is computed at each Ieaf of the contrast tree, constructed detect deviation from the target rate, Each leaf node labeled with its size, namely, the fraction of the test set it represents.
Table I:
Test set maximum deviation loss fM evaluated on various conformalized intervals. The best result achieved for each dataset highlighted in bold.
Max. Conditional Coverage Deviation (%), target miscoverage
10%
Method
Method
Dataset
Improvement
Improvement
Local
Boosted
CQR
Boosted
bike
10.979
5.638
-48.5%%
4.934
4.925
-0.17%
bio
5.303
4,862
-8.31%
5.069
4.700
-7.29%
community
25.755
13.466
47.71%
12.688
12.105
-4.59%
concrete
10.740
8.763
-18.40%
9.039
8.265
-8.56%
meps-19
15.357
5.656
-63.17%
5.507
5.507
-0.00%
meps-20
16.939
6.998
-58.69%
7.614
7.184
-5.65%
meps-21
17.627
7.832
-55.57%
8.165
8.067
-1.20%
5.2.2 Empirical results on real data
We apply our boosted conformal procedure the 11 datasets previously analyzed in 23 18 22 Details on the datasets are provided in Section A.6 in the Appendix. In each dataset, we randomly hold out 20% test data. All experiments are repeated 10 times, starting from the data splitting. we employ for the training
We refer to Section A.7 for details on the models and and boosting stages.
We evaluate the conditional coverage of the prediction intervals the maximum within-group devi- ations across partitioned test set 15). This partition is obtained through contrast tree algorithm described in Section 5.1 Figure illustrates the comparison between miscoverage rates of predic- i
tion intervals at each Ieaf of the contrast tree. These intervals are derived under the classical Local conformal procedure and our boosted conformal procedure. Notably, the conditional coverage of the boosted prediction interval more closely aligns with the target rate 1 a.
The experiment results summarized in Table Table[indicatt Lindicate that applying boosting significantly enhances the performance of the baseline Local procedure. In contrast, boosting on CQR does not yield sig- nificant improvements- a sign that CQR already targets conditional coverage. (Before boosting, the prediction intervals generated by the baseline Local procedure exhibit conditional coverage devia- tions up to three times greater than those of the baseline CQR procedure.) It is noteworthy, however, conditional coverage the Local procedure improves to a level comparable
that after boosting, to that of the boosted CQR procedure. While generally slightly less effective, nevertheless surpasses the performance of the boosted CQR procedure in two cases. Results on the remaining datasets are deferred to Tables A2 and A3

---

Boosting for length
We begin by specifying the oracle prediction interval with minimum length. For a random variable Z, the High Density Region (HDR) at specified significance level a. denoted as H is defined as the shortest deterministic interval that covers Z with probability at least 1 - a. The boundaries of HDR, the lower limit Qi(a) and the upper limit Qu(a). obey the condition (
(Qita), Q Qu(ail) 2 a. For a pair of (X, Y) drawn from P, for every value of IR", the oracle prediction interval at that point is expressed as
XY x) = [Xax) = x), Quta) =x)]-
(18)
Before introducing the boosting strategy, we present word of caution against optimizing exclu- sively for this objective. Importantly, to maintain valid marginal coverage, the shortest prediction interval is prone to overcover when the spread of Y|X (the conditional distribution of Y given X) is small, and undercover when the spread of Y|X is large. This may be undesirable.
Similar to Proposition 5.1, we can show that the generalized score families exhibit the necessary expressiveness contain the oracle conformity score, achieving optimal length while ensuring valid marginal coverage. The formal proof is deferred to Section A.3
6.1 A measure for length
Consider a dataset {(,,, Y)) Ji and score function E. Denote the corresponding confor- malized prediction interval by C.(,), with its quality measured by the average length:
(F;
"E (X,)--
(19)
To derive a differentiable approximation of LL. we approximate the empirical quantile Q1-a in the conformalized intervals (7) and (1O) with the smooth quantile estimator Qi-a constructed in (T6). Here, we demonstrate calculating the derivative of the smooth approximation of l with respect to each component of the generalized Local score, expanding as follows based on the previously outlined approximation steps:
l (F: = 20Qi-a(EE:)
=Y; pal/or
with μ; u(X.), (X.). n - os. As result, for each feature X; within D, we can evaluate dl, (E;D) 10p; and 0h, (E; /0a, via the chain rule. For instance,
d3 (E; D) -2 oQi aQi-a((E)))))) (LE,)=) sign ub)
0p4
0E,
O
6.2 Empirical results on real data
We apply our boosted conformal procedure to the same datasets described in Section 5.2.2 De- tailed information on the models and hyperparameters used during the training and boosting stages be found in Section A.7 Partial experiment results summarized in Table 2 Notably, the
boosting performance highlighted in bold exhibits significant improvement compared to previously documented results [17] (3) We see pronounced enhancement with the blog dataset; before boost- ing, the Local prediction intervals are on average 42% longer than those generated by CQR. After boosting, these intervals outperform the boosted CQR intervals by 32%. Using CQR as the baseline also yields substantial improvements, a decrease in averaged length exceeding 10% in six out of the eleven datasets. The meps-21 dataset, in particular, shows an improvement of up 18% relative to the baseline. Results on the remaining datasets can be found in Tables A4 and AS Figure 4 compares the conformalized prediction intervals derived from bascline Local and CQR scores with those obtained from the boosted scores. To effectively visualize the impact of boosting, we conduct a regression tree analysis on the training set to predict the label Y, setting the maximum number of tree nodes to four. This regression tree is then applied to the test set, allowing for detailed comparison of the prediction intervals across each of the four distinct leaves.
7 Discussion
We introduced post-training conformity score boosting scheme aiming to optimize for conditional coverage or length of the conformalized prediction interval. An intriguing avenue for future explo- ration involves simultancously optimizing both length and conditional coverage, potentially trading

---

Classical
Boosted
Local Length
log(LocaWLooaab
CQR Length
log/CaR/CoRb)
6.4
0.2
0.1
Local Longth
log(Locall
CQR Longth
log(COR/CORb)
blog
0.1
0.5
0.05
regression loaf
Figure 4: Comparison of test set average interval length evaluated on the meps-19 and blog datasets: classical Local and CQR conformal procedure versus the boosted procedures (abbreviated 'Lo- calb' and 'CQRb') compared each of the 4 leaves of a regression tree trained on the training set to predict the label Y. A positive log ratio value between the regular and boosted interval lengths indicates improvement from boosting. The target miscoverage rate is set at 10%.
Test set average interval length l, evaluated on various conformalized prediction intervals. The best result achieved for each dataset highlighted in bold.
Table 2:
Average Length, target miscoverage a = 10%
Method
Method
Dataset
Improvement
Improvement
Local
Boosted
CQR
Boosted
blog
2.056
0.972
-52.74%
1.415
1.434
-0.71%
facebook-1 I
1.896
1.383
-27.03%
1.198
1.072
-10.47%
facebook-2
1.854
1.363
-26.51%
1.200
1.075
-10.41%
meps-19
2.070
1.685
-18.60%
2.554
2.136
-16.35%
meps-20
2.081
1.836
-11.80%
2.667
2.357
-11.62%
meps-21
2.063
1.795
-12.99%
2.585
2.105
-18.55%
Additionally, we can readily adapt our procedure to meet various application-specific objectives. For instance, we can optimize for con- ditional coverage on predefined feature groups, common task in enhancing faireess in distributing social resources across different demographic groups (27). Similarly, we can modify our procedure to reduce the length of prediction intervals for predefined label groups, which can be seen reallo- cating resources to decrease uncertainty for certain groups at the expense of higher uncertainty for other groups 26).. Candidate loss functions tailored these objectives are detailed in Section A. Lastly, the primary emphasis of this paper centers on the design of the conformity score boosting scheme and formalizing the optimization of conditional coverage in mathematical terms, leaving room for computational optimization to enhance performance and runtime efficiency. In essence, the gradient boosting algorithm in our procedure be replaced with any gradient-basee machine learning model. Thus, another interesting future direction would be to explore whether alternative algorithms could enhance performance.
off these objectives by incorporating user-specified weights 29
Acknowlegements
E.C. supported by the Office of Naval Research under Grant No. N00014-24-1-2300 the Na- tional Science Foundation under Grant No. DMS2032014, and the Simons Foundation under Award 814641. R.F.B. was supported by the Office of Naval Research via grant N00014-20-1-2337, and by the National Science Foundation via grant DMS-2023109.
10

---

References
[1] Frank E. Harrell and C. E. Davis. "A New Distribution-Free Quantile Estimator". In: Biometrika 69.3 (1982), pp. 635-640. ISSN: 00063444. URL: http: www jatorrstrr.... org stable/3335999
[2] Stephen P Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[3] "Classification with conformal predictors". In: Algorithmic Learning in a Random World.
Boston, MA: Springer US, 2005, pp. 53-96. ISBN: 978-0-387-25061-5 DOI: 10. 1007 /0-
387-25061-1_3 URL: httpt://p// oww/oo///// 1101110770-377725061-11111
[4] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random
world. Vol. 29. Springer, 2005.
[5] I-Cheng Yeh. Concrete Compressive Strength. UCI Machine Learning Repository. DOI:
2007.
[6]
C.M. Achilles et al. Tennessee student teacher achievement ratio (STAR) project. 2008.
[7] Michael Redmond. Communities and Crime. UCI Machine Learning Repository. DOI: http:///oo..rg/1002243///3333
2009.
[8] Vladimir Vovk. "Conditional validity of inductive conformal predictors". In: Asian confer-
ence on machine learning. PMLR. 2012, pp. 475-490.
[9] Hadi Fanaee-T. Bike Sharing Dataset. UCI Machine Learning Repository. DOI:
http:///oo.org/10224442//C
2013.
[10] Prashant Rana. Physicochemical Properties of Protein Tertiary Strueture. UCI Machine
Learning Repository. DOI:
htt.:orrg//..442/555033.H 2013.
[11] Krisztian Buza. BlogFeedback. UCI Machine
Learning Repository. DOI:
https:/dbi.org/10.2444222 10.24432/CC883F 2014.
[12] Tianqi Chen and Carlos Guestrin. "Xgboost: A scalable tree boosting system".. In: Proceed- ings of the 22nd acm sigkdd international conference on knowledge discovery and data min- ing. 2016, pp. 785-794.
[13] Kamaljot Singh. Facebook Comment Volume Dataset. UCI Machine Learning Repository. DOI: https://doi.c http:://doiorr///.... www.oor...43///53/55...
2016.
[14] Andre Esteva et al. "Derratologiistllvel classification of skin cancer with deep neural net- works". In: nature 542.7639 (2017), pp. 115-118.
[15] Guolin Ke al. "Lightgbm: A highly efficient gradient boosting decision tree". In: Advances in neural information processing systems 30 (2017).
[16] Jing Lei et al. "Distribution Free Predictive Inference for Regression". In: Journal of the
American Statistical Association 13.523 (2018), pp. 1094-1111, DOL: 10. 1wt.100000012500
2017 1307116 eprint: https: :tapps///dae //doi. org/10. 1080 /06610/9124999 2017 1307116 https: / /oi. org/10 000000062145555 2017. 1307116
URL:
[17] Yaniv Romano, Evan Patterson, and Emmanuel Candes. "Conformalized quantile regression".
In: Advances in neural information processing systems 32 (2019).
[18) Yaniv Romano, Evan Patterson, and Emmanuel J. Candès. "Conformalized Quantile Regres- sion". In: Proceedings of the 33rd International Conference on Neural Information Process- ing Systems. Red Hook, NY, USA: Curran Associates Inc., 2019.
[19] Mathieu Blondel et al. "Fast differentiable sorting and ranking". In: International Conference
Machine Learning. PMLR. 2020, pp. 950-959.
[20] Rina Foygel Barber et al. "The limits of distribution-
conditional predictive inference". In: Information and Inference: A Journal of the IMA 10.2 (Aug. 2020), pp. 455-482, ISSN:
2049-8772. DOI: 10. 1093 imaiai iaaa017 eprint: https : / academic oup. com/
imaiai Amaiai/article-.. http://pppp//.//////////////////
/0t//////////// 00/////849992222 iaai017aa111.... påf URL: https: httpss://ool:: //doi. org/
10. 10.1098//maiai/iaaad
[21] Jerome H. Friedman. "Contrast trees and distribution boosting". In: Proceedings of the
National Academy of Sciences 117.35 (2020), pp. 21175-21184
DOI: 10 1073 pnas
1921562117 eprint: https://www: pnas org do /df/ /10pff.ff.//// 1073 1073/paas...... pnas 1921562117
URL: https:///wwttt http://wwwwpnas...amm pnas org /tt///di///b///// /073//nas.../. httpppas....11111..1111 1921562117
[22] Danijel Kivaranovie, Kory D. Johnson, and Hannes Leeb. Adaptive, Distribution- FFree Pre-
diction Intervals for Deep Networks. 2020. arXiv: 1905. 10634
1stat.ML)
11

---

[23] Matteo Sesia and Emmanuel J Candès, "A comparison of some conformal quantile regression methods". In: Stat 9.1 (2020), e261.
[24] John Jumper et al. "Highly accurate protein structure prediction with AlphaFold". In: Nature 596.7873 (2021), pp. 583-589.
[25]
Kuang-Ting Ko et al. "Structure of the malaria vaccine candidate Pfs48/45 and its recognition by transmission blocking antibodies". In: Nature Communications 13.1 (2022), p. 5603.
[26] David Stutz et al. "Learning Optimal Conformal Classifiers". In: International Conference on Learning Representations. 2022. URL: https : / openrevier net forum -8-
4LKFVX
[27] Isaac Gibbs, John J. Cherian, and Emmanuel J. Candès. Conformal Prediction With Condi- tional Guarantees. 2023. arXiv: 2305 12616 Estat.ME)
[28]
Kexin Huang et al. Uncertainty Quantification over Graph with Conformalized Graph Neural Networks. 2023. arXiv: 2305 14535 [cs .GG]
[29]
Lahav Dabah and Tom Tirer. On Temperature Scaling and Conformal Prediction of Deep
Classifiers. 2024. arXiv: 2402. 05806 [cs. LG]
URL: http:://arxivv.rgg/../ www./bs///aa///eee 2402.
05806
[30]
Medical expenditure panel survey, panel 19, URL: data_stats,
[31]
Medicul expenditure panel survey, panel 20. URL: httpp:://aegg.aarrrgg////////aee
Seatasstas/doee cests/ddwnlooe load, http:/ooodddddaaaaaaaaeaaaaa seraaffles___ee filee_dettail__lll_eelle__ell_le http
[32]
/epov///p/wb////// Medical expenditure panel survey, panel 2/. URL: htt.hhhtsss///mpp...rrg....00
data_ iitaasttts/ddut http://oo.....aaaa......eee/eeer
12

---

A Appendix
A.1 Candidate loss functions for additional application-speciii objectives
Conditional coverage on predefined feature groups: this task can be viewed specialized ap- plication within our broader strategy of boosting for conditional coverage, as detailed in Section 5 There, the primary challenge to develop a loss function that accurately measures deviations from the target conditional coverage rate. We achieved this by using contrast trees to identify partitions in the feature space that maximize these deviations, effectively identifying subgroups in need of pro- tection. This process is simplified when the partitions correspond to prespecified groups, allowing us continue using the empirical maximum deviation candidate loss function.
Consider dataset D {(X,, X= and score function E. Denote by Cn the conformal- ized prediction interval constructed from E. Let GI...., Gm be prespecified feature index groups. Within each set D, {(X,. ,e),, compute the absolute deviation d, as
1
d, (C., D.)
|G,| Ejea, 1(2 E (x)))) (1 oz)
(20)
The overall empirical maximum deviation is then defined as
L (E; = maxisis (C,()):
(21)
Interval length conditional on predefined label groups: for a dataset D = (X,, Y)) and score function E, let ........J. Vas be the prespecified label groups. A natural minimization objective for balancing uncertainty among these groups is defined as:
M
( D)
Σ m E 10, V.)
Σ 1V, E (X)))(())
where (Wi...., Was) represents set of user-specified weights.
A.2 Proof of Proposition 5.1
Our proof relies on the following lemma.
Lemma A.1 (Expressiveness)) Given any sample pair X and Y with a continuous joint probability density distribution, and prediction interval [ea(-), c-(-) with marginal coverage equal to there exist specific function sets: (.)) the Local type, and (p1()):2(()) for the CQR rype, such that asymptotically:
1. The conformalized prediction interval 7 derived using the generalized Local type confor- mity score Ju.o. accurately recovers (ea(-), cx()))
2. Similarly, the conformalized prediction interval (7O.. based on the generalized CQR type conformity score Eny, also recovers [cr(-), (-)-
Proof of Lemma A.I Recall that the generalized Local score (6) characterized by (u, takes the form
E (z, y) 1y m(xx)
(22)
a (xx
Asymptotically, the conformalized prediction interval is given by
(u(X) - Q (F (X),, ((X) + Q1-a (F
(23)
Here, Q1-a represents the population quantile. Set
(*x) cx(x) + (x)
o(x)= Cu(x) c7(x)
2
2
By assumption, we have
P(Y E (c(()) = a.
13

---

With a simple change of variables, the above inequality is equivalent to
(>-uX) (X)
Y m(x)
P
1 1
In other words, this is equivalent
(1-a(E =1
We have thus proved the result for the generalized Local type conformity score Ex,o- In the same spirit, we can prove the result for the generalized CQR type conformity score Epi. by taking
#1 = + Cu (z) ci(x)
Ca - - (z)
ō Ca(a) c(x))
P2 Cu
2
2
2
Recall that generalized CQR score function p characterized by (#1, M2. σ) is defined as:
Epr (z, y) max (2) y. y - -2(=)) /o(x)) which leads to the asymptotic conformalized prediction intervals of the form
(24)
(x) o ( M2 (2(X) + (X) (Epr.
(25)
Plugging in Ml, U2, ē defined above, we immediately have
P(Y E - -,, 50) =1
-1-a
((a(()(-X - Y
P -VX)) 1
Y -x)((X)++ + a())
$1 =
((X)
o(X)
Q1-a oo 1.
ca(r) = Q1-a/2(X|X = Proof of Proposition 5.7 It suffices to take (xx) = Qa/22Y =
and apply Lemma A.1
A.3 Boosting for length: theoretical guarantees
Similar to Proposition 5.T we show in Proposition A.2 below that the generalized Local and CQR score families exhibit the necessary expressiveness contain the oracle score, achieving optimal length while ensuring valid marginal coverage.
Proposition A.2 (Asymptotic expressiveness). Under the assumptions of Proposition 5.1 for any target coverage rate 1 Or, as n - xo, the following statements hold true:
1. There exists (u²,a") such that the corresponding generalized Local score function 6 re- covers the shortest oracle prediction interval (18).
2. There exists (μi (u2 o²) such that the corresponding generalized CQR score function 0 recovers the shortest oracle prediction interval (78)
Proof of Proposition A.2 suffices to take (xxx) (Y|x = x),, cu(x)) = Quía (Y = x) and apply Lemma A.I where Qi(a) and Qu(a) are the lower and upper limits of the High Density Region defined in [8).
A.4 CQR type conformity score boosting
A generalized CQR score function 9 is uniquely defined by a triple (+i(-), (e()),(())) We will show how searching for a generalizedCCRR score can be reduced searching for Local generalized score. To begin with, we shall say that score functions are equivalent if they recover identical conformalized prediction intervals.
Definition A.3. Let [X,, Yi (Xat1, be i...d. with continuous joint probability density distribation, and let [rd] be partitioned into a training set In and a calibration set Iz. Consider two conformity score functions, En and Ez, which produce conformalized prediction intervals C1 (.) and C2(-), respectively. For any target coverage rate a, Ey and Fez are equivalent if (.) = C2(.) when marginal coverage rates IP (Yn+1 E (x(x++1) and IP (Yn+1 E C2(Xn+1)) match.
14

---

Building on this definition, we are now equipped to establish the following equivalences:
Lemma A.4. Under the assumptions of Definition A.3 the following statements hold:
1. For the CQR-r score function defined in Section 2 there is an equivalent generalized Local score function characterized by a pair (p()), ()), where μ (Ju + p2)/2, a = (12 /2)/2.
2. For any generalized Local score function characterized by the pair (n(+), o.)), there is an equivalent generalized CQR score function characterized by triple (1()),
The proof the above Lemma is deferred to Section A.5 Leveraging these equivalences, we carry out the boosted conformal procedure follows: first, we initialize triple μi (0) 4a/2 M2 (0)
h--a/2, 0 (0) 41-a/2 la/2. which characterizes the CQR-r score function. Next, we find an equivalent generalized Local score function characterized by a pair (n((), _a(0)) chosen according Lemma A.4 After boosting rounds, obtain the boosted pair (u(P), a(rr) and the corresponding score function. Finally, we recover an equivalent generalized CQR score function
(x, v) max
(T) (x) - y: y - /22 (x)) / (x),,
[1
characterized by the triple (n (T), μ2 (T), 01 (7)) chosen according to Lemma A.4
A.5 Proof of Lemma A.4
Recall that the generalized Local score 6 characterized by (μ, takes the form E (z. v)
|/ - [-())
(26)
(zz)
The conformalized prediction interval is given by [u(X) Q1-a (Ea. (2) (X) + Q (E ())(())
(27)
A generalized CQR score function y characterized by (H1, P2, σ) is defined as:
Epr.ng (x. y) max (x)(x) y. y-w-ma - p2(x)) /o(w),
which leads to conformalized prediction intervals of the form
[w (X)Q (E 12), (2 (X) ( 12)..
1. Plugging in the triple (1(x) 2a/2 M2(x) a-a22 (I) 4- da/2. which characterize the CQR-r score function, we have the conformalized prediction interval
(xx(X) (1)))111d))) oa..on I2), (x)) 01 xx 1o I2)].
Set
((X) (i( + (2(X) ,a(x)= (2(X) - (X)
2
2
11y00l---0 then the generalized Local conformity score En,o (I, = 1y 1u-p66 m(x))//(()) recovers con- formalized prediction intervals of the form
[m(X)- - (X)Q(Ep 12), ((X) )+ + o 12]
(X) + (2()) (x(X) (x(X)
Q ((E (0n,,11))) 12),
2
2
M1 (X) + (2(
N2(X) -1 (x)
Q Q(E
2
2
Q (F T2)-1 -1
[pa(X) - (p2(X) (x))
2
Q(E (.u.ootz) Iz)
(x(X) + (H2(X) - (x))))
2
(x) Q QEE 12) - [2(
(X) + 01 (X) Q(E 12) -1
2 From the monotonicity of the interval lengths with respect to the empirical quantiles, we have that the two score functions are equivalent by Definition A.3 A.3
2
15

---

2. Let a generalized Local score function be (2, = --1xx)//o(a)) Then it suffices observe that
ly [(=)) | maxfy p(2), y).
A.6 Additional information real datasets
In Table AT provide the predicted label, dimensions, and source for each dataset. Data cleaning and preprocessing are in accordance with the methods described by Romano et al. 17
Table Al: Datasets for our empirical analyses, with the predicted label, number of samples (zi), and features (d).
Name bike
Label
T2
d
Source
bike rental counts
10886
18
bio
deviation of predicted from native protein structure
45730
blog
number of comments in the next 24 hours
52397
280
community
crime rate per community concrete compressive strength Facebook comment volume Facebook comment volume utilization of medical services utilization medical services utilization of medical services total student test scores up to the third grade
1994
100
concrete
1030
facebook-1
40948
53
facebook-2
81311
meps-19
15785
139
meps-20
17541
139
meps-21
15656
139
star
2161
39
All datasets, except for the meps and star data sets, are licensed under CC-BY 4.0. The Medical Expenditure Panel Survey (meps) data is subject copyright and usage rules. The licensing status of the star dataset could not be determined.
A.7 Experimental Setup
In each dataset, we randomly hold out 20% test data. The remaining data divided into a training set and calibration set, each taking proportion of and 1 y. We explore training ratios ranging from 10% to 90%. Results corresponding to the optimal value of the hyperparameter y are recorded in Table [ following the practice of Sesia er al. (23).
In the training stage, we employ the random forest regressor from Python's scikit-learn package to learn the baseline Local score function. The hyperparameters are the package defaults, except for the total number of trees, which we set to 1000, and the minimum number of samples required at leaf node, which we set to 40, as recommended by Romano al. [17). For the baseline CQR score function, we adopt black-box neural network quantile regressor with three fully connected layers and ReLU non-linearities, following the practice Sesia et al. [23). In the boosting stage, we set the prree-pprramteer 72 in the approximated loss (17) to 50. The approximated loss is then passed the Gradient Boosting Machine from Python's XGBoost package along with base conformity score. We set the maximum tree depth to 1 to avoid overfitting and perform volss-vaiiottion for the number boosting rounds, as outlined in Section All other hyperparameters are set to package defaults.
All experiments were conducted on dual-socket AMD EPYC 7502 32-Core Processor system, utilizing 8 its 128 CPUs each time. The runtime for each dataset and random seed varies by dataset size, ranging from 10 minutes to 5 hours.
A.8 Additional results and error bars
In Tables A2 to A5 we present additional results on marginal coverage, maximum conditional coverage deviation (fM), and average interval length (Ép) for each real dataset (including those not reported in Tables [1] and 2). both before and after boosting. Notably, in each case, boosting applied to optimize either conditional coverage or average interval length. a result, the non- targeted characteristic may or may not improve after boosting.
16

---

Table A2: Additional information on conformalized intervals obtained before and after boosting for conditional coverage with the Local conformity score as baseline. The target miscoverage rate is set to a 10%.
EMd (%6)
Marginal
Cov.(%)
Dataset
Local
Boosted
Local
Boosted
Local
Boosted
bike
1.775
2.201
10.979
5.638
89.927
89.646
bio
1,602
1.614
5.303
4.862
90.024
90.093
blog
2.080
3,403
51.353
48.591
89,995
90,040
community
1,824
10.759
25.755
13,466
90.376
89.549
concrete
1.058
1.062
10.740
8.763
90.583
91.359
1acehnok-1
1.896
4.790
26.020
25.917
90.201
90.056
facehook-2
1.881
2.273
42.807
42.437
89.966
89.989
meps-19 19
2.074
2.926
15.357
5.656
90.120
90,497
meps-20
2.102
2.778
16.939
6.998
89.963
90.031
meps-21 2,069
2,537
17.627
7.832
90.064
90.054
star
0.189
0.179
9.658
9.348
90.808
90.831
Table A3: Additional information conformalized intervals obtained before and after boosting for conditional coverage with the CQR conformity score as baseline. The target miscoverage rate is set to 10%.
(xr(%)
Marginal CQR
Cov.(%)
Dataset
CQR
Boosted
CQR
Boosted
Boosted
bike
0.555
0.540
4.934
4,925
90.073
90.184
hio 1,518
1.515
5.069
4.700
89.841
89.853
blog
1.761
1.766
27.760
26.836
90.222
90.244
community
1.718
1.740
12,688
12.105
90.340
90.194
concrete
0.484
0.489
9.039
8.265
90.451
90.652
facebook-1
1.374
1.371
13.407
13.255
90.465
90.247
facebook-2
1.465
1.409
18.257
18.002
89.763
90.001
meps-19 19
2.784
2.784
5,507
5,507
90.257
90.257
meps-20 20
2.769
2.743
7,614
7.184
89.991
90,006
meps-21
2.834
2.815
8.16
8.067
90.169
90.067
0.199
0.209
9.728
9.630
91.085
91.339
Table A4: Additional information on conformalized intervals obtained before and after boosting for
length with the Local conformity score baseline. The target miscoverage is
10%.
Ex (6)
Marginal Cov.(%)
Dataset
Local
Boosted
Local
Boosted
Local
Boosted
bike
1.775
1.360
22.590
19.616
89.927
89.862
1,562
1,514
5,995
5,791
89,937
89,962
blog
2,056
0.972
52.440
54,858
89,988
89,978
community
1.728
1.678
26.066
24.822
89.499
89.323
concrete
1.010
0.698
11.029
10.518
90.631
90.728
facebook-1
1.896
1.384
26.020
34.259
90.201
89,944
facebook-2
1.854
1.363
42.624
50.697
90.020
89,972
meps-19
2.070
1.685
18.626
14.623
90.054
90.070
meps-20 2,081
1.836
17.897
14.643
89.869
89,849
meps-21
2.063
1.795
18.795
13.324
89.914
89,920
star
0.179
0.179
9,976
9.407
90.901
90.577

---

Table A5: Additional information on conformalized intervals obtained before and after boosting for length with the CQR conformity score as baseline. The target miscoverage rate is set to = 10%.
(%)
Marginal (ov.
Dataset
CQR
Boosted
CQR
Boosted
CQR
Boosted
bike
0.553
0,489
9,530
10.092
90.041
90,418
bio
1.516
1.468
5,408
5.265
89.670
89.880
blog
1.445
1.434
29.875
34.011
90.149
90.260
community
1.693
1.699
14.006
13.536
89.499
89.699
concrete
0.391
0.393
10.342
10.700
88.932
89.223
facebook-1
1.198
1.073
20.132
30.901
89.937
90.013
facebook-2
1.200
1.075
29.233
34.475
90.035
90.010
meps-19 19
2.554
2.136
10.357
11.285
90.228
90.399
meps-20
2.667
2.357
10.826
10.720
89.875
89.838
meps-21 2.585
2.105
10.968
10,565
89.946
89,863
star
0.195
0.194
9,982
10.142
91.455
91,432
We have previously reported the evaluated losses EM and L for each dataset, averaged ten random seeds. Tables A6 and below detail the distribution of these evaluations, providing the mean, 10% quantile, and 90% quantile for the
deviations in conditional coverage (Ex) and average interval length (fz.). These statistics are derived from 110 test set evaluations across datasets and 10 random training-test splits. We opt report empirical quantiles instead of standard deviations due to the asymmetric and non-Gaussian nature of the data.
Table A6: Distribution of the test set conditional coverage deviation y evaluated on various con-
formalized prediction intervals across datasets and 10 random training-test splits.
Max. Conditional Coverage Deviation (%), target miscoverage 10%
Method
Method
Statistics
Local
Boosted
CQR
Boosted
mean
21.140%
16.319%
11.106%
10.771%
10% quantile
7.267%
4.604%
4.890%
4.910%
90% quantile
47.832%
44.712%
22.585%
18.697%
Distribution of the test set average interval length é, evaluated on various conformalized prediction intervals across 11 datasets and 10 random training-test splits.
Table A7:
Average Length, target miscoverage a = 10%
Method
Method
Statistics
Local
Boosted
CQR
Boosted
mean
1.677
1.317
1.483
1.319
10% quantile
0.950
0.513
0.346
0.351
90% quantile
2.082
1.829
2.655
2.210
A.9 Experiments under different miscoverage rates
In Tables A8 and A9 we illustrate the performance of boosting for length with the Local conformity score baseline with miscoverage rates set to 5% and 20%, respectively.
18

---

Table A8: Additional information on conformalized intervals obtained before and after boosting for length with the Local conformity score as baseline. The target miscoverage rate is set to 5%.
(16(
Marginal
Cov.(%)
Dataset
Local
Boosted
Improvement
Local
Boosted
Local
Boosted
bike
2.523
1.828
-27.54%
11.353
11.626
94.927
94,972
bio
1.881
1.801
-4.25%
5.013
4.694
94.857
94.802
blog
4.217
1.923
-5.39%%
37.976
34.505
95.027
95.048
community
2.578
2.217
-13.99%
16.406
14.554
95.414
94.812
concrete
1.129
0.838
-25.80%
8.55
8.086
94.709
94.515
facebook-1 1
2.667
2.262
-15.18%
20.85
20.052
95.104
95.049
facebook-2
2.441
2.092
-14.30%
41.645
39.943
94.941
95.016
meps-19
3.618
2.825
-21.92%
10.515
9.178
95.005
95.008
meps-20
3.951
3.049
-22.84%
11.043
9.693
94.907
94,922
meps-21 4.030
3.033
-24.74%
8,918
8,910
95.061
95.026
star
0.207
0.207
-0.11%%
6.57
6.906
95.358
95.289
Table A9: Additional information on conformalized intervals obtained before and after boosting for
length with the Local conformity score baseline. The target miscoverage rate is
20%.
t
(%)
Marginal Cov.(%)
Dataset
Local
Boosted
Improvement
Local
Boosted
Local
Boosted
bike
1.312
0.983
-25,07%
24.649
28.561
79.403
80.152
bio
1.248
1.213
-2.80%
10.838
10.540
79.729
79.594
blog
1.904
0.513
-73.07%
45.813
64.360
79.756
79.920
community
1.337
1.234
-7.71%
25.047
28.831
79.674
79.674
concrete
0.833
0.549
-34.10%
13.831
20.177
80.728
80.825
facebook-1
1.624
0.747
-54.02%
29.384
47.994
80.190
79.731
facebook-2
1.580
0.749
-52.60%
36.488
62.070
79.890
79.930
meps-19 19
1.821
1.020
-44.0%%
20.561
24.763
80.326
80.013
meps-20
1.843
1.095
40.60%
19.137
24.194
79.684
79.823
meps-21
1.831
1.064
41.86%
18.227
23.172
80.674
80.057
star
0.142
0.141
-0.71%
15.420
14.517
80.647
80.370
A.10 Training gradient boosting algorithm with our custom loss functions
Our proposed boosted conformal procedure serves as a post-training step designed to refine the conformity score : f) obtained during model training. This procedure leverage ire-trained models when available. the absence of pre- pre-trained models, we can alternatively train a gradient boosting algorithm directly using our custom loss functions. For example, in the context of the local score, may initialize with (0) 0, (0) 1 and then apply boosted conformal procedure. As discussed in Section 7 this approach is flexible enough to replace gradient boosting with any madieent-based algorithm, such as neural networks, trained under custom loss functions. This framework aligns with that of (26). which introduces a neural network trained with a custom loss function to minimize the average prediction set size in classification tasks.
In Figure Al we compare the performance of the two approaches optimizing for average interval length, searching within the generalized Local score family defined in (6). The primary distinction between the two procedures lies the initialization: the first approach employs ja = 0, (0) 1, while the second derives (0) and (9) from trained random forests model. We run the experiments on the meps-19 dataset and compare the performance across different splits of the training and cali- bration data. In this context, the percentage of training data refers the proportion of training data within the combined training and calibration datasets. Our results indicate that cross validation se- lects a greater number of boosting iterations when we directly train the gradient boosting algorithm, resulting in longer runtime. However, the average interval length and maximum conditional cover- age deviation after boosting are notably smaller for the boosted conformal procedure we introduced in this paper.
19

---

Training under custom loss
Boosted conformal procedure (ours)
Length
Marginal Cov.
En
Runtime
1.9
0.95
40
0.16
0.85
20
0.14
8
8
30
00
80
Percentage of training data
Figure Al: Comparison of boosted interval length, marginal coverage, maximum conditional cov- erage deviation (EM) ) and runtime between direct training of gradient-based algorithm (red) and boosting on a pre-trained conformity score (blue).
CV
Validation
Boosted Length
Boosted Length SD
Runtime
1.45
0.05
1.44
10
2
7
3
6
7
Ratio between training and validation data
Figure A2: Comparison the average boosted length, standard deviation of boosted length, and average runtime of the boosting procedure when selecting the optimal number of boosting rounds using 5-fold cross-valles noss-validation versus hold-out validation set of varying sizes. Experiments are conducted on the bike dataset, with target miscoverage rate of a 10%.
A.11 Selecting optimal boosting rounds via hold-out validation set
cross- volsdvaliottion the training set to determine the optimal number of boosting rounds, a process that can time-consuming. An alternative approach to hold out fraction of the training set for validation. While more computationally efficient, this method introduces a trade-off: a smaller validation set can lead to greater variability in prediction intervals and model performance, whereas larger validation set may reduce the effective training set size, potentially limiting the model's performance. To explore this trade-off, we conduct experiments on the bike dataset, optimizing for prediction interval length. We compare performance across two settings: 5-fold cross-validation and a hold-out validation set, with the training-to-validann ranging from 1:1 to 8:1. For each setting, we 100 experiments, recording the average boosted length, the standard deviation of boosted lengths, and the average runtime. The results are shown in Figure A2
In boosted conformal procedure,
set ratio
A.12 Additional figures on individual datasets
In this section, we present a series supplementary figures. First, we showcase the improvements in conditional coverage achieved through the boosted procedure for each benchmark dataset. Figure|A3 details results for datasets meps-20 and meps-21. Figure A4 details results for datasets community, bike, and concrete.
20

---

Next, we illustrate enhanced interval lengths. Figure A5 details results for datasets meps-20, meps- 21, and bike. Figure/A6|jetaily results for datasets facebook- facebook-2 2, and conerete. Finally, we demonstrate in Figure A7 how cross-validating the number of boosting rounds effectively prevents the gradient boosting algorithm from overfitting.
Boosted Local
11%
Node
11%
Nodo
18%
12%
19%
Node
Nodo
5%
Nodie 5
Node
10%
12%
17%
Node 3
15%
Nodo
13%
Node
16%
17%
Naxt
14%
10%
20
90
26
30
Niccoarage (95)
Viscorcrig: (86)
(a) meps-20 dataset.
Local
Boosted Local
11%
Nodw
12%
Nodo
16%
Nodo
11%
10%
Nodo:4
12%
Tect
Nodo
Twet Data
17%
Node 2
2495
10%
Node:
10%k
26
Miccovenogn 15)
Miecoverage 1l0
(b) meps-21 dataset.
Figure A3: See the caption of Figure
for details.
21

---

Boosted Local
19%
Node: /
20%
Noda
0%
10%
14%
Node 8
20%
Noda
0%
Node
2%
13%
Seet Date
10%
Nodo
12%
Tee Dats
21%
Nodo
53%
Node: 1
10
20
Miscoverage (S)
30
20
30
Miacoverage (44
(a) community dataset.
12%
Naxse
Noste 8
Node
30%
Node 5
1.7%
Node:
10%
X20
Nodo 4
17%
12%
Nade
12%
0%
See Tats
10%
Node:
Ders
10%
Node 2
11%
Node:
20%
Nodo
Miscowraag (50)
Misoverage
(b) bike dataset.
24%
Node
20%
28%
Node
4%
Jest Data
Test Data
19%
10
30
Miscoverage
Miscoverage
concrete dataset.
Figure A4: See the caption of Figure 3 for details.
22

---

Clessical
Boosted
Log Ratio
Local Power
Local Log Ratio
COR Power
CDR Log Ratio
0.2
0.2
0.1
regression tree leaf
meps-20 dateset.
Classical
Boasted
Log Ratio
Local Power
Local Log Ratio
COR Power
CQR Ratio
0.3
3
0.3
0.1
0.1
leaf
(b) meps-21 dateset.
Classical
Boosted
Log Ratio
Local Power
Local Log Ratio
CQR Power
CQR Log Ratio
1.5
0.5
-0.1
0.5
0.7
-0.3
regression tree
(c) bike dateset.
Figure A5: See the caption of Figure for details.
23

---

Clessical
Boosted
Log Ratio
Local Power
Local Log Ratio
COR Power
CDR Log Ratio
8
0.2
2
-0.2
regression tree leaf
(a) facebook-1 dateset.
Classical
Boasted
Log Ratio
Log Ratio
CQR Power
CQR Log Ratio
0.4
10
0.2
-0.2
0
2
3
leaf
(b) facehook-2 dateset.
Classical
Boosted
Log Ratio
Local Power
Local Log Ratio
CQR Power
CQR Log Ratio
0.9
0.8
0.4
0.0
0.3
0.2
2
regression tree
(c) concrete dateset.
Figure A6: See the caption of Figure for details.
24

---

Average
Test Loss
0.16
0.115
0.110
0.14
0.105
0.12
0.095
0.10
0.08
0.085
100
200
300
400
500
100
200
300
400
500
(a) seed 9.
Average CV Loss
Test Loss
0.14
0.155
0.150
0.12
0.145
0.11
0.140
0.10
0.135
0.09
0.130
0.08
100
400
100
200
(b) seed
Average CV Loss
Test Loss
0.110
0.100
0.105
0.095
0.095
0.090
0.090
0.085
0.080
0.080
0.075
100
200
300
400
500
0
100
200
200
400
500
t
t
(c) seed
Figure A7: Empirical maximum deviation bxr across 7 500 boosting rounds evaluated on dataset concrete under random seeds 8, 9, train-calibration ratio 60% The left panel illustrates the cross- validated loss, computed as the average across sub-calibration folds. The right panel displays the test loss. The optimal number of boosting rounds т, determined through cross- voos-validation as specified in (12). is highlighted in red.
25

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, ad- dressing issues of reprodueibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes]. [No), or [NA]-
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors their evalu- ation. While "[Yes] is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification elaborate, All supporting evidence can appear either the main paper the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
provided a proper justification is given (e.g.. "error
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Assumptions necessary for establishing the theoretical guarantees are detailed in each result presented in the paper. Additional limitations are discussed in Section
26

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or a speech-to-tens system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[Yes]
Justification:
See Sections A.4
A.2 and A.3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
See Sections 5.2.2 and A.7
Guidelines:
The answer NA means that the paper does not include experiments.
27

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recogmize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Real data sets for our empirical analyses are publicly accessible and described in Section A.66 Currently, the code accessible upon request. Although the code has not been released yet, we have provided detailed documentation for reproducibility in Sections
5.2.2 and A.7
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co cc/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
wwi.ps.cc/publi//uuiie.oe httpa////ggiieeee/ooebbebbbt Cow////deeSeSSeeeeeeeeeee ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
28

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
See Section A.8
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors verified.
For asymmetric distributions, the authors should careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See Section|..7
Guidelines:
The answer NA means that the paper does not include experiments.
29

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g.. preliminary or failed experiments that didn't tt make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymitt (e.g., if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Introduction.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
30

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yss]
Justification: See Section A.7
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the pack- age should be provided. For popular datasets, (paperswithooee com datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer:
[Yes]
Justification:
See Sections 5.2.2 A.7 and A.6
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsoureing experiments and research with human subjects, does the pa- per include the full text instructions given participants and screenshots, applicable, well as details about compensation (if any)?
Question:
31

---

Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "KXlR5HswQo", "reviewer": "Reviewer_trTq", "summary": "The paper resents a novel method to enhance conformal prediction intervals using gradient boosting. The proposed method focuses on improving specific properties such as conditional coverage and interval length. The key idea is to iteratively refine a predefined conformity score function via boosting, guided by a loss function designed to measure deviations from the desired properties. Experiments demonstrate that starting from conventional conformal methods, the boosted procedure significantly reduces interval lengths and better aligns with target conditional coverage.", "questions": "1. Can your method work well with different error rates $\\alpha$?\n\n2. If the post-training set is small, can your method still perform effectively?", "limitations": "Please refer to the weakness.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Post-Training Enhancement: The boosting process is applied post-training, meaning it doesn't require modifications to the original model, making it applicable to a wide range of pre-trained models.\n2. Flexibility: The approach can be tailored to optimize various properties beyond interval length and conditional coverage, such as fairness or other specific criteria.\n3. Robustness: The use of cross-validation to select the number of boosting rounds helps prevent overfitting and ensures the method's robustness across different datasets and applications.", "weaknesses": "1.  A comprehensive description of the experiment setting is necessary. Since the method is post-training, it is crucial to explain how the training samples used for post-training differ from those used for the base model. Additionally, it is important to address potential outcomes if a distribution shift occurs. If the data used for post-training is different from the one use for calibration and testing, can it still works well? (or if the data used for (posted-training,calibration, testing) is different from the one used for base model training, can it still achieves to improve CP?)\n2. For experiment presentation, why the tables contain only conditional coverage and size solely? I mean,  to convincingly demonstrate the method's effectiveness, it is essential to show that the method can improve interval length without significantly compromising conditional coverage, or improve conditional coverage without significantly increasing interval length.(rather than showcase them solely) \n3. The introduction of a boosting step increases the computational burden, potentially requiring significant resources and time, especially for large datasets. An analyze for time complexity is required for better illustration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to carefully review our manuscript. In response to each of the reviewer’s specific comments on weaknesses:\n\n1. \n> A comprehensive description of the experiment setting is necessary.\n\nIn our revision, we will more clearly describe our experimental setting. For concreteness, we shall make the following clarifications: we use the same data to train the initial prediction model and to boost. Building upon the trained prediction model $\\hat{f}(x)$, we boost for an improved conformity score, and subsequently calibrate with the boosted conformity score on the calibration set. As a result, under the same exchangeability assumption as for the classical split conformal procedure, an expected marginal coverage rate of $1-\\alpha$ is guaranteed on a new test point drawn from the same distribution.\n\n> Additionally, it is important to address potential outcomes if a distribution shift occurs.\n\nWe of course acknowledge that if the exchangeability assumption is violated (e.g., in the case of a distribution shift), the marginal coverage property may no longer be valid. Robustness vis a vis distribution shifts is the subject of an immense literature and outside of the scope of this paper; that said, this is an interesting direction to explore. It would be interesting to see how our boosting procedure could be combined with a line of research which relaxes the exchangeability assumption: for instance, \n\n[1] Barber R F, Candes E J, Ramdas A, et al. Conformal prediction beyond exchangeability. *The Annals of Statistics*, 2023, 51(2): 816-845.\n\n[2] Gibbs I, Candès E J. Conformal inference for online prediction with arbitrary distribution shifts. *Journal of Machine Learning Research*, 2024, 25(162): 1-36.\n\n2. Please refer to point 1 in the global rebuttal. \n\n3. Please refer to point 2 in the global rebuttal. \n\nResponse to questions:\n\n1. We believe that different target coverage rates would not affect the effectiveness of our method. We shall report additional results with different nominal miscoverage rates in our revised manuscript.\n\n2. We are somewhat confused about the reviewer's question. As mentioned previously, we use the training data in combination with cross validation during the boosting stage. Regarding the calibration set, as with any method that relies on a holdout set, an extremely small holdout set will inevitably lead to high variability in the performance of the method. This is not specific to our method, and in general we would not recommend relying on a small holdout set via any procedure, so we do not explicitly consider this question in our work. It is worth pointing out, however, that marginal coverage guarantees do hold regardless of sample size, since these results are “on average” and do not account for variability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rmk6eXTraK", "reviewer": "Reviewer_tKW7", "summary": "The paper proposes to utilize the training data and gradient boosting (of the model's predictions) to optimize loss functions related to conformal prediction interval length and conditional coverage with respect to the score function. The optimized score function is then used for \"plain\" CP usage, via calibration set.", "questions": "- What can you say about the computational complexity of the boosting procedure?\n\n- In practice, both conditional coverage and the interval length are likely to be important for a user.  \nTherefore, when you optimize the score for better conditional coverage report also the effect on the interval length.  \nSimilarly, when you optimize the score for better interval length report also the effect on the conditional coverage.  \n\n- Regarding the previous comment, when discussing these two properties it is worth mentioning the work:   \nDabah, L., & Tirer, T. (2024). \"On Temperature Scaling and Conformal Prediction of Deep Classifiers\". arXiv preprint arXiv:2402.05806.  \nthat shows a simple way to trade between them in CP methods for classification. \n\n- While I am aware that your approach do not require access to the model's parameters, it would be insightful to compare the approach (even at some of the settings) to the approach of [26], namely optimizing the model's parameters to minimize a loss that includes the term that you design.", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The approach is novel and significantly differs from other methods that utilize training data to improve CP performance.\n\n- The authors derive asymptotic theory for the sufficient expressiveness of the approach.\n\n- Experiments demonstrate that the approach consistently boosts the CP metric associated with the designed loss (interval length or conditional coverage).", "weaknesses": "- There is no discussion and investigation on the impact of optimizing the score for conditional coverage on the interval length and vise versa. In practice, both can be important for a user.\n\n- Even though there is theory of the \"asymptotic expressiveness\" of the approach, the method in practice is based on  differentiable approximations and greedy optimization that do not guarantee optimality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments. In response to each of the reviewer’s specific comments on weaknesses:\n\n- Please refer to point 1 in the global rebuttal.\n- Please refer to point 5 in the global rebuttal.\n\nResponse to questions:\n- Please refer to point 2 in the global rebuttal. \n- Please refer to point 1 in the global rebuttal.\n- We thank the reviewer for pointing us towards this literature. We shall include this in our revised manuscript when discussing the trade off between conditional coverage and interval length.\n- In our revised manuscript, we will include additional results comparing the performance of our method with the approach outlined in [26]. While [26] primarily focuses on reducing prediction set size for classification tasks, our work is centered on regression tasks. Another difference is that [26] considers conformity scores based solely on predicted probabilities for each class. In a regression context, this approach is analogous to using only $\\hat{y}$ or $\\mu$ in the Local conformity score. To bridge this gap, we plan to conduct an experiment using two neural networks to fit $\\mu$ and $\\sigma$ in the generalized Local conformity score in an alternating fashion. The objective will be the differentiable approximation of the average prediction interval length, as formulated below line 264. Originally, the training model's objective for fitting $\\mu$ was the mean squared error of $y-\\mu$, and for fitting $\\sigma$, it was the mean squared error of $|y-\\mu|-\\sigma$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pxkw7bbuLL", "reviewer": "Reviewer_D558", "summary": "This paper introduces a methodology for learning a conformal score function after training. Notably, the proposed method does not require model retraining, and instead learns the score function using trained model predictions using a cross-validation approach on the training data.", "questions": "- The boosting stage uses the same data on which the baseline score is trained. I'm curious whether the authors also considered splitting off some data for this stage (instead of doing the CV approach)? Although this would be less sample efficient, it seems like it may have greater benefit than reusing data for more powerful baseline score models.\n\n- It seems to me that the choice of boosting here for improving the baseline score could be replaced by some alternative means of mapping trained model predictions to the necessary mean/variance parameters for the conformal prediction intervals, and I'm wondering whether the authors considered such alternatives?", "limitations": "Although not in a separate section, limitations are adequately discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality:** Although the paper is in the same vein as works such as ConfTr [1], the fact that it can work directly on top of trained predictions on the same data is a clever formulation and a nontrivial benefit.\n\n**Quality:** Overall, the paper is technically sound; I have no concerns regarding the theory. My reservations regarding the empirical results are discussed under weaknesses.\n\n**Clarity:** The paper is well-written and all necessary ideas regarding conformal prediction for understanding the results in the paper are discussed appropriately.\n\n**Significance:** The idea of learning a conformal score function after training (using the same data) but before calibration is, to my knowledge, a novel formulation and worth building on.\n\n\n[1] https://arxiv.org/abs/2110.09192", "weaknesses": "**Marginal empirical benefit.** My main reservation with the proposed method is the seemingly marginal benefit it provides over CQR for conditional coverage, as shown in Table 1. This is especially relevant for large scale settings where the overhead of the boosting/CV stage is going to be highly nontrivial. Additionally, conditional coverage here is evaluated using contrast trees, which are baked into the objective of the boosting approach. It would be interesting to see if the improvements hold under, for example, just group-conditional coverage.\n\n**Baseline score functions.** The baseline score functions as described in Appendix A.8 are relatively weak, untuned models. While I do not expect an expansive evaluation of models to be within the scope of this work, it would be good to know how sensitive the improvements in the paper are to the choice of baseline score model.\n\nOverall, while I have reservations regarding the empirical results, I do find the formulation of learning the conformal score in this paper useful. As a result, I lean accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time spent reviewing the manuscript. In response to each of the reviewer’s specific comments on weaknesses:\n1. \n> My main reservation with the proposed method is the seemingly marginal benefit it provides over CQR for conditional coverage, as shown in Table 1. \n\nPlease refer to point 3 in the global rebuttal.\n\n> Additionally, conditional coverage here is evaluated using contrast trees, which are baked into the objective of the boosting approach.\n\nWe would also like to clarify that while we use contrast trees in our loss function, we execute the contrast tree algorithm at each iteration of gradient boosting. In other words, we re-partition the feature space at each iteration as soon as the conformity score is updated. When we evaluate performance on the test set, we rerun the contrast tree algorithm on the test set, taking as inputs the features, the labels and the conformalized prediction intervals calculated on the test set. Therefore, the partition of the feature space at test time is different from that at boosting time.\n\n> It would be interesting to see if the improvements hold under, for example, just group-conditional coverage.\n\nPlease refer to point 4 in the global rebuttal.\n\n2. We only partially agree with the reviewer that the improvement depends on the accuracy of the underlying model. Imagine we have infinite training data and that the underlying model perfectly identifies the conditional mean $E[Y|X]$ and the conditional expected absolute deviation $\\text{MAD}(Y|X)$. Then plugging this in the local conformity score would not be optimal in terms of average interval length or conditional coverage. Below, we illustrate this with a simulated example. Assume that each X follows a uniform distribution Unif(0.1,1.1), and that $Y = X\\cdot e$, where e is a random variable distributed as $\\text{Beta}(a,b)-a/(a+b)$; this says that e follows a beta distribution shifted to have mean zero. We can theoretically calculate $E[Y|X]=0$, $\\text{MAD}(Y|X) = X\\cdot 2a^ab^b/\\left(B(a,b)\\cdot (a+b)^{(a+b+1)}\\right)$. Here, $B(\\cdot,\\cdot)$ stands for the beta function. However, because a beta distribution is not symmetric, even if the training model perfectly fits $E[Y|X]$ and $\\text{MAD}(Y|X)$, the conformalized prediction intervals will not be of optimal length, as shown in the left panel in the attached pdf. If we take the theoretical mean and MAD as trained, then the boosted Local score effectively adjusts for the non-symmetric nature of the underlying distribution, as shown in the right panel. The respective average conformalized interval lengths are 0.237 and 0.216. This is about a 10% reduction in size. \n\nResponse to questions:\n\n1. We agree with the reviewer that an alternative approach could involve holding out a validation set in addition to the training, calibration, and test sets. However, as noted by the reviewer, this method presents a trade off: a small validation set can lead to high variability in prediction intervals and performance, while a large validation set may reduce the number of samples available for training, potentially limiting the model's effectiveness. In contrast, our procedure uses a form of cross-validation to avoid this trade off. To illustrate this, we will include experiments comparing the performance of the two procedures on real datasets in the revised manuscript.\n\n2. We agree with the reviewer that the gradient boosting algorithm could potentially be replaced by other machine learning models. In this sense, the gradient boosting algorithm is not central to our contribution. Within our boosting framework, once the appropriate search space and loss function are established, any suitable machine learning model can, in principle, be used to search for an enhanced conformity score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2M6TYQmI5A", "reviewer": "Reviewer_3gAf", "summary": "This paper introduces a gradient boosting-based approach to tailor a conformal score function to better satisfy desirable properties. A more general score function \"family\" is introduced which adheres to a specific form dependent on parameters (e.g. $\\mu, \\sigma$). Instead of directly using the parameter estimates obtained from model training of the predictor $f$, a post-hoc boosting stage is introduced in which the parameters are further refined leveraging additional data to minimize property-specific losses, such as conditional coverage deviation or conformal interval length. After boosting for $\\tau$ iterations, the refined parameter estimates $\\mu^{(\\tau)}, \\sigma^{(\\tau)}$ can be used for down-stream conformal calibration and prediction interval generation. Results are compared to default local conformal scores (Local) and CQR on a range of regression tasks, obtaining slightly more favorable empirical properties.", "questions": "- Could you please comment on some of the raised points above, such as on clarifying the data used by the boosting procedure, the applicability of the \"contrast tree\" method, the practicality of defining new optimization objectives, or the distinction to other existing methods aiming for conditional coverage.\n- The approach to obtain a differentiable loss objective from Eq. 14 seems quite involved. Could you comment on the motivation of leveraging \"contrast trees\" and the tools used to go from Eq. 14 to a differentiable loss? \n- Are there any results that can be made on how the boosting procedure affects down-stream conformal coverage guarantees, or statements of optimality or improvement with regards to the obtained \"boosted\" conformity scores over the default ($0$-th iteration) conformity scores?", "limitations": "I did not find the limitations of the method thoroughly discussed anywhere, and I believe the distinction to other related works could be improved. There are some comments made in the discussion on improvements, as well as L218-L222 on not guaranteeing an optimal solution.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Addressing the practicality of obtained conformal sets is a relevant problem, since results can sometimes be overly conservative to adhere to the conformal guarantees\n- The proposed method seems relatively lightweight and has the benefit of not requiring changes to the model weights of $f$, since the boosting derivatives are computed w.r.t. the inputs of the scoring function $E$, which are the predictions $\\mu(x), \\sigma(x)$\n- The improvements for the local conformal score function seem substantial at times", "weaknesses": "My main concerns are in regards to evaluation and practicality. The authors propose two applications of the boosting procedure to improve conditional coverage and prediction set size. For conditional coverage, the boosting loss is introduced as a relatively complicated group deviation metric leveraging a particular \"contrast tree\" mechanism, and in practice requires further non-trivial approximations to make it differentiable for boosting. Since the results are very strongly tied to this \"contrast tree\" approach, I was struggling to see how the boosting loss and conditional coverage target (Eq 12) connect in practice and how the results (e.g. Fig. 3) are more widely applicable, since the procedure is motivated as an \"effective evaluation metric (for target conditional coverage) of independent interest\". Is this procedure amenable to any black-box predictor $f$ and can be considered a particular gradient boosting mechanism? In general, I found the obtained improvements not overly convincing. For example, in Fig. 3 if we consider the % of data for which conditional coverage ends up being violated (above the red line) we find 33% for Local and 37% for Boosted Local. Granted, the boosting seems to distribute miscoverage somewhat more evenly, but the overall violation rate remains similar. In Table 1, boosting CQR only gives marginal improvements. It would be good to compare the results to other conformal algorithms which claim to empirically (or sometimes even theoretically, under relaxed conditions or varying interpretations of conditionality) improve conditional coverage, e.g. [1,2,3,4]. This would help clarify if the approach is more widely beneficial. Similarly, all experiments are run leveraging the same random forest and quantile network regressors -- do results translate and are amenable across different models $f$ as well? Another hurdle might be the fact that a differentiable property-specific loss needs to be defined for every target property, whose amenability may also depend in some way on the underlying model's design, and is fixed. Thus, tackling a more flexible scoring function comes at the cost of introducing another (fixed) functional to optimize over, with required approximations. \n\nOn another note, I was somewhat struggling to understand how the boosting procedure connects to the overall conformal procedure. In Fig. 1 it is suggested that boosting is an independent step inbetween model training and conformalization, but in the experiment description it seems that boosting leverages existing datasets (namely training data). Should I understand that the boosting dataset $\\mathcal{D}$ is a separate set? If so, how do its sampling requirements (e.g. exchangeability) connect to calibration and test data. If not, should I understand that the boosting step actually equates to an iterative conformal calibration procedure leveraging $\\mathcal{D}_{cal}$? Since the boosting objectives (Eq. 14, 17) actually require computing conformal sets and quantiles. Thank you for clarifying.\n\nOther comments:\n- I believe the $j$ index in the 3rd final line of Algorithm 1 should be omitted, since the final boosted score functions are not taken w.r.t. a particular fold.\n- Fig. 4 is missing a legend, making it hard to interpret\n- The use of the term \"Power Loss\" seems odd, since the goal is to reduce prediction set size, and from my understanding a smaller prediction set/interval will equate higher power, since the permutation testing interpretation of the null hypothesis is on inclusion in the set.\n- Theorem 5.1 does not seem like a theorem but more like a remark to me, since it is merely suggesting that the \"generalized\" score family may contain the optimal oracle solution, but in no way relates any optimality to the practically obtained solutions via the boosting procedure.\n- It would be beneficial to use hyper-referencing for Equations and the Algorithm throughout the paper.\n\nReferences\n- [1] Romano, Yaniv, et al. \"With malice toward none: Assessing uncertainty via equalized coverage.\" Harvard Data Science Review 2.2 (2020): 4.\n- [2] Gibbs, Isaac, John J. Cherian, and Emmanuel J. Candès. \"Conformal prediction with conditional guarantees.\" arXiv preprint arXiv:2305.12616 (2023).\n- [3] Sesia, Matteo, and Yaniv Romano. \"Conformal prediction using conditional histograms.\" Advances in Neural Information Processing Systems 34 (2021): 6304-6315.\n- [4] Jung, Christopher, et al. \"Batch multivalid conformal prediction.\" arXiv preprint arXiv:2209.15145 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments. In response to each of the reviewer’s specific points:\n\n> Is this procedure amenable to any black-box predictor and can be considered a particular gradient boosting mechanism? \n\n- As an evaluation metric, the contrast tree algorithm applies to any black-box predictor f, as it only requires three inputs, the features, the labels and the conformalized prediction intervals, which can be derived from the conformity scores calculated via the values the model predicts. Then the contrast tree algorithm automatically detects regions, where coverage is most uneven.\n\n> In general, I found the obtained improvements not overly convincing. For example, in Fig. 3 if we consider the % of data for which conditional coverage ends up being violated (above the red line) we find 33% for Local and 37% for Boosted Local.\n\n- Assuming exchangeability, conformal prediction ensures that the average miscoverage rate over the test set is approximately 10%. The dream would be to achieve exactly 10% miscoverage within any subset of the feature space. While this might seem counterintuitive, the guarantee provided by conformal prediction implies that if one subset in the feature space has a miscoverage rate smaller than 10%, another disjoint subset must have a miscoverage rate greater than 10%. This results in prediction intervals that sometimes overcover and sometimes undercover. Figure 3 illustrates that before boosting, the conditional coverage in each leaf significantly deviates from the target rate of 10%; it is either too large or too small. After boosting, however, all groups have coverage rates that hover around 10%. Concretely, before boosting, the contrast tree identifies splits in the feature space that deviate from the 10% miscoverage rate by 15%. After boosting, the largest deviation identified by the contrast tree is reduced to no more than 6%.\n\n> In Table 1, boosting CQR only gives marginal improvements.\n\n- Please refer to point 3 in the global rebuttal.\n\n> It would be good to compare the results to other conformal algorithms which claim to empirically (or sometimes even theoretically, under relaxed conditions or varying interpretations of conditionality) improve conditional coverage, e.g. [1,2,3,4].\n\n- In comparison to other works aiming to improve conditional coverage, such as [1] and [2] mentioned by the reviewer, our approach offers a distinct advantage.  Those works measure group conditional coverage based on user-specified groups. However, relying on predefined groups may not always be ideal. Specifically, please refer to point 4 in the global rebuttal. \n\n>Thus, tackling a more flexible scoring function comes at the cost of introducing another (fixed) functional to optimize over, with required approximations.\n\nWe agree with the reviewer that optimizing a targeted property requires a customized objective. We would like to highlight that the conformity score functions we discuss in the manuscript are arguably the two most widely used score functions for regression tasks. Although we do not address classification tasks in this project, it is worth noting that commonly used score functions in that domain, such as the threshold conformal predictor (THR) and adaptive prediction sets (APS), are also differentiable by design.\n\n> Should I understand that the boosting dataset is a separate set?\n\n- During the boosting stage, we use the training set to simulate the conformal procedure, assessing the corresponding interval length and conditional coverage. Specifically, at each iteration of cross-validation, we divide the training set into a sub-boosting set and a sub-validation set (the held-out fold). The sub-boosting set serves as both the boosting set and the calibration set, while the sub-validation set acts as the test set. Please refer to Figure 2 for an illustrative schematic drawing. We shall work to explain this more clearly in the revised manuscript.\n\n\nResponse to other comments:\n\n- We thank the reviewer for pointing out the typo in the 3rd final line of Algorithm 1. We shall fix this in the revised manuscript.\n- We shall add a legend for clarity of presentation in the revised manuscript.\n- We agree with the reviewer that a smaller prediction set or interval corresponds to higher power. In the manuscript, by \"power loss,\" we intended to convey \"loss of power,\" meaning that when the interval length is larger, we lose power. If the reviewer finds this phrasing misleading, we will rephrase it in the revised manuscript.\n- We agree with the reviewer that Theorem 5.1 only proves that the search space contains the optimal solution, but does not guarantee the optimality of the numerical boosting procedure.\n- We shall add hyper-referencing in the revised manuscript.\n\nResponse to questions:\n- Please see our reply to the 1, 2, 3, 5 and 6th items in the Weaknesses section.\n- For the motivation behind leveraging contrast trees, please refer to point 4 in the global rebuttal. One of the primary reasons we chose to approximate the empirical quantile using the Harrel-Davis quantile estimator is that it transforms the original linear combination of two samples into linear combinations of all samples, significantly enhancing the robustness of the optimization objective.\n- Please refer to point 5 in the global rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful to all the reviewers for their valuable feedback and constructive suggestions for improving our manuscript. In the rebuttals below, we have responded to each reviewer’s point individually. The attached PDF includes additional simulation results. Below, we address several common points raised by the reviewers:

**1. Trade-off between conditional coverage and interval length (trTq,tKW7).**

We agree with the reviewers that in practice, both conditional coverage and interval length are likely to be important. In the revised manuscript, we will expand Tables 1 and 2 to report additional results on both conditional coverage and interval length to address this concern.  

**2. Computational complexity of our boosted conformal procedure (trTq,tKW7).**

- The computational complexity of our procedure stems from three main components: 1)  evaluating the customized loss function at each boosting iteration, 2) using the gradient boosting algorithm to update the conformity score function, and 3) performing cross-validation. We acknowledge the importance of improving the computational efficiency of our algorithm while maintaining or even enhancing its performance. One potential improvement could involve reducing runtime by replacing cross-validation with a simpler approach, namely, splitting the training set into a boosting set and a validation set. However, as discussed in our response to question 1 raised by reviewer D558, this simplification may introduce trade offs between the variability of the prediction intervals and the effectiveness of the training model. Additionally, in our current implementation, the contrast tree algorithm is executed at every iteration of gradient boosting to identify regions in the feature space that deviate from the target coverage rate. A more computationally efficient strategy might involve partitioning the feature space into regions that maximize deviation at specific intervals during the boosting iterations, such as every 5 or 10 iterations. In between these iterations, we would retain the partition computed last, and use it to evaluate performance. 

- In our revised manuscript, we will report a more detailed comparison of the runtime between the unboosted split conformal procedure and our boosted procedure.

**3. Modest improvement on conditional coverage for CQR (D558,3gAf).**

We acknowledge the observation from the reviewers that the improvement our procedure affords over CQR is generally somewhat limited. However, we would like to emphasize a few points: first, applying boosting after running CQR does not degrade its performance. Second, the experiments we presented on the selected datasets are by no means exhaustive. It is possible that one might see substantial improvements on datasets where the base CQR learner is less than ideal. Finally, our main goal is to demonstrate the flexibility of our method, highlighting its adaptability to various conformity scores and loss functions. While the improvement in conditional coverage may be modest in the specific context of this paper, we anticipate potential benefits in other applications. For instance, we foresee extending this approach to boost for the power of multiple hypothesis testing while controlling for the false discovery rate (FDR), particularly in tasks such as drug discovery and candidate screening. Following the framework proposed by [1], we would first calculate conformal p-values from the conformity scores associated with each sample and then apply the Benjamini-Hochberg procedure to control for FDR. P-values depend on the choice of conformity scores and optimizing these scores would yield lower p-values and hence more rejections. Similarly, conformal methods have been used for outlier detection and boosting conformity scores to improve the ROC (receiver operating characteristic) curve would equally be of great interest. 

**4. Comparison between the contrast tree algorithm and the group conditional coverage metric (D558, 3gAf).**

We would like to argue that one of the key innovations here is the ability to identify regions in the dataset without requiring users to pre-specify them. By employing the contrast tree algorithm, we can automatically detect regions where coverage is most uneven. This feature is particularly beneficial if the user's objective is to achieve even conditional coverage across the feature space. Relying on user-specified groups may not always be appropriate, as it might overlook subgroups within these predefined categories where conditional coverage significantly deviates from the target rate, even if coverage appears even across the broader groups. 

**5. Clarifications on theoretical guarantees (tKW7,3gAf).**

We agree with the reviewers that we currently cannot give any formal guarantee on the improvement our algorithm affords. We can only claim the following: 1) under the usual exchangeability assumption, our boosted procedure has the same coverage guarantees as the split conformal procedure on the test set; and 2) the search space contains the optimal solution. We can of course not guarantee that our algorithm will find this optimal solution. 

References: 

[1] Jin Y, Candès E J. Selection by prediction with conformal p-values. *Journal of Machine Learning Research*, 2023, 24(244): 1-41.

Meta-review: This paper presents a novel approach to enhancing conformal prediction intervals through a post-training boosting process, which offers substantial flexibility and robustness. While some concerns were raised regarding the empirical benefits and the use of the contrast tree approach, the authors' thorough responses and proposed revisions have addressed these points adequately. Given the innovative contributions and the positive reception from reviewers, I recommend accepting this paper.
Decision: Accept (poster)