Title: Boosted Conformal Prediction Intervals
Venue: NeurIPS 2024
Accepted: yes
Keywords: Conformal Prediction

Uncertainty Quantification

(Other) Statistical Learning
Abstract: This paper introduces a boosted conformal procedure designed to tailor conformalized prediction intervals toward specific desired properties, such as enhanced conditional coverage or reduced interval length. We employ machine learning techniques, notably gradient boosting, to systematically improve upon a predefined conformity score function. This process is guided by carefully constructed loss functions that measure the deviation of prediction intervals from the targeted properties. The procedure operates post-training, relying solely on model predictions and without modifying the trained model (e.g., the deep network).  Systematic experiments demonstrate that starting from conventional conformal methods, our boosted procedure achieves substantial improvements in reducing interval length and decreasing deviation from target conditional coverage.

Boosted Conformal Prediction Intervals

Abstract

This paper introduces a boosted conformal procedure designed to tailor conformalized prediction intervals toward specific desired properties, such as enhanced conditional coverage or reduced interval length. We employ machine learning techniques, notably gradient boosting, to systematically improve upon a predefined conformity score function. This process is guided by carefully constructed loss functions that measure the deviation of prediction intervals from the targeted properties. The procedure operates post-training, relying solely on model predictions and without modifying the trained model (e.g., the deep network). Systematic experiments demonstrate that starting from conventional conformal methods, our boosted procedure achieves substantial improvements in reducing interval length and decreasing deviation from target conditional coverage.

1 Introduction

Black-box machine learning algorithms have been increasingly employed to inform decision-making in sensitive applications. For instance, deep convolutional neural networks have been applied to diagnose skin cancer [14], and AlphaFold has been utilized in the development of malaria vaccines [24, 25]; here, scientists have employed AlphaFold to predict the structure of a key protein in the malaria parasite, facilitating the identification of potential binding sites for antibodies that could prevent the transmission of the parasite [25]. These instances highlight the critical need for understanding prediction accuracy, and one popular approach to quantify the uncertainty associated with general predictions relies on the construction of prediction sets guaranteed to contain the target label or response with high probability. Ideally, we would like the coverage to be valid conditional on the values taken by the features of the predictive model (e.g., patient demographics).

Conformal prediction [3] stands out as a flexible calibration procedure that provides a wrapper around any black-box prediction model to produce valid prediction intervals. Imagine we have a data set { ( X i , Y i ) } n i =1 and a test point ( X n +1 , Y n +1 ) drawn exchangeably from an unknown, arbitrary distribution P (e.g. the pairs ( X i , Y i ) may be i.i.d.). Taking the data set and the observed features X n +1 as inputs, conformal prediction forms a prediction interval C n ( X n +1 ) for Y n +1 with valid marginal coverage, i.e. such that /a80 ( Y n +1 ∈ C n ( X n +1 )) = 0 . 95 or any nominal level specified by the user ahead of time. This is achieved by means of a conformity score E ( x, y ; f ) , where ( x, y ) represents a data point while f represents any aspects of the distribution that we have estimated. For instance, the score may be given by the magnitude of the prediction error | y -ˆ µ ( x ) | , where ˆ µ ( x ) represents the model prediction of the expected outcome, in which case f is simply ˆ µ . Roughly, we

Training

conformity score

Boosting

boosted score

Calibration

evaluate scores

Testing

conformalized

prediction intervals

would include y in the prediction interval if E ( X n +1 , y ; f ) does not take on an atypical value when compared with { E ( X i , Y i ; f ) } , i = 1 , . . . , n . Selecting an appropriate conformity score is akin to choosing a test statistic in statistical testing, where two statistics may yield the same Type I error rate yet differ substantially in other aspects of performance.

One central issue is that while the conformal procedure guarantees marginal coverage, it does not extend similar guarantees to other desirable inferential properties without additional assumptions. In response, researchers have introduced a variety of conformity scores, including the locally adaptive (Local) conformity score [16], the conformalized quantile regression (CQR) conformity score [18], and its variants, CQR-m [22] and CQR-r [23]. Among these, CQR has often demonstrated superior empirical performance in terms of both interval length and conditional coverage [18].

This paper introduces a boosting procedure aimed at enhancing an arbitrary score function. 1 By employing machine learning techniques, namely, gradient boosting, our objective is to modify the Local or CQR score functions (or other baselines) to reduce the average length of prediction intervals or improve conditional coverage while maintaining marginal coverage. While this paper focuses primarily on length and conditional coverage, our methods can be tuned to optimize other criteria; we elaborate on this in Section 7.

Our boosted conformal procedure searches within a family of generalized scores for a score achieving a low value of a loss function adapted to the task at hand. Specifically, to evaluate the conditional coverage of prediction intervals, we build a loss function that maximizes deviation from the target coverage rate in the leaves of a shallow contrast tree [21]. Searching within a strategically designed family of score functions, rather than directly retraining or fine-tuning the fitted model under the task-specific loss function, yields greater flexibility and avoids the costs associated with retraining or fine-tuning. Further, this boosting process is executed post-model training, requiring only the model predictions and no direct access to the training algorithm.

Source code for implementing the boosted conformal procedure is available online at https:// github.com/ran-xie/boosted-conformal . Details regarding the acquisition and preprocessing of the real datasets are also provided in the GitHub repository.

2 The split conformal procedure

We begin by outlining the key steps of the split conformal procedure applied to a family { ( X i , Y i ) } n i =1 of exchangeable samples (e.g., i.i.d.).

Training. Randomly partition [ n ] into a training set I 1 and a calibration set I 2 . On the training set, train a model by means of an algorithm ❆ to produce a conformity score function E ( · , · ; f ) . The structure of this score function is predetermined, whereas the model f is learned from ❆ . An example of a conformity score is E ( x, y ; f ) = | y -ˆ µ ( x ) | , where ˆ µ ( x ) is a learned regression function so that f is here simply ˆ µ .

Calibration. Evaluate the function E ( · , · ; f ) on each instance in the calibration set and obtain scores { E i } i ∈ I 2 , 2 with each E i = E ( X i , Y i ; f ) . The (1 -α ) th empirical quantile

of the score, Q 1 -α ( E,I 2 ) , is calculated as

where Z follows the distribution 1 | I 2 | +1 ( δ ∞ + ∑ δ E i ) , and δ a is a point mass at a .

Testing. For a new observation X n +1 , output the conformalized prediction interval

If ties between { E i } i ∈ I 2 occur with probability zero, it holds that

see [16]. By introducing additional randomization during the calibration step, the prediction interval can be tuned to obey /a80 ( Y n +1 ∈ C n ( X n +1 )) = 1 -α , see [4]. This adjustment is not critical here and we omit the details.

Locally adaptive conformal prediction (Local for short) [16] introduces a score function that aims to make conformal prediction adapt to situations where the spread of the distribution of Y varies significantly with the observed features X . On the training set, run an algorithm ❆ to fit two functions µ 0 ( · ) and σ 0 ( · ) , where µ 0 ( X ) estimates the conditional mean E [ Y | X ] , and σ 0 ( X ) the dispersion around the conditional mean, frequently chosen as the conditional mean absolute deviation (MAD), E [ | Y -µ 0 ( X ) | | X ] . With f = ( µ 0 , σ 0 ) , the locally adaptive (Local) score function is:

For a new observation X n +1 , the conformalized prediction interval (1) takes on the simplified expression [ µ 0 ( X n +1 ) -Q 1 -α ( E,I 2 ) σ 0 ( X n +1 ) , µ 0 ( X n +1 ) + Q 1 -α ( E,I 2 ) σ 0 ( X n +1 )] .

Conformalized quantile regression (CQR) [17] also aims to adapt to heteroskedasticity by calibrating conditional quantiles, which often results in shorter prediction intervals. Apply quantile regression to produce a pair of estimated quantiles (ˆ q α/ 2 ( x ) , ˆ q 1 -α/ 2 ( x )) , where ˆ q β ( X ) is the estimated β th quantile of the conditional distribution of Y . The CQR score function is defined as

where f = (ˆ q α/ 2 , ˆ q 1 -α/ 2 ) . For a new observation X n +1 , following (1) yields the prediction interval

Generalized conformity score families . To construct a Local conformity score, we estimate two functions µ 0 ( · ) and σ 0 ( · ) to plug into (3). Since these components are constructed without looking at performance downstream, it is reasonable to imagine that other choices may enjoy enhanced properties. How then should we systematically select µ ( · ) and σ ( · ) ? To address this, we define a generalized Local score family F containing all potential score functions of the form

where f = ( µ, σ ) . For each E ( · , · ; f ) ∈ F , the conformalized prediction interval is given by

Turning to CQR, one notable limitation is the uniform adjustment of prediction intervals by the constant factor Q 1 -α ( E,I 2 ) , as shown in (5). This approach is suboptimal in the presence of heteroskedasticity, as it applies an identical correction to prediction intervals of varying widths for each X = x . Thus, simply updating the fitted quantiles (ˆ q α , ˆ q 1 -α/ 2 ) and plugging them into the original score function would be inadequate, as the structure of the original score imposes significant limitations on the effectiveness of conformalized prediction intervals. To address this, several variants including CQR-m [22] and CQR-r [23] have been proposed. Focusing on CQR-r, it employs a flexible score function, defined as E ( x, y ; f ) = max { ˆ q α/ 2 ( x ) -y, y -ˆ q 1 -α/ 2 ( x ) } / (ˆ q 1 -α/ 2 ( x ) -ˆ q α/ 2 ( x )) , with f = (ˆ q α/ 2 , ˆ q 1 -α/ 2 , ˆ q 1 -α/ 2 -ˆ q α/ 2 ) . Following (1), conformalized prediction intervals become

where ˆ σ = ˆ q 1 -α/ 2 -ˆ q α/ 2 . Intuitively, the adjusted score function allows prediction bands to adjust in proportion to their width, instead of adding a constant shift as in CQR. However, despite the intuitive appeal of adjusted scores as a seemingly more reasonable 'allocation' of the conformal correction, empirical studies reveal that they do not result in narrower prediction intervals when compared to CQR [23]. This phenomenon is largely due to the uniform direction of the conformal adjustment, represented by Q 1 -α ( E,I 2 ) , across all observations. In particular, if Q 1 -α ( E,I 2 ) < 0 , indicating that the true target y predominantly lies within the estimated quantile range [ˆ q α/ 2 , ˆ q 1 -α/ 2 ] , there is a uniform narrowing of the predicted interval across all samples.

In light of these insights, we propose a novel score family, H , designed to augment the flexibility of the conformity score functions:

where f = ( µ 1 , µ 2 , σ ) , which leads to conformalized prediction intervals of the form

Notably, H includes the Local, CQR, and CQR-r scores as special cases.

3 Boosted conformal procedure

It is clear from above that a model is trained to produce a conformity score E ( · , · ; f ) ; e.g., we may learn a regression function ˆ µ ( · ) to plug it into a score function | y -ˆ µ ( x ) | . To overcome the limitation of working with an arbitrarily selected score function, we introduce a boosting step before calibration, see Figure 1. In a nutshell, we use gradient boosting to iteratively improve upon a predefined score E ( · , · ; f ) now denoted as E (0) ( · , · ) , where the superscript indicates the 0 th iteration.

To achieve this, we construct a task-specific loss function ℓ , which takes a dataset D and a score function E ( · , · ; f ) as inputs, and outputs ℓ ( E ( · , · ; f ); D ) measuring how closely the conformalized prediction interval aligns with the analyst's objective. This loss function ℓ is designed to be differentiable with respect to each of the model components produced by the training algorithm. Importantly, it does not require knowledge of the gradient of f ( x ) with respect to x . In the example above, taking the labels as fixed, this means that for each feature x i ∈ D , i = 1 , . . . , n , if we set ˆ y i = ˆ µ ( x i ) , then the loss ℓ ( E ( · , · ); D ) is a function of { ˆ y i } n i =1 , and the derivative ∂ℓ ( E ( · , · ); D ) /∂ ˆ y i is well defined. In Sections 5.1 and 6.1, we present examples of such derivatives.

Each boosting iteration updates the score function sequentially, employing a gradient boosting algorithm such as XGBoost [12] or LightGBM [15]. These algorithms accept as input a dataset D , a base score function E ( · , · ; f ) , a custom loss function ℓ , gradients of ℓ with respect to f (denoted ∇ f ℓ ), and a number of boosting rounds τ . We may write the boosting procedure as

This yields a boosted score function E ( τ ) ( · , · ) , which is then used for calibration and for constructing prediction intervals. The number τ is calculated using k -fold cross-validation on the training dataset, selecting τ from potential values up to a predefined maximum T (e.g., 500). We partition the dataset into k folds and for each j = 1 , . . . , k , we hold out fold j for sub-calibration and the remaining k -1 folds for sub-training. We apply T rounds of gradient boosting (11) on the sub-training data, generating T + 1 candidate score functions E (0) j ( · , · ) , . . . , E ( T ) j ( · , · ) . Each score function is then evaluated on sub-calibration data, using the loss function ℓ to compute losses at all epochs, i.e., for each fold j = 1 , . . . k ,

Last, τ is selected as the round that minimizes the average loss across all k folds:

see Figure 2. This cross-validation step simulates the calibration step in conformal prediction and effectively prevents the overfitting of the score function.

Since boosting is conducted on the training data, the boosted procedure satisfies the same marginal coverage guarantee as the split conformal procedure, as formalized below.

-fold Cross Validation

fold j

boost rounds

0

best-round

Proposition 3.1. Let { ( X i , Y i ) } i m =1 be the held out calibration set, and ( X m +1 , Y m +1 ) be a pair of new observation. If the m + 1 samples are exchangeable, and ties between { E ( τ ) ( X i , Y i ) } i m =1 occur with probability zero, the confromalized prediction interval (1) computed from score function E ( τ ) ( · , · ) satisfies the coverage guarantee (2).

Searching within generalized conformity score families . To update the Local score function (3), we search within the generalized score family F (6). First, we initialize µ (0) = µ 0 and σ (0) = σ 0 . After completing τ iterations of boosting on the training set, we obtain the boosted score function E ( τ ) ( x, y ) = | y -µ ( τ ) ( x ) | /σ ( τ ) ( x ) . Notably, we can update any score function within F . For instance, to update E ( x, y ; f ) = | y -ˆ µ ( x ) | , we simply initialize µ (0) = ˆ µ , and take σ (0) to be the constant function equal to one. Similarly, to update the CQR score function (4), we search within the score family H (9). First, we initialize a triple µ (0) 1 = ˆ q α/ 2 , µ (0) 2 = ˆ q 1 -α/ 2 , σ (0) = ˆ q 1 -α/ 2 -ˆ q α/ 2 . After τ boosting rounds, we obtain the boosted score function E ( τ ) ( x, y ) = max { µ ( τ ) 1 ( x ) -y, y -µ 2 ( τ ) ( x ) } /σ ( τ ) ( x ) .

Algorithm 1 Boosting stage

Input:

Training data ( X , Y ) ∈ ❘ p × ❘ , i = 1 , ..., n ; base conformity score function E (0) ( · , ·

Loss function ℓ ; target mis-coverage level α ∈ (0 , i i ) 1)

Number k of cross-validation folds; maximum boosting rounds T

Procedure:

Randomly divide { 1 , ..., n } into k folds for j ← 1 to k do

Set fold j as sub-calibration set, and the remaining k -1 folds as sub-training set

On the sub-training set, call GradientBoosting (11) to obtain candidate scores { E ( t ) j } T t =0

end for

Set boosting rounds τ ← arg min t 1 k ∑ k j =1 L ( t ) j as in (12)

On the training set, call GradientBoosting (11) to obtain boosted functions { E ( t ) } τ t =0

Output:

Boosted conformity score function E ( τ ) ( · , · )

4 Related Works

Adapting the classical conformal procedure to improve properties of the conformalized intervals has been one of the primary focuses of recent literature. Noteworthy contributions-including CFGNN[28] and ConTr [26]-approach this problem by introducing modifications to the training stage

of the procedure. As outlined in Section 2, a model is trained to produce a score function E ( · , · ; f ) . The model f usually depends on a set of model parameters, e.g., neural network parameters θ . Denote the trained model f by f θ . CF-GNN and ConTr retrain or fine-tune the model by using a carefully constructed loss function, which may aim to produce narrower prediction intervals or prediction sets of reduced cardinality in classification problems. This process generates a new set of model parameters θ ′ . The new model f θ ′ is then plugged into the same predefined conformity score function-namely CQR [28] or the adaptive prediction set score (APS) [26]-to produce E ( · , · , f θ ′ ) .

There are two primary limitations. First, the score function imposes constraints on the properties of conformalized intervals as explained in Section 2. Our approach introduces more flexibility by constructing a family of generalized score functions that is a superset of { E ( · , · ; f θ ) : θ ∈ Θ } , where Θ is the parameter space of the training model. This family is strategically designed to contain an oracle conformity score ideally suited to the task at hand, e.g., achieving exact conditional coverage. Second, current methodologies necessitate fine-tuning or retraining models from scratch, requiring both access to the training model and significant computational resources. In contrast, our boosted conformal method operates directly on model predictions and circumvents these issues.

Conditional coverage of conformalized prediction intervals has also attracted significant interest, characterized by efforts to establish theoretical guarantees and achieve numerical improvements. Prior work established an impossibility result [8, 20], which states that exact conditional coverage in finite samples cannot be guaranteed without making assumptions about the data distribution. Subsequently, Gibbs et al. [27] developed a modified conformal procedure that guarantees conditional coverage for predefined protected sub-groups, i.e. subsets of the feature space. Our approach differs from the previous works by introducing a numerical method directly aimed at improving the conditional coverage, /a80 ( Y ∈ C n ( X ) | X = x ) , across all potential values of x .

5 Boosting for conditional coverage

Maintaining valid marginal coverage, our goal is to produce a prediction interval C n obeying

for all possible values of x . To this end, we present a loss function that quantifies the conditional coverage rate of any prediction interval. Requiring merely a dataset D and a prediction interval C n ( · ) as inputs, it also serves as an effective evaluation metric, which may be of independent interest.

5.1 A measure for deviation from target conditional coverage

From now on, we let E be the score function E ( · , · ; f ) . Set D = { ( X i , Y i ) } n i =1 and denote by C n ( · ) the conformalized prediction interval constructed from E . We shall assess the deviation of C n ( · ) from the target conditional coverage by means of Contrast Trees [21]. As background, a contrast tree iteratively identifies splits within the feature space X in a greedy fashion, aiming to maximize absolute within-group deviations from the target conditional coverage rate (1 -α ) . For a subset R of the data point indices [ n ] , let D R = { X j , Y j } j ∈ R . The absolute within-group deviation is computed as

The overall empirical maximum deviation is then defined as

where ˆ R 1 ∪ · · · ∪ ˆ R M is a partition of [ n ] , which itself depends on E and D . Specifically, it is computed by running a contrast tree for M iterations. At each iteration, the algorithm not only seeks to isolate regions with large deviations but also discourages splits where any subset ˆ R m is too small.

To update score functions via gradient boosting as described in (11), we would need a differentiable approximation of the maximum deviation. To this end, we construct approximations for the following three components of the loss function. With an abuse of notation, in subsequent discussions, we shall employ the same notations to denote these differentiable approximations.

Approximation for the prediction interval C n ( · ) in (14): the prediction interval is formulated as (7) for the generalized Local score, and as (10) for the generalized CQR score.

Denote the upper and lower limits of C n ( · ) by u ( · ) and l ( · ) . We approximate the empirical quantile Q 1 -α ( E,I 2 ) in u ( · ) and l ( · ) with a smooth quantile estimator Q s 1 -α . Given r scalars { z i } r i =1 , Q s 1 -α is constructed as:

where ⟨· , ·⟩ represents the dot product. Here, HD ( r ) = [ W r, 1 , ..., W r,r ] is the weight vector corresponding to the Harrel-Davis distribution-free empirical quantile estimator [1], and s ( z ) is a differentiable ordering { ˜ z ( i ) } r i =1 , arranged in the ascending order. In practice, the derivative of s ( z ) with respect to each z i is given by the package developed in [19]. This approach is a smooth approximation of the Harrel-Davis quantile estimator Q HD 1 -α , constructed as a linear combination of the order statistics, Q HD 1 -α = ⟨ HD ( r ) , { z ( i ) } ) ⟩ = ∑ r i =1 W r,i z ( i ) , where W r,i takes the value I (1 -α )( r +1) ,α ( r +1) ( i/r ) -I (1 -α )( r +1) ,α ( r +1) (( i -1) /r ) and I a,b ( x ) represents the incomplete beta function.

Approximation for absolute deviation d i (14): the indicator function in (14) can be approximated by the product of two sigmoid functions,

where τ 1 is a parameter, trading off smoothness and quality of the approximation. The sigmoid function S τ 1 ( x ) is defined as S τ 1 ( x ) = (1 + e -τ 1 x ) -1 .

Approximation for maximum deviation: we employ a log-sum-exp function [2] to derive the differentiable approximation of ℓ M as

where τ 2 is a parameter, serving the same purpose as τ 1 .

Here, we demonstrate calculating the derivative of the smooth approximation (17) with respect to each component of the generalized Local score, expanding it as follows:

where

with µ i = µ ( X i ) , σ i = σ ( X i ) , E i = | Y i -µ i | /σ i . As a result, for each feature X i within D , we can evaluate ∂ℓ M ( E ; D ) /∂µ i and ∂ℓ M ( E ; D ) /∂σ i via the chain rule.

5.2 Boosting score functions for conditional coverage

Since the empirical maximum deviation ℓ M (15) is non-differentiable, we opt for the differentiable approximation during the gradient boosting step (11). Nonetheless, we utilize the original ℓ M to select the number of boosting rounds as in step (12) and to evaluate the conditional coverage of the conformalized prediction interval on the test set.

5.2.1 Theoretical guarantees

The oracle score function achieving conditional coverage as defined in (13) belongs to both proposed generalized score families.

Proposition 5.1 (Asymptotic expressiveness) . Let { X i , Y i } n i =1 be i.i.d. with continuous joint probability density distribution. Under the split conformal procedure, for any target coverage rate 1 -α , as n →∞ , there exists ( µ ∗ , σ ∗ ) and ( µ ∗ 1 , µ ∗ 2 , σ ∗ ) such that the corresponding generalized Local (6) and CQR (9) score functions recover conditional coverage at rate 1 -α , as defined in (13).

It goes without saying that there is no reason to assume that the optimal µ ∗ corresponds to the conditional mean, median or any quantile of Y given X , or that the optimal σ ∗ corresponds to the standard deviation or the mean absolute deviation of Y given X , as in the original Local score (3). That said, our greedy strategy has no guarantee on global optimality and this is why the choice of the starting point-whether it is the Local or CQR score function-plays a role in the performance.

Test Data

11%

10%

10%

15%

15%

10%

18%

11%

(b) Boosted Conformal Procedure

Test Data

11%

15%

11%

17%

10%

13%

13%

10%

Local

Boosted Local

5.2.2 Empirical results on real data

We apply our boosted conformal procedure to the 11 datasets previously analyzed in [23, 18, 22]. Details on the datasets are provided in Section A.6 in the Appendix. In each dataset, we randomly hold out 20% as test data. All experiments are repeated 10 times, starting from the data splitting. We refer to Section A.7 for details on the models and hyper-parameters we employ for the training and boosting stages.

We evaluate the conditional coverage of the prediction intervals as the maximum within-group deviations across a partitioned test set (15). This partition is obtained through a contrast tree algorithm described in Section 5.1. Figure 3 illustrates the comparison between miscoverage rates of prediction intervals at each leaf of the contrast tree. These intervals are derived under the classical Local conformal procedure and our boosted conformal procedure. Notably, the conditional coverage of the boosted prediction interval more closely aligns with the target rate 1 -α .

The experiment results summarized in Table 1 indicate that applying boosting significantly enhances the performance of the baseline Local procedure. In contrast, boosting on CQR does not yield significant improvements-a sign that CQR already targets conditional coverage. (Before boosting, the prediction intervals generated by the baseline Local procedure exhibit conditional coverage deviations up to three times greater than those of the baseline CQR procedure.) It is noteworthy, however, that after boosting, the conditional coverage of the Local procedure improves to a level comparable to that of the boosted CQR procedure. While generally slightly less effective, nevertheless surpasses the performance of the boosted CQR procedure in two cases. Results on the remaining datasets are deferred to Tables A2 and A3.

6 Boosting for length

We begin by specifying the oracle prediction interval with minimum length. For a random variable Z , the High Density Region (HDR) at a specified significance level α , denoted as HDR α ( Z ) , is defined as the shortest deterministic interval that covers Z with probability at least 1 -α . The boundaries of HDR α ( Z ) , the lower limit Q l ( α ) and the upper limit Q u ( α ) , obey the condition /a80 ( Z ∈ [ Q l ( α ) , Q u ( α ) ]) ≥ 1 -α . For a pair of ( X,Y ) drawn from P , for every value of x ∈ R p , the oracle prediction interval at that point is expressed as

Before introducing the boosting strategy, we present a word of caution against optimizing exclusively for this objective. Importantly, to maintain valid marginal coverage, the shortest prediction interval is prone to overcover when the spread of Y | X (the conditional distribution of Y given X ) is small, and undercover when the spread of Y | X is large. This may be undesirable.

Similar to Proposition 5.1, we can show that the generalized score families exhibit the necessary expressiveness to contain the oracle conformity score, achieving optimal length while ensuring valid marginal coverage. The formal proof is deferred to Section A.3.

6.1 A measure for length

Consider a dataset D = { ( X i , Y i ) } n i =1 and a score function E . Denote the corresponding conformalized prediction interval by C n ( · ) , with its quality measured by the average length:

To derive a differentiable approximation of ℓ L , we approximate the empirical quantile Q 1 -α in the conformalized intervals (7) and (10) with the smooth quantile estimator Q s 1 -α constructed in (16). Here, we demonstrate calculating the derivative of the smooth approximation of ℓ L with respect to each component of the generalized Local score, expanding it as follows based on the previously outlined approximation steps:

with µ i = µ ( X i ) , σ i = σ ( X i ) , ¯ σ = n -1 ∑ n i =1 σ i . As a result, for each feature X i within D , we can evaluate ∂ℓ L ( E ; D ) /∂µ i and ∂ℓ L ( E ; D ) /∂σ i via the chain rule. For instance,

6.2 Empirical results on real data

We apply our boosted conformal procedure to the same datasets described in Section 5.2.2. Detailed information on the models and hyperparameters used during the training and boosting stages can be found in Section A.7. Partial experiment results are summarized in Table 2. Notably, the boosting performance highlighted in bold exhibits significant improvement compared to previously documented results [17, 23]. We see a pronounced enhancement with the blog dataset; before boosting, the Local prediction intervals are on average 42% longer than those generated by CQR. After boosting, these intervals outperform the boosted CQR intervals by 32% . Using CQR as the baseline also yields substantial improvements, a decrease in averaged length exceeding 10% in six out of the eleven datasets. The meps-21 dataset, in particular, shows an improvement of up to 18% relative to the baseline. Results on the remaining datasets can be found in Tables A4 and A5. Figure 4 compares the conformalized prediction intervals derived from baseline Local and CQR scores with those obtained from the boosted scores. To effectively visualize the impact of boosting, we conduct a regression tree analysis on the training set to predict the label Y , setting the maximum number of tree nodes to four. This regression tree is then applied to the test set, allowing for a detailed comparison of the prediction intervals across each of the four distinct leaves.

7 Discussion

We introduced a post-training conformity score boosting scheme aiming to optimize for conditional coverage or length of the conformalized prediction interval. An intriguing avenue for future exploration involves simultaneously optimizing both length and conditional coverage, potentially trading

1

2

3

4

1

2

3

1

2

3

4

0

0.5

1

1.5

1

2

3

4

2

4

6

1

2

3

4

0

0.05

0.1

0.15

0.2

Local Length

log(Local/Localb)

CQR Length

log(CQR/CQRb)

regression tree leaf

1

2

3

4

1

2

3

4

1

2

3

4

0

0.2

0.4

1

2

3

4

2

4

6

1

2

3

4

0

0.1

0.2

0.3

Local Length

log(Local/Localb)

CQR Length

log(CQR/CQRb)

Classical

Boosted

meps-19

blog

off these objectives by incorporating user-specified weights [29]. Additionally, we can readily adapt our procedure to meet various application-specific objectives. For instance, we can optimize for conditional coverage on predefined feature groups, a common task in enhancing fairness in distributing social resources across different demographic groups [27]. Similarly, we can modify our procedure to reduce the length of prediction intervals for predefined label groups, which can be seen as reallocating resources to decrease uncertainty for certain groups at the expense of higher uncertainty for other groups [26]. Candidate loss functions tailored to these objectives are detailed in Section A.1. Lastly, the primary emphasis of this paper centers on the design of the conformity score boosting scheme and formalizing the optimization of conditional coverage in mathematical terms, leaving room for computational optimization to enhance performance and runtime efficiency. In essence, the gradient boosting algorithm in our procedure can be replaced with any gradient-based machine learning model. Thus, another interesting future direction would be to explore whether alternative algorithms could enhance performance.

Acknowlegements

E.C. was supported by the Office of Naval Research under Grant No. N00014-24-1-2305, the National Science Foundation under Grant No. DMS2032014, and the Simons Foundation under Award 814641. R.F.B. was supported by the Office of Naval Research via grant N00014-20-1-2337, and by the National Science Foundation via grant DMS-2023109.

References

Frank E. Harrell and C. E. Davis. 'A New Distribution-Free Quantile Estimator'. In: Biometrika 69.3 (1982), pp. 635-640. ISSN: 00063444. URL: http://www.jstor.org/ stable/2335999 .

Stephen P Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

'Classification with conformal predictors'. In: Algorithmic Learning in a Random World . Boston, MA: Springer US, 2005, pp. 53-96. ISBN: 978-0-387-25061-8. DOI: 10.1007/0387-25061-1_3 . URL: https://doi.org/10.1007/0-387-25061-1_3 .

Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world . Vol. 29. Springer, 2005.

I-Cheng Yeh. Concrete Compressive Strength . UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C5PK67. 2007.

C.M. Achilles et al. Tennessee's student teacher achievement ratio (STAR) project . 2008.

Michael Redmond. Communities and Crime . UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C53W3X. 2009.

Vladimir Vovk. 'Conditional validity of inductive conformal predictors'. In: Asian conference on machine learning . PMLR. 2012, pp. 475-490.

Hadi Fanaee-T. Bike Sharing Dataset . UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C5W894. 2013.

Prashant Rana. Physicochemical Properties of Protein Tertiary Structure . UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C5QW3H. 2013.

Krisztian Buza. BlogFeedback . UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C58S3F. 2014.

Tianqi Chen and Carlos Guestrin. 'Xgboost: A scalable tree boosting system'. In: Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining . 2016, pp. 785-794.

Kamaljot Singh. Facebook Comment Volume Dataset . UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C5Q886. 2016.

Andre Esteva et al. 'Dermatologist-level classification of skin cancer with deep neural networks'. In: nature 542.7639 (2017), pp. 115-118.

Guolin Ke et al. 'Lightgbm: A highly efficient gradient boosting decision tree'. In: Advances in neural information processing systems 30 (2017).

Jing Lei et al. 'Distribution-Free Predictive Inference for Regression'. In: Journal of the American Statistical Association 113.523 (2018), pp. 1094-1111. DOI: 10.1080/01621459. 2017.1307116 . eprint: https://doi.org/10.1080/01621459.2017.1307116 . URL: https://doi.org/10.1080/01621459.2017.1307116 .

Yaniv Romano, Evan Patterson, and Emmanuel Candes. 'Conformalized quantile regression'. In: Advances in neural information processing systems 32 (2019).

Yaniv Romano, Evan Patterson, and Emmanuel J. Cand` es. 'Conformalized Quantile Regression'. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems . Red Hook, NY, USA: Curran Associates Inc., 2019.

Mathieu Blondel et al. 'Fast differentiable sorting and ranking'. In: International Conference on Machine Learning . PMLR. 2020, pp. 950-959.

Rina Foygel Barber et al. 'The limits of distribution-free conditional predictive inference'. In: Information and Inference: A Journal of the IMA 10.2 (Aug. 2020), pp. 455-482. ISSN: 2049-8772. DOI: 10.1093/imaiai/iaaa017 . eprint: https://academic.oup.com/ imaiai/article-pdf/10/2/455/38549621/iaaa017.pdf . URL: https://doi.org/ 10.1093/imaiai/iaaa017 .

Jerome H. Friedman. 'Contrast trees and distribution boosting'. In: Proceedings of the National Academy of Sciences 117.35 (2020), pp. 21175-21184. DOI: 10.1073/pnas. 1921562117 . eprint: https://www.pnas.org/doi/pdf/10.1073/pnas.1921562117 . URL: https://www.pnas.org/doi/abs/10.1073/pnas.1921562117 .

Danijel Kivaranovic, Kory D. Johnson, and Hannes Leeb. Adaptive, Distribution-Free Prediction Intervals for Deep Networks . 2020. arXiv: 1905.10634 [stat.ML] .

Matteo Sesia and Emmanuel J Cand` es. 'A comparison of some conformal quantile regression methods'. In: Stat 9.1 (2020), e261.

John Jumper et al. 'Highly accurate protein structure prediction with AlphaFold'. In: Nature 596.7873 (2021), pp. 583-589.

Kuang-Ting Ko et al. 'Structure of the malaria vaccine candidate Pfs48/45 and its recognition by transmission blocking antibodies'. In: Nature Communications 13.1 (2022), p. 5603.

David Stutz et al. 'Learning Optimal Conformal Classifiers'. In: International Conference on Learning Representations . 2022. URL: https://openreview.net/forum?id=t8O4LKFVx .

Isaac Gibbs, John J. Cherian, and Emmanuel J. Cand` es. Conformal Prediction With Conditional Guarantees . 2023. arXiv: 2305.12616 [stat.ME] .

Kexin Huang et al. Uncertainty Quantification over Graph with Conformalized Graph Neural Networks . 2023. arXiv: 2305.14535 [cs.LG] .

Lahav Dabah and Tom Tirer. On Temperature Scaling and Conformal Prediction of Deep Classifiers . 2024. arXiv: 2402.05806 [cs.LG] . URL: https://arxiv.org/abs/2402. 05806 .

Medical expenditure panel survey, panel 19 . URL: https://meps.ahrq.gov/mepsweb/ data_stats/download_data_files_detail.jsp?cboPufNumber=HC-181 .

Medical expenditure panel survey, panel 20 . URL: https://meps.ahrq.gov/mepsweb/ data_stats/download_data_files_detail.jsp?cboPufNumber=HC-181 .

Medical expenditure panel survey, panel 21 . URL: https://meps.ahrq.gov/mepsweb/ data_stats/download_data_files_detail.jsp?cboPufNumber=HC-192 .

A Appendix

A.1 Candidate loss functions for additional application-specific objectives

Conditional coverage on predefined feature groups : this task can be viewed as a specialized application within our broader strategy of boosting for conditional coverage, as detailed in Section 5. There, the primary challenge was to develop a loss function that accurately measures deviations from the target conditional coverage rate. We achieved this by using contrast trees to identify partitions in the feature space that maximize these deviations, effectively identifying subgroups in need of protection. This process is simplified when the partitions correspond to prespecified groups, allowing us to continue using the empirical maximum deviation as a candidate loss function.

Consider a dataset D = { ( X i , Y i ) } n i =1 and a score function E . Denote by C n ( · ) the conformalized prediction interval constructed from E . Let G 1 , . . . , G M be prespecified feature index groups. Within each set D i = { ( X j , Y j ) } j ∈ G i , compute the absolute deviation d i as

The overall empirical maximum deviation is then defined as

Interval length conditional on predefined label groups : for a dataset D = { ( X i , Y i ) } n i =1 and a score function E , let Y 1 , . . . , Y M be the prespecified label groups. A natural minimization objective for balancing uncertainty among these groups is defined as:

where ( w 1 , . . . , w M ) represents a set of user-specified weights.

A.2 Proof of Proposition 5.1

Our proof relies on the following lemma.

Lemma A.1 (Expressiveness) . Given any sample pair X and Y with a continuous joint probability density distribution, and a prediction interval [ c l ( · ) , c u ( · )] with marginal coverage equal to 1 -α , there exist specific function sets: ( µ ( · ) , σ ( · ) ) for the Local type, and ( µ 1 ( · ) , µ 2 ( · ) , ˜ σ ( · ) ) for the CQR type, such that asymptotically:

The conformalized prediction interval (7), derived using the generalized Local type conformity score f µ,σ , accurately recovers [ c l ( · ) , c u ( · )] .

Similarly, the conformalized prediction interval (10), based on the generalized CQR type conformity score E µ 1 ,µ 2 , ˜ σ , also recovers [ c l ( · ) , c u ( · )] .

Proof of Lemma A.1. Recall that the generalized Local score (6) characterized by ( µ, σ ) takes the form

Asymptotically, the conformalized prediction interval is given by

Here, Q 1 -α represents the population quantile. Set

By assumption, we have

With a simple change of variables, the above inequality is equivalent to

In other words, this is equivalent to

We have thus proved the result for the generalized Local type conformity score E µ,σ . In the same spirit, we can prove the result for the generalized CQR type conformity score E µ 1 ,µ 2 , ˜ σ by taking

Recall that a generalized CQR score function (9) characterized by ( µ 1 , µ 2 , σ ) is defined as:

which leads to the asymptotic conformalized prediction intervals of the form

Plugging in µ 1 , µ 2 , ˜ σ defined above, we immediately have

Proof of Proposition 5.1. It suffices to take c l ( x ) = Q α/ 2 ( Y | X = x ) , c u ( x ) = Q 1 -α/ 2 ( Y | X = x ) and apply Lemma A.1.

A.3 Boosting for length: theoretical guarantees

Similar to Proposition 5.1, we show in Proposition A.2 below that the generalized Local and CQR score families exhibit the necessary expressiveness to contain the oracle score, achieving optimal length while ensuring valid marginal coverage.

Proposition A.2 (Asymptotic expressiveness) . Under the assumptions of Proposition 5.1, for any target coverage rate 1 -α , as n →∞ , the following statements hold true:

There exists ( µ ∗ , σ ∗ ) such that the corresponding generalized Local score function (6) recovers the shortest oracle prediction interval (18).

There exists ( µ ∗ 1 , µ ∗ 2 , σ ∗ ) such that the corresponding generalized CQR score function (9) recovers the shortest oracle prediction interval (18).

Proof of Proposition A.2. It suffices to take c l ( x ) = Q l ( α ) ( Y | X = x ) , c u ( x ) = Q u ( α ) ( Y | X = x ) and apply Lemma A.1, where Q l ( α ) and Q u ( α ) are the lower and upper limits of the High Density Region defined in (18).

A.4 CQR type conformity score boosting

A generalized CQR score function (9) is uniquely defined by a triple ( µ 1 ( · ) , µ 2 ( · ) , σ ( · ) ). We will show how searching for a generalized CQR score can be reduced to searching for a Local generalized score. To begin with, we shall say that score functions are equivalent if they recover identical conformalized prediction intervals.

Definition A.3. Let { X i , Y i } n i =1 , ( X n +1 , Y n +1 ) be i.i.d. with continuous joint probability density distribution, and let [ n ] be partitioned into a training set I 1 and a calibration set I 2 . Consider two conformity score functions, E 1 and E 2 , which produce conformalized prediction intervals C 1 ( · ) and C 2 ( · ) , respectively. For any target coverage rate 1 -α , E 1 and E 2 are equivalent if C 1 ( · ) = C 2 ( · ) when marginal coverage rates /a80 ( Y n +1 ∈ C 1 ( X n +1 )) and /a80 ( Y n +1 ∈ C 2 ( X n +1 )) match.

Building on this definition, we are now equipped to establish the following equivalences:

Lemma A.4. Under the assumptions of Definition A.3, the following statements hold:

For the CQR-r score function defined in Section 2, there is an equivalent generalized Local score function characterized by a pair ( µ ( · ) , σ ( · )) , where µ = ( µ 1 + µ 2 ) / 2 , σ = ( µ 2 -µ 1 ) / 2 .

For any generalized Local score function characterized by the pair ( µ ( · ) , σ ( · )) , there is an equivalent generalized CQR score function characterized by a triple ( µ ( · ) , µ ( · ) , σ ( · )) .

The proof of the above Lemma is deferred to Section A.5. Leveraging these equivalences, we carry out the boosted conformal procedure as follows: first, we initialize a triple µ (0) 1 = ˆ q α/ 2 , µ (0) 2 = ˆ q 1 -α/ 2 , σ (0) 1 = ˆ q 1 -α/ 2 -ˆ q α/ 2 , which characterizes the CQR-r score function. Next, we find an equivalent generalized Local score function characterized by a pair ( µ (0) , σ (0) ) chosen according to Lemma A.4. After τ boosting rounds, we obtain the boosted pair ( µ ( τ ) , σ ( τ ) ) and the corresponding score function. Finally, we recover an equivalent generalized CQR score function

characterized by the triple ( µ 1 ( τ ) , µ 2 ( τ ) , σ 1 ( τ ) ) chosen according to Lemma A.4.

A.5 Proof of Lemma A.4

Recall that the generalized Local score (6) characterized by ( µ, σ ) takes the form

The conformalized prediction interval is given by

A generalized CQR score function (9) characterized by ( µ 1 , µ 2 , σ ) is defined as:

which leads to conformalized prediction intervals of the form

Plugging in the triple µ 1 ( x ) = ˆ q α/ 2 , µ 2 ( x ) = ˆ q 1 -α/ 2 , σ 1 ( x ) = ˆ q 1 -α/ 2 -ˆ q α/ 2 , which characterize the CQR-r score function, we have the conformalized prediction interval

Set

then the generalized Local conformity score E µ,σ ( x, y ) = | y -µ ( x ) | /σ ( x ) recovers conformalized prediction intervals of the form

From the monotonicity of the interval lengths with respect to the empirical quantiles, we have that the two score functions are equivalent by Definition A.3.

Let a generalized Local score function be E µ,σ ( x, y ) = | y -µ ( x ) | /σ ( x ) . Then it suffices to observe that

A.6 Additional information on real datasets

In Table A1, we provide the predicted label, dimensions, and source for each dataset. Data cleaning and preprocessing are in accordance with the methods described by Romano et al. [17].

All datasets, except for the meps and star data sets, are licensed under CC-BY 4.0. The Medical Expenditure Panel Survey (meps) data is subject to copyright and usage rules. The licensing status of the star dataset could not be determined.

A.7 Experimental Setup

In each dataset, we randomly hold out 20% as test data. The remaining data is divided into a training set and a calibration set, each taking up a proportion of γ and 1 -γ . We explore training ratios γ ranging from 10% to 90% . Results corresponding to the optimal value of the hyperparameter γ are recorded in Table 1, following the practice of Sesia er al. [23].

In the training stage, we employ the random forest regressor from Python's scikit-learn package to learn the baseline Local score function. The hyperparameters are the package defaults, except for the total number of trees, which we set to 1000, and the minimum number of samples required at a leaf node, which we set to 40, as recommended by Romano et al. [17]. For the baseline CQR score function, we adopt a black-box neural network quantile regressor with three fully connected layers and ReLU non-linearities, following the practice of Sesia et al. [23]. In the boosting stage, we set the hyper-parameters τ 1 , τ 2 in the approximated loss (17) to 50. The approximated loss is then passed to the Gradient Boosting Machine from Python's XGBoost package along with a base conformity score. We set the maximum tree depth to 1 to avoid overfitting and perform cross-validation for the number of boosting rounds, as outlined in Section 3. All other hyperparameters are set to package defaults.

All experiments were conducted on a dual-socket AMD EPYC 7502 32-Core Processor system, utilizing 8 of its 128 CPUs each time. The runtime for each dataset and random seed varies by dataset size, ranging from 10 minutes to 5 hours.

A.8 Additional results and error bars

In Tables A2 to A5, we present additional results on marginal coverage, maximum conditional coverage deviation ( ℓ M ), and average interval length ( ℓ P ) for each real dataset (including those not reported in Tables 1 and 2), both before and after boosting. Notably, in each case, boosting is applied to optimize either conditional coverage or average interval length. As a result, the nontargeted characteristic may or may not improve after boosting.

We have previously reported the evaluated losses ℓ M and ℓ L for each dataset, averaged over ten random seeds. Tables A6 and A7 below detail the distribution of these evaluations, providing the mean, 10% quantile, and 90% quantile for the test set deviations in conditional coverage ( ℓ M ) and average interval length ( ℓ L ). These statistics are derived from 110 test set evaluations across 11 datasets and 10 random training-test splits. We opt to report empirical quantiles instead of standard deviations due to the asymmetric and non-Gaussian nature of the data.

A.9 Experiments under different miscoverage rates

In Tables A8 and A9, we illustrate the performance of boosting for length with the Local conformity score as baseline with miscoverage rates set to 5% and 20% , respectively.

A.10 Training a gradient boosting algorithm with our custom loss functions

Our proposed boosted conformal procedure serves as a post-training step designed to refine the conformity score E ( · , · ; f ) obtained during model training. This procedure can leverage pre-trained models when available. In the absence of pre-trained models, we can alternatively train a gradient boosting algorithm directly using our custom loss functions. For example, in the context of the local score, we may initialize with µ (0) = 0 , σ (0) = 1 and then apply our boosted conformal procedure. As discussed in Section 7, this approach is flexible enough to replace gradient boosting with any gradient-based algorithm, such as neural networks, trained under our custom loss functions. This framework aligns with that of [26], which introduces a neural network trained with a custom loss function to minimize the average prediction set size in classification tasks.

In Figure A1, we compare the performance of the two approaches optimizing for average interval length, searching within the generalized Local score family F defined in (6). The primary distinction between the two procedures lies in the initialization: the first approach employs µ (0) = 0 , σ (0) = 1 , while the second derives µ (0) and σ (0) from a trained random forests model. We run the experiments on the meps-19 dataset and compare the performance across different splits of the training and calibration data. In this context, the percentage of training data refers to the proportion of training data within the combined training and calibration datasets. Our results indicate that cross-validation selects a greater number of boosting iterations when we directly train the gradient boosting algorithm, resulting in longer runtime. However, the average interval length and maximum conditional coverage deviation after boosting are notably smaller for the boosted conformal procedure we introduced in this paper.

10

20

30

40

50

60

70

80

90

1.7

1.8

1.9

10

20

30

40

50

60

70

80

90

0.8

0.85

0.9

0.95

1

10

20

30

40

50

60

70

80

90

0.14

0.16

0.18

10

20

30

40

50

60

70

80

90

20

40

60

Length

ℓ

ₘ

Runtime

Training under custom loss

Boosted conformal procedure (ours)

Percentage of training data

1

2

3

4

5

6

7

8

1.43

1.44

1.45

1.46

1

2

3

4

5

6

7

8

0.045

0.05

1

2

3

4

5

6

7

8

10

20

Boosted Length

Boosted Length SD

Runtime

Validation

Ratio between training and validation data

A.11 Selecting optimal boosting rounds via hold-out validation set

In our boosted conformal procedure, we use cross-validation on the training set to determine the optimal number of boosting rounds, a process that can be time-consuming. An alternative approach is to hold out a fraction of the training set for validation. While more computationally efficient, this method introduces a trade-off: a smaller validation set can lead to greater variability in prediction intervals and model performance, whereas a larger validation set may reduce the effective training set size, potentially limiting the model's performance. To explore this trade-off, we conduct experiments on the bike dataset, optimizing for prediction interval length. We compare performance across two settings: 5-fold cross-validation and a hold-out validation set, with the training-to-validation set ratio ranging from 1:1 to 8:1. For each setting, we run 100 experiments, recording the average boosted length, the standard deviation of boosted lengths, and the average runtime. The results are shown in Figure A2.

A.12 Additional figures on individual datasets

In this section, we present a series of supplementary figures. First, we showcase the improvements in conditional coverage achieved through the boosted procedure for each benchmark dataset. Figure A3 details results for datasets meps-20 and meps-21. Figure A4 details results for datasets community, bike, and concrete.

Next, we illustrate enhanced interval lengths. Figure A5 details results for datasets meps-20, meps21, and bike. Figure A6 details results for datasets facebook-1, facebook-2, and concrete. Finally, we demonstrate in Figure A7 how cross-validating the number of boosting rounds effectively prevents the gradient boosting algorithm from overfitting.

Test Data

11%

13%

10%

17%

13%

17%

19%

Test Data

11%

12%

11%

10%

13%

17%

10%

16%

Local

Boosted Local

(b) Boosted Conformal Procedure

(a) meps-20 dataset.

Test Data

11%

12%

11%

10%

13%

17%

10%

16%

Boosted Local

Test Data

11%

12%

11%

10%

13%

17%

10%

16%

Local

(b) meps-21 dataset.

(b) Boosted Conformal Procedure

Test Data

Test Data

21%

16%

21%

13%

16%

13%

12%

14%

14%

13%

12%

15%

20%

(b) Boosted Conformal Procedure

Local

Boosted Local

(a) community dataset.

Test Data

11%

18%

13%

13%

16%

29%

Test Data

20%

14%

12%

22%

20%

12%

Local

Boosted Local

(b) Boosted Conformal Procedure

(b) bike dataset.

Test Data

38%

38%

24%

Test Data

35%

39%

26%

Local

Boosted Local

(b) Boosted Conformal Procedure

(c) concrete dataset.

1

2

3

4

1

2

3

4

1

2

3

4

0

0.1

0.2

0.3

1

2

3

4

2

4

6

1

2

3

4

0

0.1

0.2

0.3

Local Power

Local Log Ratio

CQR Power

CQR Log Ratio

Classical

Boosted

regression tree leaf

(a) meps-20 dateset.

1

2

3

4

1

2

3

4

1

2

3

4

0

0.1

0.2

0.3

0.4

1

2

3

4

2

4

6

1

2

3

4

0.1

0.15

0.2

0.25

0.3

Local Power

Local Log Ratio

CQR Power

CQR Log Ratio

Classical

Boosted

regression tree leaf

(b) meps-21 dateset.

1

2

3

4

0.5

1

1.5

2

1

2

3

4

0

0.5

1

1

2

3

4

0.2

0.4

0.6

0.8

1

1.2

1

2

3

4

-0.3

-0.2

-0.1

0

0.1

Local Power

Local Log Ratio

CQR Power

CQR Log Ratio

Classical

Boosted

regression tree leaf

(c) bike dateset.

1

2

3

4

0

2

4

6

8

1

2

3

4

-0.2

0

0.2

0.4

1

2

3

4

0

2

4

6

8

1

2

3

4

-0.4

-0.2

0

0.2

0.4

Local Power

Local Log Ratio

CQR Power

CQR Log Ratio

Classical

Boosted

regression tree leaf

(a) facebook-1 dateset.

1

2

3

4

0

2

4

6

8

1

2

3

4

0

0.2

0.4

1

2

3

4

0

2

4

6

8

10

1

2

3

4

-0.4

-0.2

0

0.2

0.4

Local Power

Local Log Ratio

CQR Power

CQR Log Ratio

Classical

Boosted

regression tree leaf

(b) facebook-2 dateset.

1

2

3

4

0.4

0.6

0.8

1

1.2

1

2

3

4

0.4

0.6

0.8

1

1

2

3

4

0.2

0.3

0.4

0.5

1

2

3

4

-0.1

0

0.1

0.2

Local Power

Local Log Ratio

CQR Power

CQR Log Ratio

Classical

Boosted

regression tree leaf

(c) concrete dateset.

0

100

200

300

400

500

t

0.085

0.090

0.095

0.100

0.105

0.110

0.115

0.120

M

Average CV Loss

0

100

200

300

400

500

t

0.08

0.10

0.12

0.14

0.16

Test Loss

(a) seed 9.

0

100

200

300

400

500

t

0.130

0.135

0.140

0.145

0.150

0.155

M

Average CV Loss

0

100

200

300

400

500

t

0.08

0.09

0.10

0.11

0.12

0.13

0.14

Test Loss

(b) seed 8.

0

100

200

300

400

500

t

0.075

0.080

0.085

0.090

0.095

0.100

0.105

0.110

M

Average CV Loss

0

100

200

300

400

500

t

0.080

0.085

0.090

0.095

0.100

Test Loss

(c) seed 7.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While '[Yes] ' is generally preferable to '[No] ', it is perfectly acceptable to answer '[No] ' provided a proper justification is given (e.g., 'error bars are not reported because it would be too computationally expensive' or 'we were unable to find the license for the dataset we used'). In general, answering '[No] ' or '[NA] ' is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist' ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Assumptions necessary for establishing the theoretical guarantees are detailed in each result presented in the paper. Additional limitations are discussed in Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See Sections A.4, A.2 and A.3.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Sections 5.2.2 and A.7.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Real data sets for our empirical analyses are publicly accessible and described in Section A.6. Currently, the code is accessible upon request. Although the code has not been released yet, we have provided detailed documentation for reproducibility in Sections 5.2.2 and A.7.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section A.7.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See Section A.8.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Section A.7.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See Introduction.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: See Section A.7.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: See Sections 5.2.2, A.7, and A.6.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a gradient boosting-based approach to tailor a conformal score function to better satisfy desirable properties. A more general score function \"family\" is introduced which adheres to a specific form dependent on parameters (e.g. $\\mu, \\sigma$). Instead of directly using the parameter estimates obtained from model training of the predictor $f$, a post-hoc boosting stage is introduced in which the parameters are further refined leveraging additional data to minimize property-specific losses, such as conditional coverage deviation or conformal interval length. After boosting for $\\tau$ iterations, the refined parameter estimates $\\mu^{(\\tau)}, \\sigma^{(\\tau)}$ can be used for down-stream conformal calibration and prediction interval generation. Results are compared to default local conformal scores (Local) and CQR on a range of regression tasks, obtaining slightly more favorable empirical properties.\n\nSTRENGTHS:\n- Addressing the practicality of obtained conformal sets is a relevant problem, since results can sometimes be overly conservative to adhere to the conformal guarantees\n- The proposed method seems relatively lightweight and has the benefit of not requiring changes to the model weights of $f$, since the boosting derivatives are computed w.r.t. the inputs of the scoring function $E$, which are the predictions $\\mu(x), \\sigma(x)$\n- The improvements for the local conformal score function seem substantial at times\n\nWEAKNESSES:\nMy main concerns are in regards to evaluation and practicality. The authors propose two applications of the boosting procedure to improve conditional coverage and prediction set size. For conditional coverage, the boosting loss is introduced as a relatively complicated group deviation metric leveraging a particular \"contrast tree\" mechanism, and in practice requires further non-trivial approximations to make it differentiable for boosting. Since the results are very strongly tied to this \"contrast tree\" approach, I was struggling to see how the boosting loss and conditional coverage target (Eq 12) connect in practice and how the results (e.g. Fig. 3) are more widely applicable, since the procedure is motivated as an \"effective evaluation metric (for target conditional coverage) of independent interest\". Is this procedure amenable to any black-box predictor $f$ and can be considered a particular gradient boosting mechanism? In general, I found the obtained improvements not overly convincing. For example, in Fig. 3 if we consider the % of data for which conditional coverage ends up being violated (above the red line) we find 33% for Local and 37% for Boosted Local. Granted, the boosting seems to distribute miscoverage somewhat more evenly, but the overall violation rate remains similar. In Table 1, boosting CQR only gives marginal improvements. It would be good to compare the results to other conformal algorithms which claim to empirically (or sometimes even theoretically, under relaxed conditions or varying interpretations of conditionality) improve conditional coverage, e.g. [1,2,3,4]. This would help clarify if the approach is more widely beneficial. Similarly, all experiments are run leveraging the same random forest and quantile network regressors -- do results translate and are amenable across different models $f$ as well? Another hurdle might be the fact that a differentiable property-specific loss needs to be defined for every target property, whose amenability may also depend in some way on the underlying model's design, and is fixed. Thus, tackling a more flexible scoring function comes at the cost of introducing another (fixed) functional to optimize over, with required approximations. \n\nOn another note, I was somewhat struggling to understand how the boosting procedure connects to the overall conformal procedure. In Fig. 1 it is suggested that boosting is an independent step inbetween model training and conformalization, but in the experiment description it seems that boosting leverages existing datasets (namely training data). Should I understand that the boosting dataset $\\mathcal{D}$ is a separate set? If so, how do its sampling requirements (e.g. exchangeability) connect to calibration and test data. If not, should I understand that the boosting step actually equates to an iterative conformal calibration procedure leveraging $\\mathcal{D}_{cal}$? Since the boosting objectives (Eq. 14, 17) actually require computing conformal sets and quantiles. Thank you for clarifying.\n\nOther comments:\n- I believe the $j$ index in the 3rd final line of Algorithm 1 should be omitted, since the final boosted score functions are not taken w.r.t. a particular fold.\n- Fig. 4 is missing a legend, making it hard to interpret\n- The use of the term \"Power Loss\" seems odd, since the goal is to reduce prediction set size, and from my understanding a smaller prediction set/interval will equate higher power, since the permutation testing interpretation of the null hypothesis is on inclusion in the set.\n- Theorem 5.1 does not seem like a theorem but more like a remark to me, since it is merely suggesting that the \"generalized\" score family may contain the optimal oracle solution, but in no way relates any optimality to the practically obtained solutions via the boosting procedure.\n- It would be beneficial to use hyper-referencing for Equations and the Algorithm throughout the paper.\n\nReferences\n- [1] Romano, Yaniv, et al. \"With malice toward none: Assessing uncertainty via equalized coverage.\" Harvard Data Science Review 2.2 (2020): 4.\n- [2] Gibbs, Isaac, John J. Cherian, and Emmanuel J. Candès. \"Conformal prediction with conditional guarantees.\" arXiv preprint arXiv:2305.12616 (2023).\n- [3] Sesia, Matteo, and Yaniv Romano. \"Conformal prediction using conditional histograms.\" Advances in Neural Information Processing Systems 34 (2021): 6304-6315.\n- [4] Jung, Christopher, et al. \"Batch multivalid conformal prediction.\" arXiv preprint arXiv:2209.15145 (2022).\n\nQUESTIONS:\n- Could you please comment on some of the raised points above, such as on clarifying the data used by the boosting procedure, the applicability of the \"contrast tree\" method, the practicality of defining new optimization objectives, or the distinction to other existing methods aiming for conditional coverage.\n- The approach to obtain a differentiable loss objective from Eq. 14 seems quite involved. Could you comment on the motivation of leveraging \"contrast trees\" and the tools used to go from Eq. 14 to a differentiable loss? \n- Are there any results that can be made on how the boosting procedure affects down-stream conformal coverage guarantees, or statements of optimality or improvement with regards to the obtained \"boosted\" conformity scores over the default ($0$-th iteration) conformity scores?\n\nLIMITATIONS:\nI did not find the limitations of the method thoroughly discussed anywhere, and I believe the distinction to other related works could be improved. There are some comments made in the discussion on improvements, as well as L218-L222 on not guaranteeing an optimal solution.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a methodology for learning a conformal score function after training. Notably, the proposed method does not require model retraining, and instead learns the score function using trained model predictions using a cross-validation approach on the training data.\n\nSTRENGTHS:\n**Originality:** Although the paper is in the same vein as works such as ConfTr [1], the fact that it can work directly on top of trained predictions on the same data is a clever formulation and a nontrivial benefit.\n\n**Quality:** Overall, the paper is technically sound; I have no concerns regarding the theory. My reservations regarding the empirical results are discussed under weaknesses.\n\n**Clarity:** The paper is well-written and all necessary ideas regarding conformal prediction for understanding the results in the paper are discussed appropriately.\n\n**Significance:** The idea of learning a conformal score function after training (using the same data) but before calibration is, to my knowledge, a novel formulation and worth building on.\n\n\n[1] https://arxiv.org/abs/2110.09192\n\nWEAKNESSES:\n**Marginal empirical benefit.** My main reservation with the proposed method is the seemingly marginal benefit it provides over CQR for conditional coverage, as shown in Table 1. This is especially relevant for large scale settings where the overhead of the boosting/CV stage is going to be highly nontrivial. Additionally, conditional coverage here is evaluated using contrast trees, which are baked into the objective of the boosting approach. It would be interesting to see if the improvements hold under, for example, just group-conditional coverage.\n\n**Baseline score functions.** The baseline score functions as described in Appendix A.8 are relatively weak, untuned models. While I do not expect an expansive evaluation of models to be within the scope of this work, it would be good to know how sensitive the improvements in the paper are to the choice of baseline score model.\n\nOverall, while I have reservations regarding the empirical results, I do find the formulation of learning the conformal score in this paper useful. As a result, I lean accept.\n\nQUESTIONS:\n- The boosting stage uses the same data on which the baseline score is trained. I'm curious whether the authors also considered splitting off some data for this stage (instead of doing the CV approach)? Although this would be less sample efficient, it seems like it may have greater benefit than reusing data for more powerful baseline score models.\n\n- It seems to me that the choice of boosting here for improving the baseline score could be replaced by some alternative means of mapping trained model predictions to the necessary mean/variance parameters for the conformal prediction intervals, and I'm wondering whether the authors considered such alternatives?\n\nLIMITATIONS:\nAlthough not in a separate section, limitations are adequately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to utilize the training data and gradient boosting (of the model's predictions) to optimize loss functions related to conformal prediction interval length and conditional coverage with respect to the score function. The optimized score function is then used for \"plain\" CP usage, via calibration set.\n\nSTRENGTHS:\n- The approach is novel and significantly differs from other methods that utilize training data to improve CP performance.\n\n- The authors derive asymptotic theory for the sufficient expressiveness of the approach.\n\n- Experiments demonstrate that the approach consistently boosts the CP metric associated with the designed loss (interval length or conditional coverage).\n\nWEAKNESSES:\n- There is no discussion and investigation on the impact of optimizing the score for conditional coverage on the interval length and vise versa. In practice, both can be important for a user.\n\n- Even though there is theory of the \"asymptotic expressiveness\" of the approach, the method in practice is based on  differentiable approximations and greedy optimization that do not guarantee optimality.\n\nQUESTIONS:\n- What can you say about the computational complexity of the boosting procedure?\n\n- In practice, both conditional coverage and the interval length are likely to be important for a user.  \nTherefore, when you optimize the score for better conditional coverage report also the effect on the interval length.  \nSimilarly, when you optimize the score for better interval length report also the effect on the conditional coverage.  \n\n- Regarding the previous comment, when discussing these two properties it is worth mentioning the work:   \nDabah, L., & Tirer, T. (2024). \"On Temperature Scaling and Conformal Prediction of Deep Classifiers\". arXiv preprint arXiv:2402.05806.  \nthat shows a simple way to trade between them in CP methods for classification. \n\n- While I am aware that your approach do not require access to the model's parameters, it would be insightful to compare the approach (even at some of the settings) to the approach of [26], namely optimizing the model's parameters to minimize a loss that includes the term that you design.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper resents a novel method to enhance conformal prediction intervals using gradient boosting. The proposed method focuses on improving specific properties such as conditional coverage and interval length. The key idea is to iteratively refine a predefined conformity score function via boosting, guided by a loss function designed to measure deviations from the desired properties. Experiments demonstrate that starting from conventional conformal methods, the boosted procedure significantly reduces interval lengths and better aligns with target conditional coverage.\n\nSTRENGTHS:\n1. Post-Training Enhancement: The boosting process is applied post-training, meaning it doesn't require modifications to the original model, making it applicable to a wide range of pre-trained models.\n2. Flexibility: The approach can be tailored to optimize various properties beyond interval length and conditional coverage, such as fairness or other specific criteria.\n3. Robustness: The use of cross-validation to select the number of boosting rounds helps prevent overfitting and ensures the method's robustness across different datasets and applications.\n\nWEAKNESSES:\n1.  A comprehensive description of the experiment setting is necessary. Since the method is post-training, it is crucial to explain how the training samples used for post-training differ from those used for the base model. Additionally, it is important to address potential outcomes if a distribution shift occurs. If the data used for post-training is different from the one use for calibration and testing, can it still works well? (or if the data used for (posted-training,calibration, testing) is different from the one used for base model training, can it still achieves to improve CP?)\n2. For experiment presentation, why the tables contain only conditional coverage and size solely? I mean,  to convincingly demonstrate the method's effectiveness, it is essential to show that the method can improve interval length without significantly compromising conditional coverage, or improve conditional coverage without significantly increasing interval length.(rather than showcase them solely) \n3. The introduction of a boosting step increases the computational burden, potentially requiring significant resources and time, especially for large datasets. An analyze for time complexity is required for better illustration.\n\nQUESTIONS:\n1. Can your method work well with different error rates $\\alpha$?\n\n2. If the post-training set is small, can your method still perform effectively?\n\nLIMITATIONS:\nPlease refer to the weakness.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}