Adaptive Conformal Regression with Split-Jackknife+ Scores
Authors: Nicolas Deutschmann
Mattia Rigotti
Maria Rodriguez Martinez
Venue: Accepted by TMLR ()

Abstract: We introduce an extension of conformal predictions (CP) based on a combination of split-CP and the Jackknife+ procedure that enables tuning score functions to calibration data and designed to produce dynamically-sized prediction interval in regression settings.
We motivate this method with theoretical results on distribution-dependent conditional coverage guarantees for split-CP and Jackknife+ prediction sets which are determined by the statistical dependence between input data and prediction scores.
This dependence can be reduced by adapting the score function to the data distribution, thereby improving the conditional validity of conformal prediction sets.
As an illustration, we construct a variant of the MADSplit conformal regression procedure where conditional mean estimates are computed in-distribution and show through empirical validation that our method is more robust to overfitting effects than the original method, while being more sample-efficient than modern ECDF-based methods.

Published in Transactions on Machine Learning Research (04/2024)
Adaptive Conformal Regression with Split-Jackknife+ Scores
Nicolas Deutschmann IBM Research
Mattia Rigotti' IBM Research
mrg@taurich.ii ibm.
María Rodríguez Martínez IBM Research*
Reviewed on OpenReview: https: // optpreveeeer.ee....mmmm net/ forum? id= 1
Abstract
We introduce an extension of conformal predictions (CP) based on a combination of split- CP and the Jackknife+ procedure that enables tuning score functions to calibration data and designed to produce dynamically-sized prediction interval in regression settings. We motivate this method with theoretical results on distribution-deeendeant conditional cover- age guarantees for split-CP and Jackknife++ prediction sets which determined by the statistical dependence between input data and prediction scores. This dependence can be reduced by adapting the score function to the data distribution, thereby improving the con- ditional validity conformal prediction sets. As an illustration, we construct a variant of the MADSplit conformal regression procedure where conditional mean estimates are computed in-distribution and show through empirical validation that our method more robust to overfitting effects than the original method, while being more sample-efficient than modern ECDF- methods.
1 Introduction
Developing reliable estimates of machine learning model predictions is crucial component of making these predictions safcly usable in many application domains. If model predictions used inform decision making, understanding predictive errors can help better understand risks in ways that raw predictions cannot. A number of approaches have been proposed to estimate potential prediction errors, based on heuristics Lakshminarayanan et al.
(017); Guo et al. (2017). bayesian modeling (Gal & Ghahramani 2016 Kuleshov et al. 2018 Maddox et al.
2019 or frequentist approaches such as conformal predictions.
Conformal prediction (CP) (Vokk et al.
1999 Saunders al. 1999: Vovk et al. 2005) provide framework
to perform rigorous post-hoc uncertainty quantification on machine learning predictions. CP converts the predictions of model into prediction sets that verify guaranlees on expected coverage (the probability that the realization is contained in the predicted set) with finite calibration data and constraints distributions. These guarantee are highly important in safety-critical applications [Lckeufack et al. 2023 or domains where enacting decisions costly (Fannjiang et al.) 2022). but come al the price of less flexibility compared to alternative approaches.
The original formulation of CP was best-suited for classification problems, and indoed the carly work Lei & Wasserman (2012) identified that straight application of the formalism would predict constant-
NNw at Cradle
*Corresponding author Now at Yale School of Medicine

---

Published in Transactions on Machine Learning Research (04/2024)
size intervals prediction sets, highlighting the need for methods to make these prediction intervals (PI) dynamic. A theoretical basis for conformal regression (CR) by Lei et al. (2017) was followed by multiple new approaches for CR that maintain the original guarantees of CP while also improving local behavior (Romano
ct al. 2019
Guan 2020 2022
Han et al. 2023 Lin et al. 2021
Despite these developments proposing solutions that empirically improve the adaptivity of conformal pre- diction intervals, theoretical results have been established showing that several definitions for finite-sample conditional guarantees (Barber al. 2020a Vovk 2012 Lei & Wasserman 2012), so that general, well- distributed coverage seems impossible. Our work, however, shows that weaker notion of conditional coverage guarantee is possible, but can only be established post-hoc and depends on properties of the data and conformal score distributions. This motivates the use of adaptive conformal scores, for which we propose a general method in which we use a fixed model with a held-out calibration set la split-conformal and re-purpose the Jackknife+ (Barber et al. 2020b) procedure data. To this end, we slightly extend the original Jackknife+ formulation apply to general score functions and show that our data-dependent conditional coverage guarantee extends to Jackknife++ prediction sets. We propose practical realization of this scheme in the form of an improvement of the MADSplit method and show that this construction yiclds satisfying empirical results, avoiding issucs of the original MADSplit when models overfit and obtaining better results than methods based on conditional score ECDF in low data regimes.
adapt the score function to the calibration
1.1 Definitions
Let us establish definitions that will be reused throughout this manuscript. We consider a probability space Mxxy = (2x Fx & Fy, defined over a product of measurable spaces (0x, and (Ry, to which we refer respectively as input- and label-space.
We work in the context of split-conformal predictions, é.e., we sample a training dataset which we use to produce a predictive model m(X) 2x - Gy through a training algorithm. We then further sample i..dd ())........ also independent from the training data and m. We refer to the first N sampled points as the calibration dataset Cv = (X,, =1.. and to (XN+1. +1) as the test point. For conformal predictions, we consider score function s(X,)) xxy - IR and label 8, s(X,, We define the usual calibration intervals with global risk [0, 5* = (v (2ys((X (X, y) S Qq-a (s:)), where qı-a (s:) is the aN + score among fsi...., $
of the empirical score distribution on CN, i.e. the [(1 - (N) 1)1-th sorted
1.2
Related Work
1.2.1
Conformal predictions
Conformal predictions were introduced by Vovk et al. 1999 2005) and define procedure to make set predic- tions from single point predictors such that distribution- free, finite sample guarantees hold under condition of exchangeability between training and test data. In recent years, many additions have been developed to better adapt the hod to modern machine learning. Among these, split conformal predictions (Papadopou- los et al. 2002) roly on a single, independently-trained predictive model and a held-out calibration set Cy. This is the formalism used to define predictions sets Sa in Section 1.2
In split conformal predictions, under the condition that the calibration data CN is exchangeable with the test point XN+1, +1, a global coverage guarantec can be established independently of the data probability function π, which is valid marginally over Cn.
Theorem (Global Split- Conformlll Coverage Guarantee (Papadopoulos et al. 2002
PUNN+ S (XN+1)) 21 a.
It. goes without saying that the probability distribution not product measure, f.c. y / X.
2

---

Published in Transactions on Machine Learning Research (04/2024)
A pedagogical review of CP methods, which the authors have found useful, can be found in Angelopoulos & Bates (2021)
1.2.2 Conditional coverage in conformal predictions
A limitation of Theorem ! that it does not offer conditional guarantees on sub-regions of Ω2x, so that undesirable behavior is still possible, such as an uneven coverage. This can result in coverage that is unfairly distributed across groups, labels, generally concentrated in some data space regions. While satisfying solutions for finite nunbers of groups and labels exist (Vovk) 2012 Lu et al. 2022. there are impossibility
theorems strongly constraining more general notions of coverage (Brber et al. 2020a Vovk 2012
Lei &
Wasserman 2012 Barber et al. (2020a) introduce two such definitions:
Definition 1 (Distribution-free conditional coverage (Barber et al. 2020a A conformal prediction procedure with prediction sets S (XN+1) verifies distribution-free conditional (DFC) coverage if Vz E Qxx
P (yN++ E (XN44)(XN+L = x) 2 a,
for any data distribution x.
Definition 2 (Approximate distribution-fres conditional coverage [Barber et al. 2020a A conformal prediction procedure with prediction sets S (XN+1) verifies approrimate distribution--ree con- ditional (ADFC) coverage for any wx 2x such that PP (x: E wx) 6,
PP PUNN+I E Sa (NN+i) Ewx) 21-10 a;
for any data distribution π.
Results from Vovk 2012 Barber et al. 2020aa imply the impossibility of DFC coverage and Barber et al. (2020a) show the impossibility of ADFC coverage with finite sample guarantees.
Despite these results, there a growing body work on improving the local properties of conformal pre- dictions and especially conformal prediction intervals (PT), usually combining theoretically proven global bounds and empirically established local or conditional properties. Three major types of approaches have been developed with this goal, which we describe below.
1.2.3
Reweighted scores
The earliest efforts on adaptive CR proposes to rescale the usual nonconformity scores PPapadopoulos et al. Lei et al. 2017) with local information. The most prominent approach is the MADSPLIT approach (Lei et al. 2017), which uses (X)()),2) v) (Ø(X) (X), where a (XX) is an estimate of the conditional score mean E ( xl| Importantly comparison with our proposed method, in MADSPLIT this estimate is performed the training split, which preserves CP coverage guarantees, but yiclds poor performance when errors differ between training and test sets, which can happen due to overfitting distribution shifts.
2008
1.2.4 Model-based interval proposals
A second type of approach is based on choosing machine-learning models that themselves encode some notion of uncertainty. The canonical formulation is conformal quantile regression (Romano et al. 2019), although other approaches based predicting distribution parameters exist. While attractive in principle, limitation of these methods is that poor modeling performance immediately leads to poor PI. This makes PI for diffieult-to-predict examples the least trustworthy, despile often being the most important.
1.2.5 Reweighted score distributions
The most recent type adaptive conformal regression methods basod on computing adaptive distortions of the conforttty score empirical cumulative distribution function (ECDF), proposed originally by Guan b020 Given test point, the ECDF is modified by giving similarity- weight to each calibration point.
3

---

Published in Transactions on Machine Learning Research (04/2024)
yielding estimate of the rest-point-conditiona score ECDF. If the ECDF estimator is good, the result is perfect local coverage and adaptivity. Multiple variants have been proposed to optimize computational complexity or ECDF estimation, including LCR (Guan) 2020), LVD (Lin et al. 2021), SLCP (Han et al, 2023 Despite only verifying global coverage guarantees with finite samples, SLCP also was proved to achieve DFC coverage asymplotically. With finile samples, however, estimating the high quantiles of the conditional CDF can require signíficant statistics, as many calibration points are needed near every test point. Furthermore, as discuss in Section 5 the proposed localization-weight tuning methods often yield poor results complex, highddimensional data.
2 Local Coverage and Score-Input Independence
We begin by defining and characterixing the goals of adaptive conformal regression in terms of guarantees and tightness. This analysis is general and extends beyond the method we propose, but will also provide motivation for our approach and a framework to understand its applicability.
Informally, the objective of PT prediction achieve two desirable properties: local coverage guarantees for any input point X and interval bounds that perfectly adapt to capture the label distribution.
2.1 Local Coverage Guarantees
We introduce a further weakened notion of conditional coverage compared to ADFC and DFC coverage, abandoning distribution--iddeeenddnee
Definition 3 (Approximate n-dependent conditional coverage)
A conformal prediction procedure with prediction sets S (XN+1) verifies approsimaat r-dependent condi- tional (AmDC) coverage for given probability function there is an a' [0, 1) such that for any wxx C Ωx where P (x wx)== 2 8,
P P(N++1 E Sa (XN-1) (XN+1) E wx) 2 1-a',
The formulation of Definition is amenable to sufficient condition to ensure strong conditional coverage for conformal intervals defined as in Section 1.1
Proposition 1 (Sufficient condition for ArDC) Given data distribution n, if (X) is defined from score s(X,v) such that
X X xls((.y)
then AC is verified with a' = a and = O.
Indeed, if the score distribution input-independent, then its conditional quantiles are as well, so that estimating quantiles globally is the same as estimating them locally. This has been observed in previous work and is tightly related to the definition of the orthogonal loss in Feldman et al. (2021). which measures the correlation between the local coverage and the local interval size.
Of course obtaining an independent score function impossible in general, but it helps understanding the origin of the following guarantee, which can be used to guarantee ArC coverage for certain problems.
Proposition 2 (Bound on conditional coverage) If the mutual information between X and s(X,y), MI(X,s) finite, for any wx Fx such that IP (X wx) > 8, then
P
(1- ā)
where à a v - MI(X,))
In plain words, conformal prediction sets verifying a global coverage guarantee with a risk can verify A&DC where the guarantee - is the global coverage guarantee 1 degraded by a penalty term v1 eM

---

Published in Transactions on Machine Learning Research (04/2024)
This follows from Lemma 1 and is proven in the supp. material. The bound is vacuous for low-probability sets, an issue that we argue is inherent to this type of results (see Appendix F.1
Unlike global conformal guarantees which can be specified priori, Proposition provides a distribution- and score-functio ciinetionndependeee conditional coverage guarantee. Nevertheless, given a probability function n, a score function 8 yielding a sufficiently small penalty coefficient. v - exp exp(-MT(X, s)) can provide conditional guarantees. As we discuss in Section 4.1 this helps understand how the rescaled score used by MADSPLIT and our method achieve adaptive intervals.
While the conditional risk â is problem-dependent, Proposition 2 is still result about conformal predictions in that it characterizes their conditional properties and applies generally for finite samples and without making hypotheses about classes of distributions.
2.2 Measuring the Adaptivity of Pl
In the context of regression, all CP methods we are aware of predict intervals(i.e. error bars), creating a tight connection between coverage and the usual absolute error metric used in regression. Indeed, if model produces prediction m(X) for a sample (X, typical CR prediction interval will take the form ((X) im(X) - 1(X), m(X) + u(X)], so that the value y is covered if AE(X, y) = y m(X) S ((x( For symmetric methods such as our own and the methods from literature we compare to, we further have u(X) = ((X) so that coverage is equivalent to AE(X,)) y) I 11(X)1/2. Conditional coverage with risk a is obtained if this inequality is verified with a conditional probability of - a (given X). We can rephrase this statement in terms of the conditional absolute error quantile Q1-a (AE(X, X) as follows:
@1 Q1 (AE(X,)))x) μ(X)
(1)
Of course, the results of (Vovk 2012 Lei & Wasserman 2012) state that no CP method can achieve this goal generally and exactly, but approximating this relationship on practical application yiclds real benefits: such error bars can be trusted to cover most points even in high-error or high-noise inpul. regions while avoiding being over-conservative.
It common to evaluate adaptive CR methods based on two metrics computed on held-out test data: the coverage rate (Cov.) and the mean PI size (IS), under the understanding that, at fixed coverage, smaller intervals means tighter correlation between PI size and error rate. However, as we show in Appendix D an oracle predicting ideal intervals might in fact increase the mean interval size compared to non-adaptive approach. We therefore propose several new metrics to evaluate adaptivity of PI prediction methods which capture more fine-grained information about the alignment between interval sizes and errors. In all cases, we mcasure errors in regression with the absolute error score.
2.2.1 Score-Interval Size correlation TSI
A natural metric to measure whether absolute errors and interval sizes have a monotonous relationship (higher errors map to larger intervals) to use a rank correlation metric and we therefore define Tsi as the Kendall-r rank score between these raw measures. A downside of this metric is that it is sensitive to problem-dependent score distributions. Note that we refer to absolute errors are scores in this case for generality, this metric could be applied to other notions.
2.2.2
Quantile correlation 7SIQ
While evaluating the relationship in Eq. (1) is impractical, we can marginalize over inputs X that approxi- mately share the same interval size [[X)], which we obtain by grouping points by quantiles.
We group test samples by their predicted interval size quantiles (deciles in our experiments) and compute absolute error score averages for each quantile. The rank correlation between the quantile mid-point value and these averages, TSiQ, captures similar information as 7sI but gets closer to properly testing Eq. (
5

---

Published in Transactions on Machine Learning Research (04/2024)
2.2.3 Score- Quantila vs Interval Quantile determination
In order to properly verify the Eq. [ averaged over approsimatel--dentical interval size samples, we turn to the coefficient of determination R
For a given d-decile (d = ,,...., 10), with a left-edge value la, we ideally have a coverage, which is obtained if Ia/2 is approximately equal to the 1 a-error-score quantile in this interval-size quantile, @1 (Aa((A(X))Taa & T(X) Ta+1)- We therefore expect
9 (A-a(AE()) & 1(X) & Ia+1) = Ia 2
(2)
We introduce Rig the coefficient of determination verifying this relationship. high value of &go indicates that higher interval sizes are neither too large or too small in each quantile.
3 Approaching Conditional Coverage with Adaptive Scores
As we show in Section 2 a problem--depnndet notion of conditional coverage can be guaranteed based on the statistical dependence between input-space data and the conformal score s(.yy) y). We interpret this distribution dependence, manifest in the mutual information penalty term of Proposition 2 as sign that information about the data distribution needs to be injected into the conformal calibration procedure to obtain conditional coverage guarantees.
Learning parameters of score functions from data is not entirely new and indeed C-ECDF-based approach learn their conditional distributions while MADSplit relies on an estimator of the conditional error mean. Both approaches rely on kernels to give local weights to the data used for this estimates but neither provides a satislying mechanism tune the hyperparameters of these kernels and rely on heuristics, which we emprically show fail in certain experiments. Furthermore, the conditional mean score estimates of MADSplit are obtained from training data which can have significantly different errors from test data due to overfitting.
This motivates the need for a more systematic approach to learn the parameters of adaptive conformal score functions, which we formulate below.
3.1 Learning scores on calibration data: Split-Jackknifet Calibration
We propose new conformal calibration approach that combines elements of split-conformal and Jackknife++ conformal predictions (Barber et al. 2020b): like with split-conformal predictions, the predictive model is trained onco on a separate training set and conformal calibration is performed on a held-out calibration dataset. Unlike split-conformal predictions however, we wish to introduce information from the calibration set in the definition of the score function, crucially taking care of not breaking exchangcability.
The Jackknife+ procedure was originally introduced to allow leave-one-out approach to conformal calibra- tion, training N predictive models with single sample left out, which is then used to compute calibration score value. In our approach, the same procedure is applied to train N score functions evaluated on the output of a single pre-trained predictive model.
We consider a family of score functions y) with parameters and construct score matrix Sif = s(X,, yu: 0ty,, indexod by the union of the calibration set and the test point, í,j J+ J+ [1...., N 1]. Each trained parameter 0,5 is obtained by applying a learning algorithm
a restricted training set Tú {(X,, [kk] E J+/ (i, Of this formal construction, we can generally construet only N entries without access to the test label 4N+1 and the Jackknife+ construction relies only these accessible entries, 11(N-1)-
At test-time, we exploit the symmetry of the learned parameters 0,ÿ to define the prediction set C. Ca(XN+1)
Co (XN+1) =f = 2g dn s(XN+1, V. (( ((ND))
8((++1)
2 21)
(3)
6

---

Published in Transactions on Machine Learning Research (04/2024)
This formulation is extension of the original formulation in (Barber et al. 2020b which limited to conformal regression with an absolute error score and allows for a generalized applicability of the method. Tn this manuscript, we consider the task of adapting score function, but equation 3 can be used for the original intended goal of Jackknife+ if the learned parameters fiy are those the predictive model.
As we prove Appendix this formulation extends the global coverage guarantee of the Jackknife+ method:
Proposition
Consider the data given in Section 1.1 and the prediction intervals defined in Eq. 33 then, marginally over the calibralion sel,
P (UNNI E C. (Xx+1)) 21 -20.
Proposition
Consider the data in Section 1.1 the score matric Sij defined above and define ka [(1 a) 1)}- We define the score ratio as the ka-th among the sorted score ratios S(N+1)// ((N+1)//5(NN-) S1(N-1)- Then for any wx E Fx such that P(X E wxx) 2
PON++ E Ca((X41))X E 2 a", where a" 22a + V1 - e e-Mi(X,o)/8
Note that this result applies also to the original Jackknife++ approach where the model, rather than the score, is trained N times.
Proposition 4 provides a way estimate conditional mis-coverage risk given a Split-Jacknife+ procedure with global risk very similar to that of Proposition However, because the Split-Jacknifc+ procedure allows the optimization of the score function, this also opens the door to a principled way of optimizing for improved local guarantees, as we discuss below.
3.2 Optimizing scores with mutual-information objective
When a score function has free parameters to optimize, such as kernel parameters, there is currently no satisfying loss function guide parameter choice. The penalty term in Proposition motivated target to minimize in order to provide stricter conditional coverage guarantees. We therefore propose the following score loss:
/ provides a well-
Lycare Moture=M(((a
-MM((((X)) ((X), (X,,:)))
(4)
where is an embedding function (possibly the identity). The use of is motivated by practical considerations in modern deep learning applications where inputs can be high-dimensional, making mutual information estimation challenging. As long as the score can be re-cast as a function of <(X) minimizing Lecare improves a notion conditional coverage: Proposition would apply on subset of the embedding space e (S2x) which is also be more meaningful for deep learning applications where the data manifold represents sliver of the full space Qx-
4 Method: Split-Jackknife+ Rescaled Scores
In this section, we provide a concrete realization of the Split-Jackkmife+ adaptive score procodure for re- gression problems, the form of an improvement over the original MADSplit method. We use the same conditional-meant mean-rescaled absolute error score but propose two changes: we use our procedure perform in-distribution estimates of the conditional mean, and if kernels need to be tuned, we Lacore to choose their parameters.
such s(X,y) score function evaluated on n(X) and y, where # is neural network with hidden layer described by

---

Published in Transactions on Machine Learning Research (04/2024)
4.1 Split- + Rescaled Scores
We first formulate the method assuming that we have fixed kernel function - R Our base score function for regression is the usual absolute error score s(X, v) = |μ(X) yl. where μt is our pre-trained model independent from the calibration and test data. We define Jackknife++ rescaled scores as
s 3(X
(5)
(X,)
where (x,) is a Nadaraya-Watson Watson estimator of the conditional mean of 8 at X; based on K and using Tiy as training set:
K(X;, Xxx)
B (X,) = /EJ
with Ziz= =
Σ
K(X,,
(6)
Zis
/EJ//(s) ((i)
the actual calibration scores are sf s formulation as a ratio permits a more explicit definition of the prediciton sets as intervals around the model predictions:
As described in Section 2.1
+(++1) for = I...., N, and their
Ca(XN+1) [ + qa ((
where the calibration scores are rescaled by the estimate of the conditional score mcan around the test point XN+1- We provide an explicit description of the calibration and inference procedure in Algorithm D
From formal point of view, rescaling scores even with a perfect conditional mean predictor is not guaranteed to improve the adaptability of prediction intervals except if the conditional score distribution of s given X belongs to a scale lamily such that changes in conditional mean entirely determine changes in conditional quantiles. Nevertheless, intuitively rescaled scores should capture some the conditional dependence of the quantiles and therefore improve adaptability. We confirm this intuition empirically in REF by showing that rescaled scores yield a significant reduction in the score-input mutual information compared to the raw absolute error score, which translates to the empirical performance of the method described in Table 1
5 Experimental Evaluation
Throughout this section, we fix the kernel for our method to simple (approximate (PyNNDescent Dong et al. 2011)) K-nearest-neighhor (KNN) kernel (K,; = 1 if j is among the KNN of i, otherwise Kij 0), setting K = 10. We've found that this approach yields competitive results in many cases without any tuning for our approach. We discuss the potential gains of kernel tuning in Appendix C.2
5.1 Comparing Fairly to Previous Work
Tuning kernels is a key element of all existing post-hoc adaptive PI and should, to some extent, be considered part of the method. Nevertheless, to perform fair comparisons, we have tried to strike a balance between considering PI proposal methods as end-to-end and evaluating the CP method in isolation by making best effort extensions.
We evaluate our method against MADSPLIT (Lei et al. 2017) and LVD (Li et al. 2021 as representatives of the two classes of post-hoo CR, which both rely RBF kernels. MADSPLIT proposes setting the bandwidth parameter to the median of the distance distribution in the model-training data, but we found this to rarely be optimal and we therefore also used the same KNN kernel as for our method. LVD optimizes an anisotropic RBF K (x, y) = exp(-IA(r ?)) where A is a matrix on the same task as the predictive model, which lead to a large fraction of infinite interval 3 A large number of neighbors would be required
2LVD can predict infinite intervals by design: estimates quantiles of the conditional score distribution around test input X by reweighting the empirical calibration distribution. If the reweighted score distribution has very low entropy, the effective sample size is very reduced, so that. the estimated quantile is infinite. In high-dlmensional setting, seems often the that the effective sample size obtained when learning a predictive model (estimating the conditional mean) is too small.
8

---

Published in Transactions on Machine Learning Research (04/2024)
for KNN kernel for LVD instead rescale A XA and tune À until the fraction of infinite intervals is negligible. Note that we also attempted to evaluate SLCP (Han et al. 2023), but were unable to tune kernels to reach comparable metrics to the other methods, and we therefore left it out of our results.
5.2 Miee-Dimnsiooal Regression
Let us start by applying our methods to simple regression problems that will help illustrate how our approach operates. We define the following random variables
y fo Jo + x sin ( + 05) - e and v(X)= = eo + Isin (k, + de),, with ~ U(O,1), e~ ~~ v()).
While serving illustrative purpose, this relationship is designed to display heteroscedatic noise whose X x-dependency is decoupled from the conditional expectation of y given X.
We use 1000 calibration and 10000 test points to evaluate our method on a random-forest regressor trained on independent data and indeed find intervals that achieve the target global coverage and dynamic interval sizes, as show in
Predictions
0.2
0.0
-0.2
Cov: 0.96
IS: 0.14
0.0
0.2
0.4
0.6
0.8
1.0
Figure 1: Evaluation of our approach on a 1D regression problem. The dashed black line is the expected value of y given x and the dashed blue line shows an interval of two standard deviations around that expected value. Orange points correspond to model predictions and the gray band display prediction intervals.
This simple setting already allows to compare performance with other adaptive PI methods. We display the salibration-set-sizee dependence of PI metrics in Fig. 2
These results highlight a more general trend: our method more robust than LVD in the low data regime but tends to be similar with enough data. MADSplit close to our method when noise is high, but if overfitting becomese visible, our method avoids the bias felt by MADSplit.
5.3 Results on complex data
We consider four datasets in addition to our 1D regression problem. all cases, we fix the training data and train an adapted regression model by optimizing the mean squared error. Details about data processing, embeddings used for localisation and predictive models can be found Appendix B Prediction intervals are predicted with low and high amouts of calibration data over 10 repetitions where calibration and test data are resampled from the held-out data. The tasks are:
Two TDC molecule property datasets Huang et al. 2021): drug solubility and clearance prediction from respectively 4.2k and 1.1k SMILES sequences" A
"string-based based description of molecular structures
Weininger
1988

---

Published in Transactions on Machine Learning Research (04/2024)
1.0
0.8
0.6
R
0.4
Method
LVD tuned)
0.2
Ours (KNN)
MADSplit (KNN)
0.0
250
500
750
1000
1250
1500 1750 2000
Calibration Set Size
Figure 2: Calibration-set-sizee dependence of PI metrics evaluated on an independent test set.
The AlphaSeq Antibody binding dataset (Engelhart al., 2022): binding score prediction for a SARS-CoV-2 target peptide from 71k amino-acid sequences.
Regression of the numeric label of MNIST images with test-time augmentations (e.g. predicting the number "9.0" when presented with a picture of the digit '9'). This task was selected due to its irregular error distribution, which is expected to challenge rescaling approach by breaking even approximate relationships between conditional means and quantiles, see Appendix C.4
Regression-MNIST (LeCun & Cortes, 2005):
We detail our results in Table ! summary, we confirm that our method outperforms LVD when data is scarce, while usually comparable with MADSPLIT with our performance improving faster with growing sample size. The results on sequence data with higher statistics confirm that our method is competitive with both methods. Interestingly, while AlphaSeq has the most data, our method dominates others. This is due to combination of increased overfitting and scattered data distribution discussed in Appendix C.3 Finally, the MNIST task is peculiar: all methods have comparable performance mcasured by correlation measures but only LVD has both tight and adaptive interval sizes, as measured by R2 We attribute this to the irregular error distributions due to possible number confusions, which leads to highly variable conditional mean/quan quantile ratios, as shown in Appendix C.4
5.3.1 Conditional Coverage Bounds
We complement the results of Table D with estimates of the data-dependent coverage bounds from Propo- sition for raw absolute error scores and MADSplit, and Proposition 4 for our method. As we discuss in Section 3.2 computing mutual information in high-dimensional spaces is challenging and therefore evaluate the mutual information between scores and mapped inputs in the latent spaces of our models, so that the bounds are valid in this space. We roly on an approximation by using a PCA to further reduce dimensionality, exploiting found that a very large fraction of the variance can be expressed with only 4 components, which greatly simplifies MI estimation. The estimation is performed with MINE (Belghazi al., 2018) as non-parametric method like the KSG estimator (Kraskov et al. 2004) yielded poor results above As we show in Table 2 the empirical conditional bounds are rather weak and most. likely not tight as the ordering between MADSplit and our method is reversed compared to other performance metrics in Table [
tendency of regression models to organize latent representation linearly: we
two dimensions.
6 Conclusions
Our empirical results show that our new approach provides satisfying solution the weaknesses of existing post-hoc adaptive conformal regression methods. On the hand, use of the Jackknife-t procedure to
10

---

Published in Transactions on Machine Learning Research (04/2024)
Dataset
Neal
Method
Cov. 0.)).
IS
Rsq1 1
TSIQ
7SI
Ours
0.96(1)
0.18(1)
0.87(3)
0.91
0.35(1)
500
MADSPLIT
0.952(9)
0.18(1)
0.80(4)
0.96(3)
0.344
LVD
0.982(3)
0.21(1)
0.61(9)
0.86(4)
0.33
ID
Ours
0.952
0.171(5)
0.90(2)
0.96(4)
0.36(1)
2000
MADSPLIT
(.952(9)
0.18(1)
0.80(4)
0.96(3)
0.344
LVD
0.968(3)
0.174(3)
0.92(2)
0.96(3)
0.361
Ours
(195(1)
5.3(5)
0.4(1)
0.6(1)
0.09(1)
400
MADSPLIT
0.95(1)
5.0(3)
0.53(5)
0.71
0.182(3)
TDC
LVD
0.958(7)
5.2(3)
0.3(5)
0.7(1)
0.07(5)
Sol.
Ours
0.96(1)
5.2(4)
0.5(1)
0.7(1)
0.11(1)
800
MADSPLIT
0.951(6)
5.1(2)
0.58(7)
0.71(5)
0.179(6)
LVD
0.960(9)
5.4(2)
0.4(3)
0.77(7)
0.07(1)
Ours
0.95(2)
1.9(3)-10=
0.2(3)
0.6(1)
0.37(3)
60
MADSPLIT
0.95(2)
1.9(5)-10*
0.2(3)
0.60(9)
0.39(1)
TDC
LVD
0.97(2)
2.2(2)-10²
<<
-0.3(6)
0.1(2)
Clear.
Ours
0.96(1)
1.65(9)- 102
0.3(2)
0.5(1)
0.39(7)
240
MADSPLIT
0.96(1)
1.64(8).100 1002
0.41(5)
0.6(1)
0.41(7)
LVD
0.97(1)
2.02(5)-10*
< 0
0.1(2)
0.1(1)
Ours
0.952(6)
4.5(1)
0.24(8)
0.5(1)
0.05(1)
2000
MADSPLIT
0.952(5)
4.6(1)
0.1(2)
0.2(2)
0.01(2)
aSeq
LVD)
0.994(1)
7.6(5)
<0 0
0.1(1)
-0.004(6)
CoVID
Ours
0.953(4)
4.54(6)
0.28(6)
0.6(1)
0.052(8)
10000
MADSPLIT
0.952(2)
4.59(6)
0.0(1)
0.2(2)
0.01(1)
LVD
0.973(4)
5.49(4)
-0.26(4)
0.5(1)
0.052(8)
Ours 0.952(6)
0.62(3)
<< 0
0.90(5)
0.20(1)
2000
MADSPLIT
0.951(7)
0.63(5)
< 0.83(8)
0.17(1)
LVD
0.988(4)
4.1(5)
-0.19(2)
0.86(5)
0.21(1)
MNIST
Ours
0.950(3)
0.601(8)
< 0
0.92(5)
0.23(1)
5000
MADSPLIT
0.947(4)
0.62(1)
< 0
0.85(7)
0.193(6)
LVD
0.984(2)
4.4(2)
-0.17(3)
0.92(3)
0. (.2
Table 1: Numbers in parentheses are the uncertainty on the last significant digit evaluated over 10 random
samples of test and calibration data. We report Riqr < 0 if it is significantly lower than -2. Metrics where lower (higher) values indicated with + (?) symbol and bold entries are significantly outperforming non-bold entries while non-significantly dominating each other.
tune and evaluate the conditional mean of the non-conformity score solves the issue of MADSPLIT when evaluated on models with significant score distribution shifts between model-training and CP calibration data. On the other hand, while our method only guarantees local coverage asymptotically under rather stringent hypothesos, our empirical results show clear value compared to the diminished statistical power compared to ECDF-reweighhing methods, especially in the lov-calibration-data regime.
More generally, the theoretical results and data-dependent score function tuning proposed this work provide avenues for new conformal prediction methods based on adapting score functions to data distribution properties and we hope that more refined approaches can be proposed predict highly adaptive uncertainty sets.
11

---

Published in Transactions on Machine Learning Research (04/2024)
Dataset
Method
MT
Penalty
Bound
AE
0.3
0.5
ID
Ours
0.06
0.3
0.1
MADSplit
0.03
0.2
0.4
AE
0.07
0.3
0.06
TDC (Sol.)
Ours
0.02
0.1
0.5
MADSplit
0.04
0.2
0.3
AE
0.01
0.12
0.6
aSeq
Ours
3-10 3
0.05
0.8
MADSplit
8-10 3
0.09
0.65
Table 2: Conditional coverage guarantees for benchmark tasks with our method, raw absolute error scores and MADSplit. The penalty is the coefficient of the penalty in Proposition and Proposition and the bound is given for 8 = 0.3.
From a thcorctical point of view, we also hope to see further improvements: while Proposition 2 points score-input dependence as an optimization target for conditional coverage, which we deem an interesting insight, we do not expect. it to be optimally tight. This is not only because mutual information replaces the unwieldy total variation term of Proposition 6 for practical reasons of computability, but also because the mcasure-theoreti formulation is probably more general than needed for practical purposes, and genuine notion of local coverage might exploit. regularity properties of conformal regression scores. We therefore look forward to future work exploring how score and data distributions interplay with coverage locally, rather than conditionally.
References
Anastasios N Angelopoulos and Stephen Bates. gentle introduction conformal prediction and distribution-fre uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021.
Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep Learning Through the Lens of Ex- ample Difficulty. Advances Neural Information Processing Systems, volume 34, pp. 10876 10889. Curran Associates, Inc., 2021. URL Abstract. html
Rina Foygel Barber, Emmanuel J. Candès, Aaditya Ramdas, and Ryan Tibshirani. The limits of distribution-free conditional predictive inference, April 2020a. URL http: htst::://axxiv.orr//b////00 www.:or/gg//......./////// 04684
Rina Foygel Barber, Emmanuel J. Candes, Aaditya Ramdas, and Ryan Tibshirani. Predictive inference with the jackknife++, May 2020b. URL
02928
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual Information Neural Estimation. In Proceedings of the 35th International Conference on Machine Learning, pp. 531-540 PMLR, July 2018. URL https://proceedding: mlr
preess/v8b/bballgazz11.....e html
Nicholas Carlini, Ülfar Frlingsson, and Nicolas Papernot. Distribution Density, Tails, and Outliers in Machine Learning: Metrics and Applications, October 2019. URL http:///arxxv.orr/aa//11000 13427
Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. ChemBERTa: Pretraining for Molccular Property Prodiction, October 2020. URL TRRL:att.///aaxvvoog//////////
Wei Dong, Charikar Moses, and Kai Li. Efficient k-nearest neighbor graph construction for generic simi- larity measures. In Proceedings of the 20th International Conference on World Wide Web, pp. 577-586, Hyderabad India, March 2011. ACM. ISBN 978- 1-4503-06322-4 URLpttts:: ://ttps://dl..... acm. org/áoi/10..1455 wtt.aaa...oo//////..11111.
1963405. 1963487
12

---

Published in Transactions on Machine Learning Research (04/2024)
Emily Engelhart, Ryan Emerson, Lestic Shing, Chelsea Lennartz, Daniel Guion, Mary Kelley, Charles Lin, Randolph Lopez, David Younger, and Matthew F. Walsh. A dataset comprised of binding interactions for 104,972 antibodies against a SARS-CoV-2 peptide. Scientific Data, 911):653, October 2022. ISSN
2052-4463. URL httpp://wwit http///wwwttu......///// cww httpt//ww...oo/.e.eceeee//////a0
Clara Famnjiang, Stephen Bates, Anastasios N Angelopoulos, Jemnifer Listgarten, and Michael Jordan. Conformal prediction under feedback covariate shift for biomolocular design. Proceedings of the National Academy of Sciences, (2)2449912 2022.
Shai Feldman, Stephen Bates, and Yaniv Romano. Improving Conditional Coverage via Orthogonal Quantile Regression, October 2021. URL
Vitaly Feldman. Does Learning Require Memorization? A Short Tale about a Long Tail, Jamuary 2021.
URL http://arxiv..o/
http:://axxi..rrg/ab///1000 arg/abs/1906 190bb00771000
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1050-1059, New York, New York, USA, 20-22 Jun 2016. PMLR. URL htttps:/proceeddns.. mlr.
preas/v48// 1a..8//gall.hhh htmml
Leying Guan. Conformal prediction with localization, July 2020. URL
08558.
Leying Guan. Localized Conformal Prediction: A Generalized Inference Framework for Conformal Predic- tion, February 2022. URL http://arxiv httt:://axiv.oor/abbsst. orw/obr///////eeeeeoo aa////bs///10/ 08460
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1321-1330. PMLR, 06-11 Aug 2017. URL hetLL::tts//pppp httpps://proceednggs.oto http://nnn.......eesssseeee www..preessvvvv wat/ss//vv///// v07.guuo/a.htm
Xing Han, Ziyang Tang, Joydeep Ghosh, and Qiang Liu. Split Localized Conformal Prediction, February 2023. URL httt:://arxiv.orrgabb////
Anne Hersey. ChEMBL Deposited Data Set AZ dataset. Technical report, EMBL-EBI, February 2015.
URL https://wwr ebi ac htt.://///.eeee/////////////////
Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development. Proceedings of the NeurIPS Track on Datasets and Benchmarks, 1, December 2021. /ttpl
www.gss.errppp.mp ctt cc/paper_pppp files/ fftees/ppper////////
.html.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical Review E, (9(6):066138, June 2004. URL https: httpiiikkapp..rggg.. http://oi/11..10//hhhheeeo http:////hyyeeee......6..... 69.0661303
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Leurning, volume 80 of Proceedings of Machine Learning Research, pp. 2796 2804. PMLR, 10 15 Jul 2018. URL
http:///cocceeeng..... mlr. wtt.::sww.....p.../e/em1m Trress/v80/kule http///0///llleeolaa.... httml
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncer- lainty estimation using deep ensembles. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fer- gus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems, vol- ume 30. Curran Associates, Inc., 2017. URL http::///prreeddngg..errrrp....o 2077/5hee66724477722888811166862
H. G. Landau. On dominance relations and the structure of animal socicties: III The condition for a score structure. The bulletin of mathematical biophysics, 15(2):143-148, June 1953. ISSN 1522-9602. URL http:///bbi..rgg/..00//////////
13

---

Published in Transactions on Machine Learning Research (04/2024)
Yann LeCun and Corinna Cortes. The mnist database of handwritten digits. undefined, 2005. URL https:
Jing Lei and Larry Wasserman. Distribution Free Prediction Bands, March 2012. URL http://arxiv.org
abs/ 0tte/1203.5444 5422
Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. Distribution-Free Predictive Inference For Regression, March 2017. URL sttp://arxiv.orr//aaat hates//114..000 16177
Jordan Lekeufack, Anastasios A Angelopoulos, Andrea Bajosy, Michael I Jordan, and Jitendra Malik. Conformal decision theory: Safe autonomous decisions from imperfect predictions. arXiv preprinl arXiv:2310005921 2023.
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Locally Valid and Discriminative Prediction Intervals for Deep Learning Models. In Advances in Neural Information Processing Systems, October 2021. URL
https://openreviert wew.t/orrrmddddxdXX-e.
Charles Lu, Andreanne Lemay, Ken Chang, Katharina Hoebel, and Jayashree Kalpathy-Cramer, Fair Confor- mal Predictors for Applications Medical Imaging, May 2022. URI.
(wrrg//bs//1000 04392
Wesley J Maddox, Pavel Ixmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple bascline for bayesian uncertainty in deep learning. Advances in neural information processing systems, 32, 2019.
Tobias H. Olsen, lain Н. Moal, and Charlotte M. Deane. AbLang: An antibody language model for com- pleting antibody sequences. Preprint, Bioinformatics, January 2022. URL http://biorriv.orr/looooo doi/10. 1101/2022.222. 01 47700.470061
Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. Inductive Confidence Ma- chines for Regression. In Tapio Elomaa, Heikki Mannila, and Hannu Toivonen (eds.), Machine Learning: ECML 2002, Lecture Notes in Computer Science, pp. 345 356, Berlin, Heidelberg, 2002. Springer. ISBN 978-3-540-36755-0
Harris Papadopoulos, Alex Gammerman, and Volodya Vovk. Normalized nonconformity measures for re- gression Conformal Prediction. In Proceedings of the 26th IASTED International Conference on Artificial Intelligence and Applications, AIA '08, pp. 64-69, USA, February 2008. ACTA Press. ISBN 978-0-88986- 710-9.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 2830, 2011.
PyNNDescent. PyNNDescent for fast Approximate Nearest Neighbors pynndescent 0.5.0 documentation, 2020. URL https::/pynndeescent readthedocs. 1o/en//atest/
Yaniv Romano, Evan Patterson, and Emmanuel J. Candès, Conformalizod Quantile Regression, May 2019.
URL http://arxive http:///arxii..oorra///////// 03222
C. Saunders, A. Gammerman, and V. Vovk. Transduction with Confidence and Credibility. In Sizteenth International Joind Conference on Artificial Intelligence (IJCAT '99) (01/01/99), pp. 722-72222277 1999. URL
https://eprints..oo www.enttsssooo..oommmooo ac wat.kk//22880 25k9//1
Murat Cihan Sorkun, Abhishek Khetan, and Süleyman Er. AqSoIDB, curated reference set of aqueous solubility and 2D descriptors for diverse set of compounds. Scientific Data, 6(1):143, August 2019. ISSN 2052-4463. URL
nature. wttp://ww...///rrr////////////a.
Vladimir Vork. Conditional validity of inductive conformal predictors, September 2012. URL http://
arxiv.or arxiv.org/abae
14

---

Published in Transactions on Machine Learning Research (04/2024)
Vladimir Vork, A. Gammerman, and Glenn Shafer. Algorithmic Learning in Random World. Springer, New York, 2005. ISBN 978-0-387-00152-4 978-0-385-25061-8.
Volodya Vovk, Alexander Gammerman, and Craig Saunders. Machine-Learning Applications of Algorithmic Randomness. In Proceedings of the Sirteenth International Conference on Machine Learning, ICML '99, pp. 444-453, San Francisco, CA, USA, June 1999. Morgan Kaufmann Publishers Inc. ISBN 612-8.
David Weininger. SMILES, chemical language and information system. Introduction methodology and encoding rules. Journal of Chemical Information and Compuler Sciences, 28(1):31-33 366 February 1988. URL http://///oo.oo//.11111////10100
15

---

Published in Transactions on Machine Learning Research (04/2024)
A
Algorithms
In this section, we describe the calibration and PI prediction procodures for two versions of our method in the form of pseudocode. The simplest is Algorithm T where the kernel is assumed to be fixed. Indeed we've found in practice that using a KNN kernel with = 10 is an efficient and performant approach. We however also define Algorithm where we tune kernels on calibration data, with an objective motivated by the bound of Proposition For sigh-dimensional data, such when using embeddings to compute kernel similarities, we low-dimensional PCA and compute the of the marginal mutual information between the score and principal component. If PCA is not used, this sum multi-dimennsioall mutual information, but we' we've ve also found that using 2-4 principal components captures an overwhelming majority of the data variance for our experiments when using latent space embeddings, and yields good empirical improvement when used for kernel tuning.
upper bound the
Algorithm Jackknife+ rescaled score conformal regression without kernel tuning Require:
1: Exchangeable data {(X,, vi) E fix E [-T...., N 1]}- learning algorithm A mapping sample of (X pairs model m(X).
A kernel K f2x f2x - R
A risk threshold E [0, 1].
6: procedure PREDICTOR TRAINING
m7 ((Xx,))))EE--T...
end procedure
All free indices below (i. k) spon [1...., NJ.
10: procedure CALIBRATION
11:
SET Kij = K(X;, Xj)
12: SET Pij = Kis/ Enzi Kik
13: SET 8; 11/t m(xi)l
14: SET Sf 8i/ Exes Pik
15:
end procedure
16: procedure INTERVAL PREDICTION
17:
SET Q (N + 1)1
18:
SET PM(N+1) = /((N+1)/ ERAiNt4 Kik
19:
SET 6 ((41) Em PK(N-1)SK
SORT S = (1(4 "..................... ((N+)NN*]
20:
21:
SET AyN+ S[Q]
22:
end procedure
Ensure: Test prediction interval (m(X) Ayv+1, + AyN+1).
16

---

Published in Transactions on Machine Learning Research (04/2024)
Algorithm Jackknife-+ with Kernel Tuning
1: procedure CALIBRATION WITH KERNEL TUNING(MPCA, Asample. Ascan, Pexpand 2 1)
2: SAMPLE Reample pairs of non-identical training inputs (Xa: X%)
3: SET dinin dmax to the extrema of ||X, X,Il
4: SET An as Tsean values evenly space in logarithmic scale |dmin/ Bexpands dmax Bexpand] DEFINE RBF kernels Km with length scales mm-
for each An do
for [1, [ do .N
8:
SET Im = An
9:
SET Kijm (m(Xi,
10:
SET Pijmm = K Kijmm/ Σ Ekyimm Kikmm
11:
SET 8 = 11/s ml (X,))
12:
SET 8pm 81/ Expiim Pik8k
13:
end for
14:
SET mimn MI (1 {xi lemm
15:
end for
16:
SET n (m) = argmin minn
17:
SET 4m (m)
18:
SET s t/
19:
end procedure
20: procedure INTERVAL PREDICTION
21:
SET Q = a) + 1)7.
22:
SET PI(N+1) = KNN+I)a/ ExSIN Kang
23:
SET ((44)) E PR(N-1)Sk
SORT $ = (3(++D1 ....., ((N+)NN*)
24:
25:
SET -
26: end procedure
Ensure: Test prediction interval [m(X) AyN+1, + AyN+1).
17

---

Published in Transactions on Machine Learning Research (04/2024)
B Dataset and Model Details
B.1 One-Dimensional Regression
We formulate toy example of regression with label noise in one dimension as follows:
X~ 1), y= fo X2 sin (kg x + (5)) +
(8)
(~)))0,(X)), v(X) = + Isin (kg X + Φ.)l
where
fo = 1001, ky = 10, 6, = 1/2, 6g = =0-2
(9)
ke 2, d. 3x 10- == 10-1
For each repetition of the experiment, sample iiid pairs of (X, and split them into 1000 training points, 10000 test points and a variable number of calibration points. A random-forest regression model is trained, using the default parameters of SCIKIT-LEARY v1. 2.2 (Pedregosa et al. 2011] for our basc model, or disabling booistralpping to produce an overfitting model. For all methods, kernel scores are evaluated on the raw input values X
B.2 Chemical Property Regression on TDC Datasets
We use two datasets from the Therapeutics Data Commons repository (Huang et al. 2021), ) each which is treated independently. Both datascts contain samples consisting of pairs of SMILES (Weininger 1988) descriptions of chemical structure as texl sequences, and a larget value for the properly of interest. The datasets are already divided into training, validation and test samples, however, we merge the validation and test data and ro-divide it randomly into calibration and test to allow for enough statistics. For each task, we fine tune a CHEMBERTA Chithrananda et al. (2020) language model pretrained SMILES language modelling, provided by HUGGINGFACE.
The first task is to predict drug solubility on data originally produced by Sorkun et al. (019 and contains 9982 samples, split into 6988 training points and 2994 test points.
The second task consists of estimating the drug microsome clearance (drug elimination rate by the liver) on 1102 samples from Hersey 2015), which are split into 772 training points and 330 test points.
Our fine-tuned CHEMBERTA models are defined HUgcINGFACE ArtehhdllTossqgeereeellesssaaaat with
weights and sequence data is processed with the adapted AutoTokenizer, with maximum sequence lengths set to the maximum sequence length in the training data. The training is performed with the mean-squared-error loss, using a batch size of 64 and a learning rate of 4.0 x 10-5 over 100 epochs.
Embeddings for kernel scores are computed using the output of the classifier .eessfieeedddee layer of the model, which is the first linear layer of the classification head of the fine-tuned model.
B.3 AlphaSeq Antibody Affinity Regression
We use the data from the AlphaSeq survey of 104,972 measurements of the binding affinity of antibody proteins to SARS-CoV-2 target peptide, paired with amino-acid sequence information for the antibodies. We pre-process the label data by dropping missing measurements and measurements on reference epilopes (keeping only those labelled as MIT_ Terget) resulting in 69,297 valid sequence-affinit pairs. The data is The talibraainn
divided into a training/ soiniig//valddtioon /wwp/vaaaaaaoo///a/aaaa wataattioonntttttaaltale -tsetetttttattattt set is randomly subsampled into a test dataset of size 7314 and a calibration dataset of variable size.
split of sizes
5Thhee models can also do regression, despite the name
18

---

Published in Transactions on Machine Learning Research (04/2024)
We the amino-acid sequences for both the heavy and the light chains of the antibody as inputs to pre- trained, frozen-weight. ABLANG Olsen et al. (2022) amino-acid language models adapted to each chain type. The embeddings thus produced are concatenated and mapped to numerical predictions with two-hidden- layer neural network with layer widths (128, 32) and ReLU activations. The final singlc-output layer does not have an activation function. This model is trained with the Adam optimizer using a learning rate 1.0 10-3, batch size of 128 and regularized with early stopping, monitoring the validation mean-squared error.
Embeddings for kernel scores are the concatenated outputs of the ABLANG models.
B.4 MNIST Regression
We use the classic MNIST dataset (LeCun & Corles 2005) repurposed as a regression task where each digit is labelled by its floating point value. We use data augmentation both training time and test time to increase the potential confusion between similar digits, which leads to non-trivial error structure we discuss in Appendix C.4.
We use the following randomized distortions, applied sequentially
Gaussian blurring with kernel size 3 x 3, applied with probability 30%.
Perspective transformation with scale 0.4, applied with probability 30% (torchvision MandomPorsppetivve)
Gaussian noise each pixel with mean and standard deviation 1/(6 + v) where U(0,5) 5).
The specific choice of transformation quite arbitrary, but is meant to ensure qualitatively that numbers are nearly always recongnizable human observers, while making significant distortions common. Trans- formations resampled every time image is used.
We train convolutional neural network (CNN) on this regression task by optimizing the mean squared error loss with ADAM optimizer with learning rate 5 10-4, The CNN has the archilecture described in Fig.
Lin.
Lin.
Conv.
Conv.
Conv.
Conv.
Conv. 64
Max Pool
Conv. 128
Max
Max
128
256
32
32
64
64
Pool
Pool
4
J
3x3
3x3
3x3
3x3
3x3
3x3
3x3
3x3
3x3
256
1
Figure 3: CNN used to perform the MNIST regression task. Each hidden layer is follwed by a batch normalisation layer and a rectified linear unit activation function.
Embeddings for kernel scores are the 256-dimensional outputs of the hidden linear layer.
B.5 Hardware
All our experiments are run a Linux HPC cluster. For model training and inference, we used 32-core, 64GB RAM environments with single NVidia A100 GPU. Conformal predictions used the same CPU and RAM but GPU. Experiments in Python 3.9 with pytorch v2.0.0+tul17.
19

---

Published in Transactions on Machine Learning Research (04/2024)
0.3
0.2
MI(s,X)
0.1
RBF
KNN
0.0
10-4
10-3
10-2
10-1
100
RBF Kernel Bandwidth
Figure 4: Dependence of the mutual information between s+ (X, v) and X as function of the kernel length scale, reported as the average over each Jackknife+ split.
C Additional Experimental Results
C.1 Original Kernel Tuning Strategies
We present in Table 3 an extended version of Table including both our re-tuned kernels for the basclines, as described in Section and the original luning techniques proposed by each method. We report uncertainties as the percentile error, i.e. the 67th percentile of absolute deviation from the mean, evaluated with bootstrapping. This is equivalent reporting the standard deviation if the metrics are normally distributed, which is motivvy hypothesis given our limited statistics. As describe in the body of the manuscript, the original tuning strategy proposed for LVD leads to many infinite intervals, sometimes a large majority. Given that it does reach reasonable performance in the 1D example, and that the worse case is ALPHASEQ, which uses very high-dimensional embeddings, we conjecture that this reflects the unsuitability of this tuning procedure in high dimensions.
In Table 4 we also present the same results for a sot of datasets from the UCI repository that were cxamined in the paper (Lin al. 2021
C.2 Jackknife+ Kernel Tuning
For for the sake of limiting computational costs and brevity, we demonstrated the performance of our method with a fixed 10-NN kernel. In this section, we compare this approach with a RBF approach, where the length scale is tuned using Algorithun
2
limiting ourselves to the one-dimensional example.
As we show Table 5 a tuned RBF kernel yields comparable results the KNN approach the low statistics regime, but achieves better results when more data is available. As we use Kozachenko-Lconenle based mutual information estimator, the improved performance in the higher statistics regime is likely due its the asymptotically vanishing bias.
This potential for squeezing extra performance is confirmed by investigating the tuning procedure. In Fig. we show that the is finite range where MT(s+, X) is minimized, and that this minimizer does improve over the same measure evaluated on the 10-NN kernel.
C.3 AlphaSeq Prediction Details
As we discuss in Section 5.3 the performance of the baselines rather poor on the AlphaSeq dataset, which might be especially surprising for LVD due to the large calibration sets. The difficulties of MADSPLIT
20

---

Published in Transactions on Machine Learning Research (04/2024)
Dataset
Neal
Method
Cov. (2 0.5))
IS
Rior
TSIQ
7S1
Ours
0.96(1)
0.18(1)
0.87(3)
0.91
0.35(1)
MADSPLIT (KNN)
0.952(9)
0.18(1)
0.80(4)
0.96(3)
0.344(9)
500
MADSPLIT (Median)
0.95(1)
0.19(1)
<
0.1(1)
0.04(1)
LVD (Tunedd
0.982(3)
0.21(1)
0.61(9)
0.86(4)
0.33(1)
LVD (Basc)
0.985(2)
0.22(1)
0.5(1)
0.79(9)
0.31
1D
Ours
0.952(
0.171(5)
0.90(2)
0.96(4)
0.36(1)
MA MADSPLIT
0.951
0.184(2)
0.81
0.93(6)
0.340(7)
2000
MADSPLIT (Median)
0.95(1)
0.19(1)
<0
0.1(1)
0.04(1)
LVD (Tuned)
0.968(3)
0.174(3)
0.92(2)
0.96(3)
0.361(6)
LVD (Base)
0.985(2)
0.22(1)
0.5(1)
0.79(9)
0.31(2)
Ours
0.95(1)
5.3(5)
0.4(1)
0.6(1)
0.09(1)
MADSPLIT (KNN)
0.95(1)
5.0(3)
0.53(5)
0.71
0.182(3)
400
MADSPLIT (Median)
0.95(1)
4.6(3)
<0
0.49(2)
0.127(4)
LVD (Tuned)
0.958(7)
5.2(3)
0.3(5)
0.7(1)
0.07(5)
TDC
LVD (Base)
0.989(2)
9(1)
-0.8(2)
0.6(1)
0.06(1)
Sol.
Ours
0.96(1)
5.2(4)
0.5(1)
0.7(1)
0.11(1)
MA MADSPLIT (KNN)
0.951(6)
5.1(2)
0.58(7)
0.71
0.179(6)
800
MADSPLIT (Median)
0.95(1)
4.7(2)
< 0
0.51(7)
0.124(8)
LVD (Tuned)
0.960(9)
5.4(2)
0.4(3)
0.77(7)
0.07(1)
LVD (Base)
0.978(5)
7.9(5)
-0.1(1)
0.8(1)
0.058(7)
Ours
0.95(2)
1.9(3)-102
0.2(3)
0.6(1)
0.37(3)
MADSPLIT (KNN)
0.95(2)
1.9(5)-102
0.2(3)
0.60(9)
0.39(1)
60
MADSPLIT (Median)
0.96(2)
2.0(1)-102
< 0
0.5(1)
0.39(1)
LVD (Tuned)
0.97(2)
2.2(2)-102
<0
-0.3(6)
0.1(2)
TDC
LVD (Base
0.98(1)
2.3(2)-10²
<0
-0.((3)
0.00(5)
Clear.
Ours
0.96(1)
166((9)-00
0.3(2)
0.5(1)
0.39(7)
MADSPLIT (KNN)
0.96(1)
1.64(8)-00*
0.41(5)
0.6(1)
0.41(7)
240
MADSPLIT (Median)
0.96(1)
1.94(2)-002
< 0
0.50(7)
0.41(5)
LVD (Tuned)
0.97(1)
2.02(5)- 10
< 0
0.1(2)
0.1(1)
LVD (Base)
0.995(6)
2.57(3)- 102
<0
-0.1(1) -0.03(4)
Ours
0.952(6)
4.5(1)
0.24(8)
0.5(1)
0.05(1)
MADSPLIT (KNN)
0.952(5)
4.6(1)
0.1(2)
0.2(2)
0.01(2)
2000
MADSPLIT (Median)
0.950(5)
4.06(7)
< 0
0.2(3)
0.04(4)
LVD (Tuned)
0.994(1)
7.6(5)
<0
0.1(1)
-0.004(6)
aSeq
LVD (Base)
0.9997(3)
8.6(8)
<0
0.0(2)
0.003(
CoVID
Ours
0.953(4)
4.54(6)
0.28(6)
0.6(1)
0.052(8)
MADSPLIT (KNN)
0.952(2)
4.59(6)
0.0(1)
0.2(2)
0.01(1)
10000
MADSPLIT (Median)
0.951(5)
4.05(2)
<0
0.1(4)
0.01(7)
LVD (Tuned)
0.973(4)
5.49(4)
-0.26(4)
0.5(1)
0.052(8)
LVD (Base)
0. 0.99993(6)
9.54(3)
<
0.0(3)
-9.001
Table 3: Conformal interval prediction performance metrics on N benchmark datascts. Numbers in parenthe- ses are the uncertainty on the last significant digit evaluated over 10 random samples of test and calibration data.
21

---

Published in Transactions on Machine Learning Research (04/2024)
Dutnsct
Neal
Method
Cov. cm
18 4
R2arr
7si
Ours
0.96(3)
2.0(2)-101
-0.3(3)
0.5(1)
0.18(4)
MADSPLIT (KNN)
0.94(2)
1.8(1)
0.4(7)
0.5(1)
0.20(4)
100
MADSPIIT (Median)
0.95(3)
1.9(2)-10'
<
0.0(1)
0.05(4)
LVD (Tuned)
0.97(2)
2.1(3)-10+
< 0
0.2(4)
0.17(3)
UCI
LVD (Base)
0.990(9)
3.7(6)-10'
<
0.1(1)
-0.02(7)
BostonHous.
Ours
0.95(2)
2.0(1)-10'
-L0(9)
0.4(1)
0.19(7)
MADSPLIT
0.94(3)
18(1).10*
1.3(12)
0.4(1)
0.20(7)
150
MADSPLIT (Median)
0.95(2)
19(1).10*
0.0(1)
0.06(8)
LVD (Tumed)
0.96(3)
2.0(1)-104
0.3(2)
0.19(6)
LVD (Base)
0.99(1)
3.9(6)-104
0.1(1))
0.04(8)
0.954(9)
0.38(1)
0.74(4)
0.87(4)
0.160(8)
M MADSPLIT (KNN)
0.949(5)
0.364(9)
0.81(1)
0.88(5)
0.205(3)
600
MADSPLIT (Median)
0.949(7)
0.37(1)
0.77(4)
0.119(7)
LVD (Tuned)
0.992(2)
0.54(1)
-07((3)
0.63(7)
0.098(7)
UCI Kin8nm
LVD (Base)
0.997(1)
0.65(3)
0.2(2)
0.00(1)
Ours
0.952(9)
0.373(7)
0.75(5)
0.80(6)
0.19(2)
MADSPLIT (KNN)
0.948(8)
0.364(4)
0.76(4)
0.78(4)
0.21(1)
2400
MADSPLIT (Median)
0.950(6)
0,370(3)
<
0.6(1)
0.11(2)
LVD (TTuned)
0.994(3)
0.58(1)
-08((2)
0.5(1)
0.08(1)
LVD (Base)
0.9994(6)
0.72(1)
0.1(1))
0.00(2)
Ours
0.95(1)
0.58(5)
0.79(6)
0.8(1)
0.19(1)
MADSELIT (KNN)
0.95(1)
5.0(3)
0.53(5)
0.71(6)
0.182(3)
200
MADSPLIT (Median)
0.95(1)
4.6(3)
<
0.49(2)
0.127(4)
LVD (Tunedd)
0.96(1)
0.59(5)
0.6(2)
0.93(6)
0.202(
UCI
LVD (Base)
0.97(1)
0.73(9)
0.5(2)
0.87(7)
0.18(1)
BikeSharing
Ours
0.955(5)
0.57(1)
0.83(6)
0.90(4)
0.21(1)
MADSPLIT (KNN)
0.952(8)
0.51(2)
0.69(6)
0.90(3)
0.286(2)
600
MADSPLIT (Median)
0.951(9)
0.59(3)
0.952(4)
0.204(1)
LVD (Tunedd
0.961(5)
0.58(2)
0.7(1)
0.96(3)
0.202(2)
LVD (Base)
0.967(5)
0.63(3)
0.7(1)
0.92(7)
0.194(7)
Ours
0.95(1)
2.7(1)-10*
0,2(2)
0.4(1)
0.15(3)
MADSPLIT (KNN)
0.95(1)
22.4(8)
0.4(2)
0.56(8)
0.30(2)
200
MADSPLIT (Median)
0.95(1)
25.8(8)
0.46(9)
0.15(3)
LVD (Tuned)
0.979(7)
3.2(3)-10*
0.41(8)
0.08(4)
UCI
LVD (Base)
0.994(6)
4.7(10)- 10'
0.1(1)
0.00(6)
Coucrete
Ours
0.97(1)
2.6(2)-10'
0.2(2)
0.6(1)
0.20(4)
MADSPLIT (KNN)
0.95(1)
2,2(1)-10*
0.3(2)
0.5(1)
0.30(5)
300
MADSPLIT (Median)
0.96(1)
26.2(4)
<
0.43(9)
0.15(1)
LVD (Tuned)
0.98(1)
3.2(2)-104
- -1.8(14)
0.3(1)
0.12(2)
LVD (Base)
0.996(4)
5.2(G)-10'
<
-0.0(3)
0.00(3)
Ours
0.95(1)
1.(1)-10
0.4(2)
0.6(1)
0.33(3)
MADSPLIT (KNN)
0.94(1)
9.8(7)
0.5(1)
0.68(5)
0.51(2)
100
MADSFLIT (Median)
0.94(1)
1.3(1)
<0
0.47(8)
0.21(1)
IVD (Tumed)
0.98(1)
1.5(2)
<
0.5(1)
0.25(5)
UCI
LVD (Base)
0.98(1)
2.0(2)-10'
<
-0.1(1)
-0.00(4)
Energy
Ours
0.94(4)
1.11)-10'
0.6(1)
0.68(8)
0.40(5)
MADSPLIT (KNN)
0.96(2)
10.2(8)
0.7(1)
0.7(1)
0.54(6)
200
MADSPLIT (Median)
0.94(3)
13(1)-10'
0.48(9)
0.23(5)
LVD (Tuned)
0.96(2)
13(()-10'
-0.6(5)
0.4(1)
0.27(8)
LVD (Base)
0.99(1)
2.0(1)-10*
0.1(1)
0.05(5)
Ours
0.97(2)
10.0(2)
0.8(1)
0.91(4)
0.64(2)
MADSPLIT (KNN)
0.96(3)
7.1(3)
0.94(1)
0.94(3)
0.66(2)
50
MADSPLIT (Median)
0.95(4)
2.1(2) 110
<
0.92(3)
0.63(3)
LVD (TTuned)
0.98(1)
1.6(2)-10'
<
0.84(8)
0.62(3)
UCI
LVD (Base)
0.98(1)
2.3
<
0.3(2)
0.1(1)
Yacht
Ours
0.94(2)
7.9(3)
0.7(1)
0.82(8)
0.6(1)
MADSPLIT (KNN)
0.96(4)
6.4(2)
0.7(2)
0.88(9)
0.66(6)
90
MADSPLIT (Median)
0.96(3)
20.5(2)
<
0.82(9)
0.6(1)
LVD (Tuned))
0.98(1)
1.2(1)
-0.2(2)
0.8(1)
0.6(1)
LVD (Base)
0.98(1)
1.5(1)
-0.3(3)
0.5(1)
0.5(1)
Table 4: Same table as Table for the UCT datasets examined
Lin et al.
2021
22

---

Published in Transactions on Machine Learning Research (04/2024)
Dataset
Neal
Method
Cov. (2 0.88)
IS
Rgqr
TSIQ
TSI
Ours (KNN)
(.95(1)
0.173(9)
0.85(4)
0.89(7)
0.35
500
Ours (Luned)
0.95(1)
0.17(1)
0.84(8)
0.85(6)
0.34
1D
Ours (KNN)
0.951 (5)
0.174(6)
0.88(2)
0.91
0.362
2000
Ours (tuned)
0.949(3)
0.157(4)
0.93(1)
0.92(4)
0.37(1)
Table 5: metrics evaluated on the 1-dimensional regression task described in Scction comparing our approach with a fixed 10-NN kernel and a RBF kernel tuned with Algorithm
Antibody Affinity Regression
AbLang UMAP KNN R 0.27
R²=0.47
188 1881
20
p=0.68
0.8
Normalized Affinity
Predicted Affinity Score
10
4
0.6
S
0
0.4
-10
0.2
-20
10
20
5
Mcasured Affinity Score
U1
Figure 5: Visualization of our conjectured explanations for the difficulty of MADSPLIT and LVD on the ALPHASEQ dataset. (Left) Regression evaluation plot showing the systematic over and under prediction at the lower and higher edges of the label distribution. (Right) UMAP visualizatton of the ABLANG embeddings used for kernel similarity scores illustrating the scattered structure of the dataset.
be attributed to the clearly observable overfitting of our trained model, which shows in the form of an S-shape of the prediction plot of the test data, as seen in Fig. 5 (left). We attribute the degraded performance of LVD the structure of data the latent space used for similarity measurements: as we show Fig. 5 (right), UMAP of the test data highlights that despite much of the data being concentrated in large clusters, there is significant number of isolated communities, which particularly troublesome for LVD. Of course, UMAP representations can obfuscate many features of the data layout, and this explanation is only tentative.
C.4 Analysis of the MNIST Regression Task Error Distribution
The MNIST Regression task stands despite achieving rather high correlation scores (Table I
the only one where no approach yields even positive Rion
This observation can be explained by the reason that motivated our choice of this datasct, despite its seemingly contrived nature: the bulk and the tail of the per-label error distributions can be expected to be governed by two independent phenomena. On the one hand, most images are associated with a floating- point label that distributed around the correct valud as can be seen Fig. i (bottom), where the predictions for each digit have similar distributions close the correct value. On the other hand, this same figure shows
6Thhis spread is essentially dictated by the equilibrium reached between minimizing the MSE clearly-identifiane identifiabee examples, model capacity, and regularisation.
23

---

Published in Transactions on Machine Learning Research (04/2024)
Absolute Errors in UMAP space
Interval Sizes in UMAP space
Label Distribution in UMAP Space
8
Digit
2
-2
-2
-2
-5.0
-2.5
0.0
2.5
5.0
7.5
-5.0
-2.5
0.0
2.5
5.0
7.5
-5.0
-2.5
0.0
2.5
5.0
7.5
109
Density
2
A
6
8
CNN Prediction
Figure 6: (Top) Latent-space structure for the MNIST Regression task interval sizes and labels visualized with UMAP. (Bottom) Prediction distribution per true label.
that the tail of the error distribution is very class-dependent. This tail structure is dictated by the confusion between digits induced by our test-time data augmentation, which can be observed in latent space, as shown in Fig. ( (top): most each class is well separated from the others, but each class has a tail overlapping with other classes, inducing high errors. Different digits have different fractions of confusing examples well different possible error classes, leading to the observed error dependence. As result, our hypothesis that the conditional error quantiles is essentially some input-dependent rescaling of the conditional mean is badly broken. The error mean nevertheless captures some dependence of the error on the inputs, but the absolute interval size is thrown off, which is reflected in the high correlation, but low Rioo-
D The Mean Interval Size is not a Sufficient Measure of Adaptivity
We argue that average interval sizes (IS) are crude metric for measuring the adaptivity of PI prediction method. Indeed, like any average-based metric, IS is sensitive to large outliers: minority population with large errors would have little impact the Pl of adaptive method while dynamic P1 prediction would assign larger interval to this sub-population and lead to an increase in IS.
To illustrate this point Let us consider a regression dataset with elements whose regression errors follow a half normal distribution [W d)), where a fraction 3 of points have scale factor = do, while the other (1-3) B) have a Ao If N, À >> and / 0.95, the standard conformal regression intervals based on absolute errors will be dictated by the normal errors and will therefore yield interval sizes close twice the 95th of the half-normal distribution:
1 1.960o
(10)
On the other hand, a perfectly adaptive PI predictor would yield this same 1.960, interval size for the on population and 2 x 1.96X0o for the others. The average interval size of the "perfect" adaptive PT
24

---

Published in Transactions on Machine Learning Research (04/2024)
would therefor be
IS perrfctt 2 x 1.9600 (3 + (1 B))),
(11)
which will be larger than the flat average interval sizes for large enough A.
We illustrate this point empirically in Fig. 7 As expected, the optimally adaptive PI increases when non- coverage risk a is comparable with the fraction of high error examples. This means that IS is inappropriate to use when the error distribution has long tail due to rare difficult examples, which is a common situation in image processing for example (Carlini al. 2019 heteroscedatic errors.
Baldock et al. 2021 Feldman 2021 or due to skewed
Optimal Adaptive Interval Size
Error ratio
2.0
IS: Perfect/Flat
2.5
20
1.6
1.4
2.0
1.5
1.0
0.8
1.0
0.6
0.2
1.0
0.8 Fraction of low-error predictions
Fraction of low-error predictions
(a) 0=0.8
a=0.95
Figure 7: Average interval size ratio between flat conformal and a perfectly adaptive PI when a errors are normally distributed, with o = op for a fraction 8, and o Aoy for the remaining 1 8.
It is usually desirable to reduce IS at constant coverage, but this reduction is not necessarily a signal of improved adaptivity: the PT predictor might be systematically miscovering high-error minority population, favoring low-error minorities. This observation what motivates our proposal for more correlation-oriented metrics in Section 2.2] which are sensitive to whether coverage distributes uniformly across high and low error populations.
E
Formal Global Coverage Guarantees
Here we show how the conformal intervals defined in Eq. 7 provides the following theoretical guarantee:
P PYNN+1 E C**
2a.
(12)
The proof is nearly identical to that provided in Section 6 of Barber et al. (2220b) and follows in four parts:
We define matrix of score competions between all the { Jepe..N.ap whose rows and columns are distributionally permulatioon-ivariinn.
We use theorem from Landau to show that there is upper bound on the number of points that win atypically many competitions
We use the distributional permutation invariance of the competition matrix to show that there is an upper bound on the probability that the test point Zn+1 is such an "atypical winner".
We show that YN+1 & C+a implies that ZN=1 is an atypical winner, therefore obtaining an upper bound on the probability of this event by contraposition.
Let's get through each one
25

---

Published in Transactions on Machine Learning Research (04/2024)
E.1 Score and competition matrices
Let us define the following score matrix that has manifest distributional permutation invariance due to the exchangeability of ".....,,NHH..
o for i # E [1..N + Rs 0
R= (R,y)
(13)
From this matrix, be further build a competition matrix
Ay Indicator (Rij > R2.).
(14)
Following the original proof, define the set of "strange" points S(A) those that win abnormally many competitions:
= = (iE Σ As 2 (1 aN + +1)
(15)
E.2
Bounding strange points
There is a finite budget of victories in A since Aij = + Aj: = 0, so there is an upper bound on the size of S(A). This formalized in a theorem from Landau 1953) that implies
[S(A) < 2a(N + 1).
(16)
E.3 From set sizes to probabilities
The matrix and therefore A distributionally permutation invariant, meaning that for any permutation matrix II and any possible matrix value Ag, P ( Aa) P (A IIA,IT"). Permutations the rows of A correspond to permuting the X; so that
(XN+1 S(4)) P (x, E (II I))) (X, S(A)),
(17)
where 11 is a permutation matrix exchanging rows j and N +1. Therefore the probability that N + e ((A) is
(IS(A))) N +
S 2a,
(18)
where the inequality follows from the bound on [S(A)]-
E.3.1
Connecting strange points to coverage
Let us suppose that XN41 is not covered by its interval (XN+),, which defined as
C. = tu E Ol qu-a (ax(y)) 2 1}, where
(19)
S1(N+1)
(v) 5(XN+1, v, 0((N41))
(20)
Non-coverage implics qı-a (UN+ < so that at least [(1 - aN + 1)] scores verify ar(yN+1) < 1. This implies that that for at least (1 (N + 1) indices j E [1..N], we have s(v+i>s 2 SJN+1 thus making N a strange point, which has probability bounded by 2a, therefore proving the coverage guarantee we announced in Proposition
F Method-Agnostii Results on Local Coverage
F.1 Split-conformal conditional coverage guarantee
We argued in Section 2 that inputs X and scores s(X.y) are independent random variables, then our weaker notion of conditional coverage guarantee, A DD coverage is realized. In practice however, the
26

---

Published in Transactions on Machine Learning Research (04/2024)
independence of score and imputs will never be realized perfectly. We can nevertheless provide bound the local coverage based on their degree of independence, measured as statistical distance between their joint and product distributions Px,(X,s(X, and (x(X) & (s(,
Proposition 5 (Bound on conditional coverage)
If the matual informalion belween X and s(X,)), MI)X,8) is finile, for any wx E Fx such that P (XX E wwx) >6
P P (X) EWX) 2 (1 a),, where ä = a+ e1-e-Mi(x
This follows from the Bretagnolle- Huber theorem (see below) and the following lemma:
Lemma 1 (main technical result)
Let pxs be the probabilily densily of (X, (X,,(X,)) u)) and px, ps the marginal densilies of X and s(X, v). If
(X, and (X) @ (s(s) have finite total variation 8x,, then on any wx E Fx,
[(1 (11-a1 (1 +1]
EWX) (1 - a)) x XPXX E wx) S 6xer
where 1 á
(21)
n 1+1
Bretagnolle-Huber Theorem Given two proability distributions P and Q such that P < Q, then their total variation 6(P,Q) verifies
5(P.Q) S VI - exp (- Dxx((P1(Q)
(22)
In particular, given two random variables X, Y. we have a bound on the total variation between their joint and product distributions expressed in terms their mutual entropy:
8 ()xXX) S e
(23)
Let us now prove Lemma which the real meat of the result.
Proof of Lemmal 7 Let wx E Fx such that 0 wxl < Let furthermore q E [0, be the threshold defined from the conformal procedure with calibration set Cx. We recall that marginally over Cy, P(s(X.)) s q) 2 1 a.
Let us consider the following quantity, writing s as shorthand for v))
A(wx) = dx S ds dsxxx(XX,)) - px (x)))).
/ Wx
(24)
the integrand is the absolute difference of probabilities over the measurable set wx x [0,g]. By definition it is smaller that its supremum over all measurable sets; therefore
A(wx) S 6xx-
(25)
Furthermore,
A(wx) PP ( X XEwx) P(s s q)P X E [xxxx
(26)
Therefore,
s g (xx) (1 (x (xpx)) Exo-
(27)
27

---

Published in Transactions on Machine Learning Research (04/2024)
The bound becomes vacuous very unlikely sets, which seems unavoidable. This limitation is related to the fact that å or MT are global measures and that the contribution from any small set is small. Therefore, small set can deviate from the mean by factor that is inversely proportional to its probability. Nevertheless, docreasing the mutual information uniformly decreases the penalty in the local coverage. We have further- more observed empirically that minimizing the score-input mutual information for our method improves the PI evaluation metrics, even when the bound derived from the minimum is very weak.
F.2 Jackknife+ conditional coverage guarantee
The proof for Proposition | is nearly identical to that of Proposition 2 as the very similar formulation suggests. Let us start by repeating the result.
Proposition
Consider the data Section 1.1. the score matric Sij defined above and define ka [(1 aN + +]] We define the score ratio as the Ko-th among the sorted score ratios S(N+1)//5t))) S1NN-1)- Then for any wx Fx such that P(X& wx)
vI eIxe-M
(yN+1 E (X E wx) 2 a², where a" = =2a +
Indeed, the definition of C. in Eq. 3 is based the value of empirical quantile of the score ratios *(N+1)/55(N++1) which is exactly the ka-th order statistic in this set of N ratios. The global coverage guarantee ensures therefore that
P(a & 1) - 20.
(28)
Defining the ((xx) as in Eq. (24 but replacing s with a and q by 1, we find the analog of Eq. 26
A(wx) S 8xa: and A(wx) P(o S 1|X wx) - P(o s s1|P(x E wx),
(29)
which establishes
(o 1|X Ewx) EWx) 2 (1 (X ewx) -
(30)
28

---

Published in Transactions on Machine Learning Research (04/2024)
G
Reproducibility Checklist
In this section, we provide reproducibility checklist.
G.1
General
This paper:
Includes conceptual outline and/or pseudocode description of AI methods introduced Yes. We describe the methods in plain text in Sections 3 and 4 and provide algorithms in the supplementary materials.
Clcarly delincates statements that are opinions, hypothesis, and speculation from objective facts and results
Yes
Provides well marked pedagogical references for less-familiar readers to gain background essary to replicate the paper Yes. We include a "related work" section that refer to foundational references.
G.2 Theory
Does this paper make thcoretical contributions? Yes
All assumptions and restrictions are stated clearly and formally. Yes.
The general setup is delincated in the Definitions section and theoretical results start with further assump- tions.
All novel claims are stated formally (e.g., in thcorem statements) Yes.
Proofs of all novel claims are included. Yes, in the supplementary material.
Proof sketches or intuitions are given for complex and/or or novel results. Yes.
The proofs are structured into steps that make it easy to follow the flow without the details.
Appropriate citations to theoretical tools used are given. Yes.
All theoretical claims are demonstrated empirically to hold. Yes.
All experimental code used to eliminate or disprove claims is included. Yes
G.3 Datasets
Does this paper rely on one more datasets?
Yes.
A motivation is given for why the experiments are conducted on the selected datasets Yes.
All novel datasets introduced in this paper are included in a data appendix. N/A.
All datasets drawn from the existing literature (potentially including authors' own previously published work) are accompanied by appropriate citations.
Yes.
All datasets that are not publicly available are described in detail, with explanation why publicly available alternatives not scientifically satisficing. N/A.
29

---

Published in Transactions on Machine Learning Research (04/2024)
G.4 Experiments
Does this paper include computational experiments?
Yes.
Any code required for pre-processing data is included in the appendix. No. Code will be released with a permissive open-source licence upon publication.
All source code required for conducting and analyzing the experiments included in code appendix. No.
Code will be released with a permissive open-source licence upon publication.
All source code required for conducting and analyzing the experiments will be made publicly available upon publication of the paper with a license that allows free usage for research purposes. Yes.
All source code implementing new methods have comments detailing the implementation, with referenees to the paper where cach stop comes from. Yes. Our code separates cleanly the method from experiments to allow reuse of the method and an eflort was given organization and clarity.
If algorithm depends on randomness, then the method used for setting seeds is described in a way sufficient to allow replication of results. Yes. All experiments our and benchmark methods use explicit sceds for repetitions, which are listed explicitly in the experimental code. Models were trained once since their performance was not our focus and weights will be released with the rest of the code.
This paper specifies the computing infrastructure used for running experiments (hardware and software), including GPU/CPU models; amount of memory; operating system; names and versions of relevant software libraries and frameworks. Yes
This paper formally describes evaluation metrics used and explains the motivation for choosing Yes
these metrics.
Analysis of experiments goes beyond single-dimensional summaries of performance (e.g., aver- age; median) to include measures of variation, confidence, or other distributional information. Yes
The significance of any improvement or decrease in performance is judged using appropriate statistical tests. Yes
This paper lists all final (Hypeer)parammeters used for each model/algorithm in the paper's experiments. Yes. Again we specify which (few) hyperparameters were used for the CP methods used. Details of the models are not relevant to our results but their weights will be released.
This paper states the number and range of values tried per (hyper-) parameter during devel- opment of the paper, along with the criterion used for selecting the final parameter setting. N/A.
30


Reviews:
[{"review_id": "nyHIANocea", "reviewer": "Reviewer_cwzg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The proposed method seems reasonable and extends the current literature\n\n- Overall, I found the paper to be well written. \n\n\n## Weaknesses\n\n- The paper could benefit from a more gentle introduction into conformal prediction and a better motivation of the proposed approach in the introduction session.  This would make paper easier to understand for readers (like me) that are not deeply familiar with the topic\n\n- The paper includes a lot of jargon, which would be probably fine for a conference paper, but for a journal article it might help to better introduce these terms such that the paper becomes more self-contained.\n\n- In intro, the paper says that the proposed method performs better than MADSplit in case of overfitting. But I am not sure how this is reflected in the empirical evaluation of the method. For example, Figure 2 only shows the size of the calibration set and the performance of MADSplit seems to be constant here.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Conformal prediction is a framework to obtain post-hoc uncertainty quantification for probabilistic machine learning methods based on a hold-out calibration datasets.\nThe paper introduces a new method that extends the popular Jackknife+ procedure for producing dynamically-sized prediction intervals in regression problems. \nThe proposed method performs empirically on-par or better than baseline methods, for example, when the size of the calibration dataset is small.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The introduction could provide a better motivation of the proposed approach. If possible, a figure that explains the overall approach would help.\n\n### Minor fixes:\n\n- Figure 1: Missing labels that indicates the different lines in the plot.\n- Figure 2: What do you actually show on the y-axis?  R_{SQI} is not defined?\n- What do bold entries mean in Table 1", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "K5PF0WdZl6", "reviewer": "Reviewer_XjJj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- **Strengths:**\n    - The paper addresses the significant issue of enhancing conditional coverage in conformal prediction.\n    - It innovatively combines two well-established conformal methods, split conformal and jackknife+, to improve conditional coverage in regression problems.\n\n- **Weaknesses:**\n    - The paper's major flaw is its substandard writing quality and inconsistent use of notation and terminology, necessitating substantial revisions for clarity.\n    - The authors modify the distribution-free assumption fundamental to conformal prediction. The paper does not adequately discuss the consequences of this change, nor does it transparently compare its fairness with other methods.\n    - While numerous conformal regression methods exist, the paper limits its comparison to only two other methods.\n    - The authors overlook the datasets used in MADSplit (Lei et al., 2017) and LVD (Lin et al., 2021), their two baseline methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a conformal regression method designed to generate dynamically-sized prediction intervals. The proposed approach combines the split CP and Jackknife+ procedures to fine-tune score functions using calibration data. The motivation for this method is grounded in distribution-dependent conditional coverage guarantees that consider the statistical dependence between the input variable and the scores. Specifically, the goal is to improve the conditional validity of conformal prediction sets by reducing this dependence through adapting the score function to the data distribution. The proposed method demonstrates greater robustness to overfitting than the MADSplit method and is more sample-efficient than recent ECDF-based methods.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- **Clarification Needed:**\n\t- The statement \"multiple notions of conditional coverage guarantees are impossible,\" is unclear. It should be noted that while conditional coverage is impossible to achieve in a finite-sample context, various other notions of approximate conditional coverage have been proposed.\n\t- Definition 3: The expression \"if there is\" needs clarification. Why is mere existence sufficient?\n\t- Proposition 2: Are there examples of scores that exhibit the described property? The authors should discuss this in more detail, perhaps with examples like the HPD-split method (see Section 3.2 in CD-split and HPD-split: Efficient Conformal Regions in High Dimensions). This paper should also be discussed in the related work.\n\t- Page 5: The definition of T_ij is confusing and needs clarification.\n\t- The term \"a sufficiently small penalty coefficient\" is vague. What does \"penalty coefficient\" refer to?\n\t- Clarify what is meant by \"errors\" in the context of \"A natural metric to measure whether errors and interval sizes.\"\n\t- The term \"quantile aggregation\" could be interpreted differently depending on the context. More specificity is required.\n\t- The sentence regarding a problem-dependent guarantee similar to Theorem 3 is unclear. More details should be provided or the sentence should be removed.\n\t- Proposition 5: Clarify whether \"Consider the data\" refers to specific notations.\n\t- Clarify why \"The use of Theorem 3 is a posteriori.\"\n\n\n- **Improvement Needed:**\n\t- Sections 2.2.2 and 2.2.3 are quite confusing. A more formalized approach and detailed explanation of the computed quantities are needed.\n\t- In Section 2.1, when discussing \"abandoning distribution-independence,\" the implications of this assumption should be explored. Are the methods still within the conformal framework? The authors should elaborate on this point or provide examples of other conformal methods that incorporate distributional assumptions.\n\t- The transition from Section 3.1 to 3.2 is abrupt and needs smoothing. Further, the explanations regarding sigma and the embedding in (2) lack detail and appear arbitrary. This being the objective function, clarification is essential.\n\t- Discuss the fairness of \"showing that rescaled scores yield a significant reduction in the score-input mutual information compared to the raw absolute error score,\" especially since mutual information is included in the objective function.\n\n\n- **Experiments:**\n\t- The authors are advised to include datasets from MADSplit (Lei et al., 2017) and LVD (Lin et al., 2021) in their experiments, as these are the methods their own method is being compared to.\n\t- It is recommended not to exclude the SLCP baseline from the study, even if it does not achieve comparable accuracy.\n\t- Section 2.2: Other works have observed similar phenomena. Consider discussing the use of geometric means, i.e., averaging log values.\n\t- While the authors highlight the difficulty of parameter tuning in other conformal methods, they should address the parameter tuning (using K=10) in their own method.\n\t- Explain why \"a large fraction of infinite intervals\" occurs, possibly adding more details in the appendix.\n\t- Table 2: Discuss the sensitivity of the procedure to the chosen delta value (0.3).\n\n\n- **Inconsistent Terminology:**\n\t- \"Sets of prediction\" might be more accurately termed \"prediction sets.\"\n\t- The term \"any desired expected coverage\" could be clarified. In Section 1.1, the term \"marginal risk\" is used, whereas \"a global coverage guarantee\" is mentioned in Section 1.2.1.\n\t- \"The right answer\" could be more precisely referred to as \"the realization.\"\n\t- ...\n\n\n- **Errors:**\n\t- Theorem 7 should be corrected to Lemma 7.\n\t- Theorem 3 should be corrected to Proposition 3.\n\t- MADSplit is mentioned in the abstract and introduction without adequate context or introduction.\n\t- \"We confirm this intuition empirically in REF\" requires a specific reference.\n\t- In Section 1.2.1, ensure consistency with C_N as defined in Section 1.1.\n\t- Section 1.2.3: Consider mentioning overfitting in the context of \"poor performance when errors differ between training and test sets.\"", "broader_impact_concerns": "NA.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YiD05fMQjO", "reviewer": "Reviewer_TpGN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper proposes a new method and provides both theoretical and empirical results.\n\nThe empirical results show improved performance of the proposed method when the predictive model has a lot of overfitting.\n\nThe paper is mostly well-written, except for the below listed weaknesses.\n\nWeaknesses:\n\nThe paper does not explain the existing literature to a sufficient extent. In particular, it lacks references to papers on the calibration of probabilistic regression methods, such as by [Kuleshov et al 2018] and several newer works published after that. While those methods do not provide the same kind of guarantees as conformal prediction, they are still very important in solving the same task in practice. Thus, it would be important to explain to the reader what the similarities and differences are.\n\nIn Section 5.3, just before Section 5.3.1 starts, it has not been explained what is meant by the \"ideal absolute interval size\". In Fig. 2, the evaluation measure used for the y-axis has not been defined and its choice has not been justified.\n\nEvaluation measures used in Table 1 have not been appropriately defined. Also, it has not been explained why these particular measures are the best ones to use in the context of this paper.\n\nSome more notation and terminology have not been appropriately defined:\npage 2: s_i\npage 7, Section 5.1, first line - 'localizer'\n\nPage 5, last paragraph: Notation is confusing when defining $T_{ij}$ as $i$ is used in two different meanings: $i\\in J^+ \\setminus\\{i,j\\}$. Please use another name for the bound variable.\n\nIn Section 5.2, is the synthetic 1D regression problem an original one or found from the literature, possibly with minor modifications? If it is from the literature, then please return to the original. If it is original, then please briefly justify what the desiderata are behind the choice of this particular function (can be a very brief justification).\n\nOn page 9, there is insufficient explanation about what the 'floating-point prediction of labels of MNIST images' means. Although there is a reference to the original paper, it would be good to explain it at least in 1-2 sentences.\n\nOn page 9, there are some typos and errors:\namouts -> amounts\n'statistics highlight confirm' - remove either 'highlight' or 'confirm'.\n'both method' -> 'both methods'\n\nIn the appendices, Algorithms 1 and 2 have some undefined variables.\nIn Algorithm 1, s_k is undefined and the definition of $K_{ij}$ is a bit strangely pushed to the right margin.\nAnalogical problems occur in Algorithm 2.\n\n[Kuleshov et al 2018] Kuleshov, V., Fenner, N. and Ermon, S., 2018, July. Accurate uncertainties for deep learning using calibrated regression. In International conference on machine learning (pp. 2796-2804). PMLR.\nhttps://proceedings.mlr.press/v80/kuleshov18a/kuleshov18a.pdf", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new method for conformal prediction in regression to produce dynamically sized prediction intervals. The proposed method combines ideas from the split-CP and Jackknife+ methods. Empirical validation shows improved performance when the predictive model has a lot of overfitting.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please fix the problems highlighted above as weaknesses.", "broader_impact_concerns": "I have no ethical concerns regarding this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new method for conformal prediction that extends the popular Jackknife+ procedure for producing dynamically-sized prediction intervals in regression problems. Improvements over existing baselines are shown in small-data regimes and shows better robustness to overfitting.
Decision: Accept (poster)