Conformal Prediction using Conditional Histograms

1 Introduction . 1.1 Problem statement and motivation . We consider the problem of predicting with confidence a response variable Y ∈ R given p features X ∈ Rp for a test point n+1 , utilizing n pairs of observations { ( X ( i ) , Y ( i ) ) } ni=1 drawn exchangeably ( e.g. , i.i.d . ) from some unknown distribution , and leveraging any machine-learning algorithm . Precisely , ∀α ∈ ( 0 , 1 ) , we seek a prediction interval Ĉn , α ( Xn+1 ) ⊂ R for Yn+1 satisfying the following three criteria . First , Ĉn , α should have finite-sample marginal coverage at level 1− α , P [ Yn+1 ∈ Ĉn , α ( Xn+1 ) ] ≥ 1− α . ( 1 ) Second , Ĉn , α should approximately have conditional coverage at level 1− α , P [ Yn+1 ∈ Ĉn , α ( x ) | Xn+1 = x ] ≥ 1− α , ∀x ∈ Rp , ( 2 ) meaning it should approximate this objective in practice , and ideally achieve it asymptotically under suitable conditions in the limit of large sample sizes . Third , Ĉn , α should be as narrow as possible . We tackle this challenge with conformal inference [ 24 , 35 ] , which allows one to convert the output of any black-box machine learning algorithm into prediction intervals with provable marginal coverage ( 1 ) . The key idea of this framework is to compute a conformity score for each observation , measuring the discrepancy , according to some metric , between the true value of Y and that predicted 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . by the black-box model . The model fitted on the training data is then applied to hold-out calibration samples , producing a collection of conformity scores . As all data points are exchangeable , the empirical distribution of the calibration scores can be leveraged to make predictive inferences about the conformity score of a new test point . Finally , inverting the function defining the conformity scores yields a prediction set for the test Y . This framework can accommodate almost any choice of conformity scores , and in fact many different implementations have already been proposed to address our problem . However , it remains unclear how to implement a concrete method from this broad family that can lead to the most informative possible prediction intervals . Our contribution here is to develop a practical solution , following the three criteria defined above , that performs better compared to existing alternatives and is asymptotically optimal under certain assumptions . It is worth emphasizing that constructing a short prediction interval with guaranteed coverage is a reasonable approach to quantify and communicate predictive uncertainty in regression problems , although it is of course not the only one . To name an alternative , one could compute a non-convex prediction set with analogous coverage [ 20 ] , which might be more appropriate in some situations , but is also more easily confusing . For example , it could be informative for a physician to know that the future blood pressure of a patient with certain characteristics is predicted to be within the range [ 120,129 ] mmHg . However , it would not be more helpful to report instead the following non-convex region : [ 120 , 120.012 ] ∪ [ 120.015 , 120.05 ] ∪ [ 121 , 122.7 ] ∪ [ 123.1 , 127.2 ] ∪ [ 127.8 , 129 ] mmHg . Indeed , in the second case it would not be clear ( a ) whether the multi-modal nature of that prediction is significant or a spurious consequence of overfitting , and ( b ) how the physician would act upon that prediction any differently than if it had been [ 120,129 ] mmHg . Therefore , we focus on prediction intervals in this paper because they are generally easier to interpret than arbitrary regions , and they are also less likely to convey a false sense of confidence . 1.2 Preview of conformal histogram regression . Imagine an oracle with access to PY |X , the distribution of Y conditional on X , which leverages such information to construct optimal prediction intervals as follows . For simplicity , suppose PY |X has a continuous density f ( y | x ) with respect to the Lebesgue measure , although this could be relaxed with more involved notation . Then , the oracle interval for Yn+1 | Xn+1 = x would be : Coracleα ( x ) = [ loracle1−α ( x ) , u oracle 1−α ( x ) ] , ( 3 ) where , for any τ ∈ ( 0 , 1 ] , loracleτ ( x ) and uoracleτ ( x ) are defined as : [ loracleτ ( x ) , u oracle τ ( x ) ] : = argmin ( l , u ) ∈R2 : l≤u { u− l : ∫ u l f ( y | x ) dy ≥ τ } . ( 4 ) This is the shortest interval with conditional coverage ( 2 ) . If the solution to ( 4 ) is not unique ( e.g. , if f ( · | x ) is piece-wise constant ) , the oracle picks any solution at random . Of course , this is not a practical method because f is unknown . Therefore , we will approximate ( 4 ) by fitting a black-box model on the training data , and then use conformal prediction to construct an interval accounting for any possible estimation errors . Specifically , we replace f in ( 4 ) with a histogram approximation , hence why we call our method conformal histogram regression , or CHR . The output interval is then Ĉn , α ( x ) = [ l̂τ̂ ( x ) , ûτ̂ ( x ) ] , ( 5 ) where l̂τ̂ ( x ) and ûτ̂ ( x ) approximate the analogous oracle quantities in ( 4 ) . The value of τ̂ in ( 5 ) will be determined by suitable conformity scores evaluated on the hold-out data , and it may be larger than 1 − α if the model for f is not very accurate . However , if the fitted histogram is close to the true PY |X , the interval in ( 5 ) will resemble that of the oracle ( 3 ) . Figure 1 previews an application to toy data , comparing CHR to conformalized quantile regression ( CQR ) [ 30 ] ; see Section 4.2 for more details . CHR finds the shortest interval such the corresponding area under the histogram is above τ , for any τ ∈ ( 0 , 1 ] , and then calibrates τ to guarantee marginal coverage above 1−α ; this extracts more information from the model compared to CQR . For example , CHR adapts automatically to the skewness of Y | X , returning intervals delimited by the 0 % –90 % quantiles in this example , which are shorter than the symmetric ones ( 5 % –95 % ) sought by CQR . 1.3 Related work . This work is inspired by the conformity scores introduced by [ 31 ] for multi-class classification , the underlying idea of which can be repurposed here . Nonetheless , the extension to our problem involves several innovations . This paper connects [ 31 ] to other conformal methods for continuous responses [ 24 , 25 , 35 ] , which sought objectives similar to ours by leveraging quantile regression [ 17 , 23 , 30 , 32 , 36 ] or non-parametric density estimation [ 10 , 21 ] , sometimes considering multi-modal prediction sets instead of intervals [ 20 ] . Our approach also exploits black-box models for distributional estimation ; however , we introduce more efficient conformity scores . We seek the shortest intervals with marginal coverage while approximating as well as possible conditional coverage , although the latter is impossible to guarantee in finite samples [ 16 , 34 ] . The performances of prior approaches have been measured in terms of these criteria , yet others have not sought them as directly . Indeed , if the black-box model is consistent for PY |X , our method becomes asymptotically equivalent to the oracle ( 3 ) – ( 4 ) , under some technical assumptions . This property does not hold for other existing methods because they tend to produce symmetric intervals , with fixed lower and upper miscoverage rates ( the probabilities of the outcome being either below or above the output interval , respectively ) , which may be sub-optimal if the data have unknown skewness . 2 Methods . The proposed method consists of four main components : the estimation and binning of a conditional model for the outcome , the construction of a nested sequence of approximate oracle intervals based on the above , the computation of suitable conformity scores , and their conformal calibration . 2.1 Estimating conditional histograms . We partition the domain of Y into m bins [ bj−1 , bj ) , for some sequence b0 < . . . < bm . With little loss of generality , assume Y is bounded : −C = b0 < Y < bm = C , for some C > 0 . Then , we solve a discrete version of the problem stated in the introduction : we seek the smallest possible contiguous subset of bins with 1− α predictive coverage . If m is large and the bins are narrow , this problem is not very different from the original one , although it is more amenable to solution . For simplicity , we present our method from a split-conformal perspective [ 24 , 35 ] ; extensions to other hold-out approaches [ 7 , 22 , 35 ] will be intuitive . Let Dtrain , Dcal ⊂ { 1 , . . . , n } denote any partition of the data into training and calibration subsets , respectively . Dtrain is used to train a black-box model for the conditional probabilities that Y is within any of the above bins : ∀j ∈ { 1 , . . . , m } , πj ( x ) : = P [ Y ∈ [ bj−1 , bj ) | X = x ] . ( 6 ) There exist many tools to approximate PY |X and obtain estimates π̂j ( x ) of πj ( x ) , including quantile regression [ 27 , 28 , 33 ] , Bayesian additive regression trees [ 11 ] , or any other non-parametric conditional density estimator [ 14 , 19 , 26 ] . Our method can directly be applied with any of these models , but we found multiple quantile regression to work particularly well [ 17 , 23 , 30 , 32 ] , and therefore we will focus on it in this paper . Referring to Supplementary Section S1.1 for implementation details and information about the computational cost of the learning algorithm ( which is comparable to that required by CQR [ 30 ] ) , we thus take these black-box estimates π̂j ( x ) as fixed henceforth . Note that estimating conditional distributions is more challenging if the number of variables is larger . However , this is a fundamental difficulty of high-dimensional regression , not a particular limitation of the proposed CHR . Although our method utilizes conditional histograms learnt from the data , its performance is not directly measured in terms of how closely these resemble the true PY |X . Instead , as we shall see , CHR only needs to detect the possible skewness of Y | X and estimate reasonably well some lower and upper quantiles of this conditional distribution . Therefore , its estimation task is not much more difficult than that of CQR [ 30 ] , as skewness is relatively easy to detect .

Summary: This paper introduces a conformity score that aims to decrease average interval lengths and improve conditional coverage from a black-box estimate  $\hat{P}(y \mid x)$ for split conformal prediction. The method, conformal histogram regression (CHR), involves first binning the space of $Y$, resulting in a conditional histogram from which approximate oracle intervals can be computed. A nested sequence of these intervals is then created for a sequence of predictive miscoverage values $\tau$, where $\tau$ will be close to $\alpha$ if $\hat{P}(y \mid x)$ is a good estimate. The value of $\tau$ is selected through a conformity score, and the authors show their method obtains finite marginal coverage and asymptotic conditional coverage. The authors then demonstrate the method on a few examples.