Conformal Prediction using Conditional Histograms

1 Introduction . 1.1 Problem statement and motivation . We consider the problem of predicting with confidence a response variable Y ∈ R given p features X ∈ Rp for a test point n+1 , utilizing n pairs of observations { ( X ( i ) , Y ( i ) ) } ni=1 drawn exchangeably ( e.g. , i.i.d . ) from some unknown distribution , and leveraging any machine-learning algorithm . Precisely , ∀α ∈ ( 0 , 1 ) , we seek a prediction interval Ĉn , α ( Xn+1 ) ⊂ R for Yn+1 satisfying the following three criteria . First , Ĉn , α should have finite-sample marginal coverage at level 1− α , P [ Yn+1 ∈ Ĉn , α ( Xn+1 ) ] ≥ 1− α . ( 1 ) Second , Ĉn , α should approximately have conditional coverage at level 1− α , P [ Yn+1 ∈ Ĉn , α ( x ) | Xn+1 = x ] ≥ 1− α , ∀x ∈ Rp , ( 2 ) meaning it should approximate this objective in practice , and ideally achieve it asymptotically under suitable conditions in the limit of large sample sizes . Third , Ĉn , α should be as narrow as possible . We tackle this challenge with conformal inference [ 24 , 35 ] , which allows one to convert the output of any black-box machine learning algorithm into prediction intervals with provable marginal coverage ( 1 ) . The key idea of this framework is to compute a conformity score for each observation , measuring the discrepancy , according to some metric , between the true value of Y and that predicted 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . by the black-box model . The model fitted on the training data is then applied to hold-out calibration samples , producing a collection of conformity scores . As all data points are exchangeable , the empirical distribution of the calibration scores can be leveraged to make predictive inferences about the conformity score of a new test point . Finally , inverting the function defining the conformity scores yields a prediction set for the test Y . This framework can accommodate almost any choice of conformity scores , and in fact many different implementations have already been proposed to address our problem . However , it remains unclear how to implement a concrete method from this broad family that can lead to the most informative possible prediction intervals . Our contribution here is to develop a practical solution , following the three criteria defined above , that performs better compared to existing alternatives and is asymptotically optimal under certain assumptions . It is worth emphasizing that constructing a short prediction interval with guaranteed coverage is a reasonable approach to quantify and communicate predictive uncertainty in regression problems , although it is of course not the only one . To name an alternative , one could compute a non-convex prediction set with analogous coverage [ 20 ] , which might be more appropriate in some situations , but is also more easily confusing . For example , it could be informative for a physician to know that the future blood pressure of a patient with certain characteristics is predicted to be within the range [ 120,129 ] mmHg . However , it would not be more helpful to report instead the following non-convex region : [ 120 , 120.012 ] ∪ [ 120.015 , 120.05 ] ∪ [ 121 , 122.7 ] ∪ [ 123.1 , 127.2 ] ∪ [ 127.8 , 129 ] mmHg . Indeed , in the second case it would not be clear ( a ) whether the multi-modal nature of that prediction is significant or a spurious consequence of overfitting , and ( b ) how the physician would act upon that prediction any differently than if it had been [ 120,129 ] mmHg . Therefore , we focus on prediction intervals in this paper because they are generally easier to interpret than arbitrary regions , and they are also less likely to convey a false sense of confidence . 1.2 Preview of conformal histogram regression . Imagine an oracle with access to PY |X , the distribution of Y conditional on X , which leverages such information to construct optimal prediction intervals as follows . For simplicity , suppose PY |X has a continuous density f ( y | x ) with respect to the Lebesgue measure , although this could be relaxed with more involved notation . Then , the oracle interval for Yn+1 | Xn+1 = x would be : Coracleα ( x ) = [ loracle1−α ( x ) , u oracle 1−α ( x ) ] , ( 3 ) where , for any τ ∈ ( 0 , 1 ] , loracleτ ( x ) and uoracleτ ( x ) are defined as : [ loracleτ ( x ) , u oracle τ ( x ) ] : = argmin ( l , u ) ∈R2 : l≤u { u− l : ∫ u l f ( y | x ) dy ≥ τ } . ( 4 ) This is the shortest interval with conditional coverage ( 2 ) . If the solution to ( 4 ) is not unique ( e.g. , if f ( · | x ) is piece-wise constant ) , the oracle picks any solution at random . Of course , this is not a practical method because f is unknown . Therefore , we will approximate ( 4 ) by fitting a black-box model on the training data , and then use conformal prediction to construct an interval accounting for any possible estimation errors . Specifically , we replace f in ( 4 ) with a histogram approximation , hence why we call our method conformal histogram regression , or CHR . The output interval is then Ĉn , α ( x ) = [ l̂τ̂ ( x ) , ûτ̂ ( x ) ] , ( 5 ) where l̂τ̂ ( x ) and ûτ̂ ( x ) approximate the analogous oracle quantities in ( 4 ) . The value of τ̂ in ( 5 ) will be determined by suitable conformity scores evaluated on the hold-out data , and it may be larger than 1 − α if the model for f is not very accurate . However , if the fitted histogram is close to the true PY |X , the interval in ( 5 ) will resemble that of the oracle ( 3 ) . Figure 1 previews an application to toy data , comparing CHR to conformalized quantile regression ( CQR ) [ 30 ] ; see Section 4.2 for more details . CHR finds the shortest interval such the corresponding area under the histogram is above τ , for any τ ∈ ( 0 , 1 ] , and then calibrates τ to guarantee marginal coverage above 1−α ; this extracts more information from the model compared to CQR . For example , CHR adapts automatically to the skewness of Y | X , returning intervals delimited by the 0 % –90 % quantiles in this example , which are shorter than the symmetric ones ( 5 % –95 % ) sought by CQR . 1.3 Related work . This work is inspired by the conformity scores introduced by [ 31 ] for multi-class classification , the underlying idea of which can be repurposed here . Nonetheless , the extension to our problem involves several innovations . This paper connects [ 31 ] to other conformal methods for continuous responses [ 24 , 25 , 35 ] , which sought objectives similar to ours by leveraging quantile regression [ 17 , 23 , 30 , 32 , 36 ] or non-parametric density estimation [ 10 , 21 ] , sometimes considering multi-modal prediction sets instead of intervals [ 20 ] . Our approach also exploits black-box models for distributional estimation ; however , we introduce more efficient conformity scores . We seek the shortest intervals with marginal coverage while approximating as well as possible conditional coverage , although the latter is impossible to guarantee in finite samples [ 16 , 34 ] . The performances of prior approaches have been measured in terms of these criteria , yet others have not sought them as directly . Indeed , if the black-box model is consistent for PY |X , our method becomes asymptotically equivalent to the oracle ( 3 ) – ( 4 ) , under some technical assumptions . This property does not hold for other existing methods because they tend to produce symmetric intervals , with fixed lower and upper miscoverage rates ( the probabilities of the outcome being either below or above the output interval , respectively ) , which may be sub-optimal if the data have unknown skewness . 2 Methods . The proposed method consists of four main components : the estimation and binning of a conditional model for the outcome , the construction of a nested sequence of approximate oracle intervals based on the above , the computation of suitable conformity scores , and their conformal calibration . 2.1 Estimating conditional histograms . We partition the domain of Y into m bins [ bj−1 , bj ) , for some sequence b0 < . . . < bm . With little loss of generality , assume Y is bounded : −C = b0 < Y < bm = C , for some C > 0 . Then , we solve a discrete version of the problem stated in the introduction : we seek the smallest possible contiguous subset of bins with 1− α predictive coverage . If m is large and the bins are narrow , this problem is not very different from the original one , although it is more amenable to solution . For simplicity , we present our method from a split-conformal perspective [ 24 , 35 ] ; extensions to other hold-out approaches [ 7 , 22 , 35 ] will be intuitive . Let Dtrain , Dcal ⊂ { 1 , . . . , n } denote any partition of the data into training and calibration subsets , respectively . Dtrain is used to train a black-box model for the conditional probabilities that Y is within any of the above bins : ∀j ∈ { 1 , . . . , m } , πj ( x ) : = P [ Y ∈ [ bj−1 , bj ) | X = x ] . ( 6 ) There exist many tools to approximate PY |X and obtain estimates π̂j ( x ) of πj ( x ) , including quantile regression [ 27 , 28 , 33 ] , Bayesian additive regression trees [ 11 ] , or any other non-parametric conditional density estimator [ 14 , 19 , 26 ] . Our method can directly be applied with any of these models , but we found multiple quantile regression to work particularly well [ 17 , 23 , 30 , 32 ] , and therefore we will focus on it in this paper . Referring to Supplementary Section S1.1 for implementation details and information about the computational cost of the learning algorithm ( which is comparable to that required by CQR [ 30 ] ) , we thus take these black-box estimates π̂j ( x ) as fixed henceforth . Note that estimating conditional distributions is more challenging if the number of variables is larger . However , this is a fundamental difficulty of high-dimensional regression , not a particular limitation of the proposed CHR . Although our method utilizes conditional histograms learnt from the data , its performance is not directly measured in terms of how closely these resemble the true PY |X . Instead , as we shall see , CHR only needs to detect the possible skewness of Y | X and estimate reasonably well some lower and upper quantiles of this conditional distribution . Therefore , its estimation task is not much more difficult than that of CQR [ 30 ] , as skewness is relatively easy to detect .

Summary: This paper proposes a new nonconformity-score based on conditional density estimates P(Y|X). The main idea is to approximate the optimal prediction interval based on the conditional density estimate. The proposed non-conformity score is experimentally assessed using the conformalization technique of split conformal. The overall method provably achieves marginal coverage, performs comparably to other methods for conditional coverage, and achieves the shortest prediction intervals on all datasets.