Conditional Coverage Estimation for High-quality Prediction Intervals

Deep learning has achieved state-of-the-art performance to generate high-quality prediction intervals ( PIs ) for uncertainty quantification in regression tasks . The high-quality criterion requires PIs to be as narrow as possible , whilst maintaining a pre-specified level of data ( marginal ) coverage . However , most existing works for high-quality PIs lack accurate information on conditional coverage , which may cause unreliable predictions if it is significantly smaller than the marginal coverage . To address this problem , we propose a novel end-to-end framework which could output high-quality PIs and simultaneously provide their conditional coverage estimation . In doing so , we design a new loss function that is both easyto-implement and theoretically justified via an exponential concentration bound . Our evaluation on real-world benchmark datasets and synthetic examples shows that our approach not only outperforms the state-of-the-arts on high-quality PIs in terms of average PI width , but also accurately estimates conditional coverage information that is useful in assessing model uncertainty . 1 INTRODUCTION . Prediction interval ( PI ) is poised to play an increasingly prominent role in uncertainty quantification for regression tasks ( Khosravi et al. , 2010 ; 2011 ; Galván et al. , 2017 ; Rosenfeld et al. , 2018 ; Tagasovska & Lopez-Paz , 2018 ; 2019 ; Romano et al. , 2019 ; Wang et al. , 2019 ; Kivaranovic et al. , 2020 ) . A high-quality PI should be as narrow as possible , whilst maintaining a pre-specified level of data coverage or marginal coverage ( Pearce et al. , 2018 ) . Compared with PIs obtained based on coverage-only consideration , the “ high-quality ” criterion is beneficial in balancing between marginal coverage probability and interval width . However , the conditional coverage given a feature , which is critical for making reliable context-based decisions , is unassessed and missing in most existing works on high-quality PIs . In the presence of heteroskedasticity and model misspecification , the marginal coverage can be very different from the conditional coverage at a given point , which affects the downstream decision-making task that relies on the uncertainty information provided by the PI . Our main goal is to meaningfully incorporate and assess conditional coverages in high-quality PIs . Conditional coverage estimation is challenging for two reasons . First is that the natural evaluation metric of conditional coverage error , an Lp distance between the estimated and ground-truth conditional coverages , is difficult to compute as it requires obtaining the conditional probability given feature x , which is arguably as challenging as the regression problem itself . Our first goal in this paper is to address this issue by developing a new metric called calibration-based conditional coverage error for conditional coverage estimation measurement . Our approach is inspired from the calibration notion in classification ( Guo et al. , 2017 ) . The basic idea is to relax conditional coverage at any given point to being averaged over all points that bear the same estimated value . An estimator satisfying the relaxed property is regarded as well-calibrated . In regression , calibration-based conditional coverage error provides a middle ground between the enforcement of marginal coverage ( lacking any conditional information ) and conditional coverage ( computationally intractable ) . Compared with conditional coverage , this middle-ground metric can be viewed as a “ dimension reduction ” of the conditioning variable from the original sample space to the space [ 0 , 1 ] , so that we can easily discretize to compute the empirical metric values . The second challenge is the discontinuity in the above metrics that hinders efficient training of PIs that are both high-quality and possess reliable conditional coverage information . To address this , we design a new loss function based on a combination of the high-quality criterion and a coverage assessment loss . The latter can be flexibly added as a separate module to any neural network ( NN ) used to train PIs . It is based on an empirical version of a tight upper bound on the coverage error in terms of a Kullback–Leibler ( KL ) divergence , which can be readily employed for running gradient descent . We theoretically show how training with our proposed loss function attains this upperbounding value via a concentration bound . We also demonstrate the empirical performance of our approach in terms of PI quality and conditional coverage assessment compared with benchmark methods . Summary of Contributions : ( 1 ) We identify the conditional coverage estimation problem as a new challenge for high-quality PIs and introduce a new evaluation metric for coverage estimation . ( 2 ) We propose an end-to-end algorithm that can simultaneously construct high-quality PIs and generate conditional coverage estimates . In addition , we provide theoretical justifications on the effectiveness of our algorithm by developing concentration bounds relating the coverage assessment loss and conditional coverage error . ( 3 ) By evaluating on benchmark datasets and synthetic examples , we empirically demonstrate that our approach not only achieves high performance on conditional coverage estimation , but also outperforms the state-of-the-art algorithms on high-quality PI generation . 2 EVALUATING CONDITIONAL COVERAGE FOR HIGH-QUALITY PIS . Let X ∈ X and Y ∈ Y ⊂ R be random variables denoting the input feature and label , where the pair ( X , Y ) follows an ( unknown ) ground-truth joint distribution π ( X , Y ) . Let π ( Y |X ) be the conditional distribution of Y given X . We are given the training data D : = { ( xi , yi ) , i = 1 , 2 , · · · , n } where ( xi , yi ) are i.i.d . realizations of random variables ( X , Y ) . A PI refers to an interval [ L ( x ) , U ( x ) ] where L , U are two functions mapping from X to Y trained on the data D. [ L ( x ) , U ( x ) ] is called a PI at prediction level 1− α ( 0 ≤ α ≤ 1 ) if its marginal coverage is not less than 1 − α , i.e. , P [ Y ∈ [ L ( X ) , U ( X ) ] |L , U ] ≥ 1 − α where P is with respect to a new test point ( X , Y ) ∼ π . We say that [ L ( x ) , U ( x ) ] is of high-quality if its marginal coverage attains a pre-specified target prediction level and has a short width on average . In particular , a best-quality PI at prediction level 1− α is an optimal solution to the following constrained optimization problem : min L , U E [ U ( X ) − L ( X ) ] subject to P [ Y ∈ [ L ( X ) , U ( X ) ] |L , U ] ≥ 1− α . ( 2.1 ) The high-quality criterion has been widely adopted in previous work ( see Section 6 ) . However , this criterion alone may fail to carry important model uncertainty information at specific test points . Consider a simple example where x ∼ Uniform [ 0 , 1 ] , y = 0 for x ∈ [ 0 , 0.95 ] and y|x ∼ Uniform [ 0 , 1 ] for x ∈ ( 0.95 , 1 ] . Then according to equation 2.1 , a best-quality 95 % PI is precisely L ( x ) = U ( x ) = 0 for all x ∈ [ 0 , 1 ] . This PI has nonconstant coverage if we condition at different points ( 1 for x ∈ [ 0 , 0.95 ] and 0 for x ∈ ( 0.95 , 1 ] ) , and can deviate significantly from the overall coverage 95 % . More examples to highlight the need of obtaining conditional coverage information can be found in our numerical experiments in Section 5.1 . To mitigate the drawback of the high-quality criterion , we define : Definition 2.1 ( Conditional Coverage and Its Estimator ) . The conditional coverage associated with a PI [ L ( x ) , U ( x ) ] is A ( x ) : = P [ Y ∈ [ L ( X ) , U ( X ) ] |L , U , X = x ] for a.e . x ∈ X , where P is taken with respect to π ( Y |X ) . For a ( conditional ) coverage estimator P̂ , which is a measurable function from X to [ 0 , 1 ] , we define its Lp conditional coverage error ( C̃Ep ) as C̃Ep : = ∥∥∥P [ Y ∈ [ L ( X ) , U ( X ) ] |L , U , X ] − P̂ ( X ) ∥∥∥ Lp ( X ) where the Lp-norm is taken with respect to the randomness of X ( 1 ≤ p ≤ +∞ ) . Note that evaluating C̃Ep relies on approximating the conditional coverage A ( x ) , which can be as challenging as the original prediction problem . To address this , we leverage the similarity of estimatingA ( x ) to generating prediction probabilities in binary classification , which motivates us to borrow the notion of calibration in classification . This idea is based on a relaxed error criterion by looking at the conditional coverage among all points that bear the same coverage estimator value , instead of conditioning at any given point . The resulting error metric then only relies on probabilities conditioned on variables in a much lower-dimensional space [ 0 , 1 ] than X . To explain concretely , we introduce a “ perfect-calibrated coverage estimator ” as : Definition 2.2 ( Perfect Calibration ) . A coverage estimator P̂ is called a perfect-calibrated coverage estimator associated with [ L ( x ) , U ( x ) ] if it satisfies P̂ ( x ) = P [ Y ∈ [ L ( X ) , U ( X ) ] |L , U , P̂ ( X ) = P̂ ( x ) ] , a.e . P̂ ( x ) ∈ [ 0 , 1 ] . ( 2.2 ) where a.e . is with respect to the probability measure on [ 0 , 1 ] induced by the random variable P̂ ( X ) . Equation 2.2 means that a point x with conditional coverage estimate P̂ ( x ) = p has an average coverage of precisely p , among all points in X that possess the same conditional coverage estimated value . That is , the average coverage of the PI restricted on the subset { x ∈ X : P̂ ( x ) = p } should be precisely p. Corresponding to Definition 2.2 , we define : Definition 2.3 ( Calibration-based Error ) . An Lp ( 1 ≤ p ≤ +∞ ) calibration-based conditional coverage error , or coverage error for short ( CEp ) , of a coverage estimator P̂ is : CEp : = ∥∥∥P [ Y ∈ [ L ( X ) , U ( X ) ] |L , U , P̂ ( X ) ] − P̂ ( X ) ∥∥∥ Lp ( X ) ( 2.3 ) where Lp-norm is taken with respect to the randomness of P̂ ( X ) . In the above definition the conditional probability P [ Y ∈ [ L ( X ) , U ( X ) ] |L , U , P̂ ( X ) ] is a measurable function of random variable P̂ ( X ) , say γ ( P̂ ( X ) ) . By a change of variable , CEpp : = ∥∥∥γ ( P̂ ( X ) ) − P̂ ( X ) ∥∥∥p Lp ( X ) = ∫ 1 0 |γ ( t ) − t|pdFP̂ ( X ) ( t ) ( 2.4 ) where FP̂ ( X ) is a probability distribution of P̂ ( X ) on [ 0 , 1 ] . Here , CEp only requires estimating γ ( t ) for t ∈ [ 0 , 1 ] , which can be done easily by discretizing [ 0 , 1 ] for empirical calculation . We call the empirical Lp calibration-based conditional coverage error ECEp . More details of this empirical error can be found in Appendix A.4 . A calibration-based error CEp provides a middle ground between the enforcement of marginal coverage and conditional coverage . The ground-truth conditional coverage is perfectly calibrated , but not vice versa . However , if we enforce the perfect calibration criterion for a coverage estimator to hold when restricted to any measurable subset in X , then the choice of estimator will reduce uniquely to the conditional coverage ( Definition A.1 and Lemma A.2 ) . In this sense , CEp is an error metric that is a natural relaxation of C̃Ep , and although less precise , CEp is computationally much more tractable than C̃Ep . Evaluation Metric for Coverage Estimator . We use ECE1 as the primary evaluation metric to measure the quality of a coverage estimator . A high ECE1 value of a coverage estimator indicates an unreliable coverage estimation while a small ECE1 value indicates that the coverage estimator is close to the perfect-calibrated property . Ideally , an effective algorithm should output a coverage estimator with a small ECE1 value .

Summary: In the submitted paper, the authors study high-quality prediction intervals (PIs). The paper proposes a novel design of loss functions to generate PIs and conditional coverage estimates. The theoretical justification for using the conditional coverage error (in Ca-module) is presented and the numerical experiments with promising results are provided on multiple benchmark datasets.