Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification

1 Introduction . Uncertainty quantification ( UQ ) in machine learning typically refers to the task of quantifying the confidence of a given prediction . This measure of certainty can be crucial in a variety of downstream applications , including Bayesian optimization [ 36 , 45 , 54 ] , model-based reinforcement learning [ 64 , 42 , 10 , 18 ] , and in high-stakes predictions where mistakes incur large costs [ 52 , 61 ] . While the common goal of UQ is to describe predictive distributions over outputs for given inputs , the representation of the distributional prediction varies across methods . For example , some methods assume a parametric distribution and return parameter estimates [ 65 , 14 , 37 ] , while others return density function estimates , as is common in Bayesian methods [ 41 , 38 , 24 , 3 , 33 , 50 ] . Alternatively , many methods represent predictive uncertainty with quantile estimates [ 15 , 53 , 58 , 49 , 27 ] . Quantiles provide an attractive representation for uncertainty because they can be used to model complex distributions without parametric assumptions , are interpretable with units in the target output space , allow for easy construction of prediction intervals , and can be used to efficiently sample from the predictive distribution via inverse transform sampling [ 22 , 32 ] . Learning the quantile for a single Code is available at https : //github.com/YoungseogChung/calibrated-quantile-uq . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . quantile level is a well studied problem in quantile regression ( QR ) [ 30 , 32 ] , which typically involves optimizing the so-called pinball loss , a tilted transformation of the absolute value function . Given a target y , a prediction ŷ , and quantile level τ ∈ ( 0 , 1 ) , the pinball loss ρτ is defined as ρτ ( y , ŷ ) = ( ŷ − y ) ( I { y ≤ ŷ } − τ ) . ( 1 ) By training for all quantiles simultaneously , recent works have made concrete steps in incorporating QR methods to form competitive UQ methods which output the full predictive distribution [ 51 , 58 ] . In this work , we highlight some limitations of the pinball loss and propose several methods to address these shortcomings . Specifically , we explore the following : • Model agnostic QR . Optimizing the pinball loss often restricts the choice of model family for which we can provide UQ . We propose an algorithm to learn all quantiles simultaneously by utilizing methods from conditional density estimation . This algorithm is agnostic to model class and can be applied to any regression model . • Explicitly balancing calibration and sharpness . While the pinball loss , as a proper scoring rule , targets both calibration and sharpness , the balance between these two quantities is made implicitly , which may result in a poor optimization objective . We propose a tunable loss function that targets calibration and sharpness separately , and allows the end-user to set an explicit balance . • Centered intervals . In practice , we often desire uncertainty predictions made with centered intervals , which are not targeted via the pinball loss . We propose an alternative loss function that is better suited for this goal . • Encouraging individual calibration . Perfect quantile forecasts will satisfy individual calibration ( Eq . 2 ) , which is a much stricter condition than the more-commonly used notion of average calibration ( Eq . 3 ) . We introduce a training procedure that aims to improve quantile predictions beyond average calibration , and demonstrate its efficacy via adversarial group calibration ( Eq . 4 ) . We proceed by first describing methods of assessing the quality of predictive UQ and the pitfalls of optimizing the pinball loss in Section 2 . Drawing motivation from this , we then present our proposed methods in Section 3 . In Section 4 , we demonstrate our methods experimentally , where we model predictive uncertainty on benchmark datasets , and on a high-dimensional , real-world uncertainty estimation task in the area of nuclear fusion . 2 Preliminaries and Background . We first lay out the notation , terminology , and class of models considered in this paper . Then we provide an overview of evaluation metrics in UQ and demonstrate how the pinball loss may be inadequate both as an evaluation metric and as an optimization objective . 2.1 Notation . Bold upper case letters X , Y denote random variables , lower case letters x , y , denote their values , and calligraphic upper case letters X , Y denote sets of possible values . We use x ∈ X to denote the input feature vector and y ∈ Y to denote the corresponding target . Additionally , we consider the regression setting where Y ⊂ R and X ⊂ Rn . We use FX , FY|x , FY to denote the true cumulative distribution of the subscript random variable . For any x ∈ X , we assume there exists a true conditional distribution FY|x over Y , and we assume Qp ( x ) denotes the true pth quantile of this distribution , i.e . FY|x ( Qp ( x ) ) = p. Any estimates of the true functions F , Qp will be denoted with a hat , F̂ , Q̂p . We will specifically refer to any family of estimates for Qp , with p ∈ ( 0 , 1 ) , as a “ quantile model ” , denoted Q̂ : X × ( 0 , 1 ) → Y . Unless otherwise noted , we will always consider the conditional problem of estimating quantities in the target space Y , conditioned on a value x ∈ X . 2.2 Assessing the Quality of Predictive UQ . While various metrics have been proposed to assess the quality of UQ , there has been a great deal of recent focus on the notions of calibration and sharpness [ 15 , 13 , 65 , 60 , 55 , 35 , 21 , 20 ] . We introduce calibration here , but for a more thorough treatment , see Zhao et al . [ 65 ] . Broadly speaking , calibration in the regression setting requires that the probability of observing the target random variable below a predicted pth quantile is equal to the expected probability p , for all p ∈ ( 0 , 1 ) . We refer to the former quantity as the observed probability and denote it pobs ( p ) , for an expected probability p , which we will write as pobs when it is clear from context . Calibration requires pobs ( p ) = p , ∀p ∈ ( 0 , 1 ) . From this generic statement , we can describe different notions of calibration based on how pobs is defined . A model is individually calibrated if it outputs the true conditional quantiles , i.e . Q̂p ( x ) = Qp ( x ) . In this case , we define the observed probability to be pobsindv ( p , x ) : = FY|x ( Q̂p ( x ) ) , ∀x ∈ X , ∀p ∈ ( 0 , 1 ) . ( 2 ) In words , this requires that the probability of observing y below the quantile prediction is equal to p , at each point x ∈ X , individually . If we can verify this property for all x ∈ X , then by definition , we will know the quantile output is correct and precisely the true conditional quantile . However , individual calibration is typically unverifiable with finite datasets in the assumption-less case [ 65 ] . A relaxed condition is average calibration . In this case , we define the observed probability to be pobsavg ( p ) : = Ex∼FX [ FY|x ( Q̂p ( x ) ) ] , ∀p ∈ ( 0 , 1 ) , ( 3 ) i.e . the probability of observing the target below the quantile prediction , averaged over FX , is equal to p. Average calibration is often referred to simply as “ calibration ” [ 13 , 35 ] . Given a dataset D = { ( xi , yi ) } Ni=1 , we can estimate pobsavg ( p ) with p̂obsavg ( D , p ) = 1N ∑N i=1 I { yi ≤ Q̂p ( xi ) } . Note that if our quantile estimate achieves average calibration then p̂obsavg → p as N → ∞ , ∀p ∈ ( 0 , 1 ) . The degree of error in average calibration is commonly measured by expected calibration error [ 60 , 13 , 21 ] , ECE ( D , Q̂ ) = 1m ∑m j=1 ∣∣p̂obsavg ( D , pj ) − pj∣∣ , where pj ∼ Unif ( 0 , 1 ) . It may be possible to have an uninformative , yet average calibrated model . For example , quantile predictions that match the true marginal quantiles of FY will be average calibrated , but will hardly be useful since they do not depend on the input x . Therefore , the notion of sharpness is also considered , which quantifies the concentration of distributional predictions [ 20 ] . For example , for non-parametric predictions , the width of a centered 95 % prediction interval is often used as a measure of sharpness . There generally exists a tradeoff between average calibration and sharpness [ 20 , 46 ] . Recent works have suggested a notion of calibration stronger than average calibration , called adversarial group calibration [ 65 ] . This stems from the notion of group calibration [ 23 , 29 ] , which prescribes measurable subsets Si ⊂ X s.t . Px∼FX ( x ∈ Si ) > 0 , i = 1 , . . . , k , and requires the predictions to be average calibrated within each subset . Adversarial group calibration then requires average calibration for any subset of X with non-zero measure . Denote XS as a random variable that is conditioned on being in the set S . For adversarial group calibration , the observed probability is pobsadv ( p ) : = Ex∼FXS [ FY|x ( Q̂p ( x ) ) ] , ∀p ∈ ( 0 , 1 ) , ∀S ⊂ X s.t . Px∼FX ( x ∈ S ) > 0 . ( 4 ) With a finite dataset , we can measure a proxy of adversarial group calibration by measuring the average calibration within all subsets of the dataset with sufficiently many points . Intuitively , individual calibration inspects the discrepancy between pobs and p for individual inputs x ∈ X , adversarial group calibration relaxes this by inspecting any subset of X with non-zero measure , and average calibration relaxes this further by considering the full distribution of X . One alternative family of evaluation metrics is proper scoring rules [ 19 ] . Proper scoring rules are summary statistics of overall performance of a distributional prediction and consider both calibration and sharpness jointly [ 20 ] . For example , negative log-likelihood ( NLL ) is a proper scoring rule that is commonly used with density predictions [ 14 , 48 , 37 ] . For quantile predictions , one proper score is the check score , which is identical to the pinball loss . Since proper scoring rules consider both calibration and sharpness together in a single value , they can serve as optimization objectives for UQ . For example , optimizing the pinball loss is the traditional method in quantile regression [ 31 ] , and many recent quantile-based UQ methods focus on optimizing this objective [ 51 , 58 , 8 , 63 ] . In this work , however , we note that the balance between calibration and sharpness implied by the pinball loss is arbitrary and depends on the expressivity of the model class—and with highly expressive models , this balance can be heavily skewed towards sharpness . In their seminal work on probabilistic forecasts , Gneiting and Raftery [ 19 ] contend that the goal of probabilistic forecasting is to “ maximize the sharpness of the predictive distribution subject to calibration ” , i.e . calibration should be first achieved and then sharpness optimized . We show that common machine learning methods that use the pinball loss objective may in fact lead to an arbitrary and miscalibrated UQ . Proposition 1 . Consider a finite dataset D , the pinball loss ρτ ( Eq . 1 ) and a quantile model f : X × ( 0 , 1 ) → Y that is average calibrated on D , i.e . ECE ( D , f ) = 0 . Then there always exists another quantile model g : X × ( 0 , 1 ) → Y , such that , for any quantile level τ ∈ ( 0 , 1 ) , g has lower pinball loss than f on D , i.e . ∑N i=1 ρτ ( yi , gτ ( xi ) ) < ∑N i=1 ρτ ( yi , fτ ( xi ) ) , but worse average calibration than f , i.e . ECE ( D , g ) > ECE ( D , f ) . Proof : The proof is given in Appendix A.1 . This proposition essentially states how the pinball loss can become detached from calibration , and we show its practical ramifications via a synthetic example in Figure 1 ( experiment details in Appendix E.1 ) . We first note in Figure 1 ( a ) and ( b ) that even while the pinball loss decreases on the test set , test calibration worsens ( while sharpness improves ) . Further , at the best validation epoch , optimizing the pinball loss converges to a solution that is sharper than the true noise level . Note that a UQ that is sharper than the true noise level will never be calibrated ( meanwhile , a less sharp prediction can still be calibrated , e.g . the marginal distribution FY ) . While this may seem like an issue that can simply be addressed with regularization , we demonstrate in Appendix E.2 how that is not the case . These pitfalls motivate our methods in Section 3 .

Summary: This work is motivated by some shortcomings of the classical pinball loss used to learn quantile models, like the required differentiability of the models and miscalibration issues. The pinball loss is also not naturally suited to build centered intervals. This work proposes alternatives ways for learning quantile models, addressing these shortcomings. Four methods are proposed: - Model Agnostic QR: it is a generic method based on conditional density estimation and standard regression. Therefore it is not particularly tied to differentiable models and doesn't use the pinball loss. - Combined calibration loss: a loss to be used instead of the pinball one, with the advantage of explicitly balancing  sharpness and calibration of the predictions. - Centered intervals: an alternative loss based on the Winkler score is proposed to train models providing centered intervals. - Group batching: a heuristic replacing uniform sampling for batch-based based methods, aiming at better individual calibration.