Quantile Regularization : Towards Implicit Calibration of Regression Models

1 INTRODUCTION . For supervised machine learning , the notion of calibration of a learned predictive model is a measure of evaluating how well a model ’ s confidence in its prediction matches with the correctness of these predictions . For example , a binary classifier will be considered perfectly calibrated if , among all predictions with probability score 0.9 , 90 % of the predictions are correct Guo et al . ( 2017 ) . Likewise , consider a probabilistic regression model that produces credible interval for the predicted outputs . In this setting , the model will be considered perfectly calibrated if the 90 % confidence interval contains 90 % of the true test outputs ( Kuleshov et al. , 2018 ) . Unfortunately , modern deep neural networks are known to be poorly calibrated ( Guo et al. , 2017 ) , raising questions on their reliability . The notion of calibration for classification problems was originally first considered in meteorology literature ( Brier , 1950 ; Murphy , 1972 ; Gneiting & Raftery , 2007 ) and saw one of its first prominent usage used in the machine learning literature by ( Platt et al. , 1999 ) in context of Support Vector Machines ( SVM ) , in order to obtain probabilistic predictions from SVMs which are inherently non-probabilistic models . Recently , there has been renewed interested in calibration , especially for classification models , after it has been shown ( Guo et al. , 2017 ) that modern deep neural networks for classification are often poorly calibrated . The pupular notions of calibration for classification include confidence calibration , multiclass calibration , classwise calibration , and confidence calibration ( Kumar et al. , 2019 ; Vaicenavicius et al. , 2019 ; Kull et al. , 2019 ) . Most calibration methods ( Platt et al. , 1999 ; Zadrozny & Elkan , 2001 ; 2002 ; Guo et al. , 2017 ; Kull et al. , 2017 ; 2019 ) for classification models are post hoc , where they learn a calibration mapping R : [ 0 , 1 ] → [ 0 , 1 ] using an additional dataset to recalibrate an already trained model . There has been recent work showing some of these popular post hoc methods are either themselves miscalibrated or sample-inefficient ( Kumar et al. , 2019 ) and they do not actually help the model output well-calibrated probabilities . An alternative to post hoc processing is to ensure that model outputs well-calibrated probabilities after model training finishes . We refer to these as implicit calibration methods . Notably , such an approach does not require an additional dataset to learn the calibration mapping . While almost all post hoc calibration methods for classification models can be seen in a unified manner as density estimation methods ( see section 2.1 ) , existing implicit calibration methods for classification models have been designed with various , often distinct , considerations/approaches . Several heuristics like Mixup ( Zhang et al. , 2017 ; Thulasidasan et al. , 2019 ) and Label Smoothing ( Szegedy et al. , 2016 ; Müller et al. , 2019 ) that were part of high performance deep networks for classification were later shown empirically to achieve calibration . ( Maddox et al. , 2019 ) show that their optimization method instrinsically improves calibration . ( Pereyra et al. , 2017 ) found that penalizing high-confidence predictions acts as a regularizer . A more principled way of achieving implicit calibration is by minimizing a loss function that is tailored for calibration ( Kumar et al. , 2018 ) . This is somewhat similar in spirit to our proposed approach which aims to do it for regression models . Among the early approaches for calibrating regression models , ( Gneiting et al. , 2007 ) were the first to propose a framework for calibrating regression models . However , they do not provide any procedure to correct a miscalibrated model . Recently , ( Kuleshov et al. , 2018 ) introduced the notion of Quantile Calibration which intuitively says that the p confidence interval predicted by model should have target variable with probability p. They use a post hoc calibration method based on Isotonic Regression ( Fawcett & Niculescu-Mizil , 2007 ) , which is a well-known calibration technique for classification models . The difference between Isotonic Calibration in classification and Isotonic Calibration in regression is in terms of ( i ) the dataset on which calibration mapping is learnt ; and ( ii ) the function with which learnt calibration mapping is pre-composed . In the former case , it is pre-composed with a probability mass function ( PMF ) and whereas in the latter , it is pre-composed with a conditional density function ( CDF ) . Both these differences have side effects ; in particular ( i ) the nature of recalibration dataset already satisfies monotonicity constraint , so there is a risk of overfitting in case of smaller calibration datsets ; and ( ii ) composing the CDF with a piecewise linear function can make the resultant CDF discontinuous and the corresponding PDF non-differentiable ( see Sec . 3 for detailed discussion for side effects of the Isotonic Calibration approach ) . In another recent work , ( Song et al. , 2019 ) proposed a much stronger notion of calibration called Distributional Calibration which guarantees that among all instances whose predicted probability density function ( PDF ) of the response variable has mean µ and standard deviation σ , the marginal distribution of the target variable should have mean µ and standard deviation σ . They too propose a post hoc recalibration method based on Gaussian processes , which can be computationally expensive . Among other work , ( Keren et al. , 2018 ) , consider a different setting where neural networks for classification are used for regression problems and showed that temperature scaling ( Hinton et al. , 2015 ; Guo et al. , 2017 ) and their proposed method based on empirical prediction intervals improves calibration for regression problems as well . Again , these are post hoc methods . Our contributions are summarized below : 1 . We analyze in detail the side effects of Isotonic Calibration for regression models . We show how using Isotonic Calibration results in truncation of the support , which will result in assigning zero likelihood fortes t time . We also discuss about Isotonic Calibration resulting in nonsmooth PDFs , and its tendency to produce miscalibration when using small calibration datasets . 2 . At test time , after composing the predicted CDF with the learned isotonic mapping , the mean prediction ( point estimate ) also changes . Kuleshov et al . ( 2018 ) do not acknowledge the changes in the mean estimate . While Song et al . ( 2019 ) acknowledge this issue , they use a trapezoidal approximation to remedy this . In contrast , we derive an analytical expression for the updated point estimate after isotonic calibration . We also provide a different expression for updated point estimate , which reduces the time-complexity from O ( m ) to O ( 1 ) , where m is calibration dataset size . 3 . In order to mitigate these shortcomings of Isotonic Calibration , we propose a simple , yet novel and general purpose , trainable loss function for quantile calibration where the smoothness of PDF/CDF is not sacrificed for well-calibrated probabilties . Our approach also eliminates the need for an additional calibration dataset . 4 . We conduct extensive experiments on a wide range of architectures using the proposed loss function ( Quantile Regularization ) and show empirically that it improves calibration on wide range of architectures that produce uncertainty estimates . 2 BACKGROUND AND DEFINITIONS . Before we proceed with definitions , we state the notation followed in the rest of paper . X , Y denotes the input and output space respectively . X , Y denote random variables modelling inputs and outputs . P denotes probability measure f , g are reserved for probability density functions ( PDF ) and F , G for cummulative density functions ( CDF ) . We use c to denote the evaluation of a CDF at some particular value i.e. , c = F ( . ) and p for evaluation of a PDF at some value i.e. , p = f ( . ) Given sequence of elements a1 , a2 , . . . , an , we use a ( 1 ) , a ( 2 ) , . . . a ( n ) for permutation s.t a ( i ) ≤ a ( i+1 ) . Also , m denotes the calibration dataset size . ( X , y ) denotes the training data . Given random variables X , Y , we use KL ( X||Y ) to denote the KL divergence between the corresponding distributions A probabilistic regression model can be seen as conditional PDF/conditional CDF . In the rest of the paper , we express it as conditional CDF M : X → ( Y → [ 0 , 1 ] ) . So , M ( x ) denotes model ’ s predicted CDF for x ∈ X denoted as Fx . In practice this is achieved by making model output parameters that parametrize the CDF , e.g. , ( µ , σ ) for Gaussian , λ for Exponential , etc . In the rest of the paper , we consider Gaussian likelihood unless stated otherwise , because it is one of the most prevalent cases . Kuleshov et al . ( 2018 ) proposed the following notion of calibration of regression models , called quantile calibration . An appearling aspect of this definition is that we get reliable confidence intervals . Definition 1 ( Quantile Calibration ) Given a regression model M : X → ( Y → [ 0 , 1 ] ) and X , Y jointly distributed as P , the model M is said to be Quantile Calibrated iff P [ [ M ( X ) ] ( Y ) ≤ p ] = p ∀p ∈ [ 0 , 1 ] ( 1 ) In words , [ M ( X ) ] ( Y ) is cumulative density that the model predicts for random input-response pairs drawn from the joint distribution of ( X , Y ) . It is important to note that , regardless of dimensions of X , Y and support of distributions of X , Y and the likelihood of model considered , [ M ( X ) ] ( Y ) is a random variable whose support is a subset of [ 0 , 1 ] because range of any real-valued CDF is [ 0 , 1 ] . So Def . 1 is quite general and covers the case when output is vector-valued , i.e. , Y = Rn , n > 1 2.1 FUNDAMENTAL THEOREM OF POST HOC CALIBRATION . The objective of post hoc calibration is to recalibrate a miscalibrated model M by learning a mapping R : [ 0 , 1 ] → [ 0 , 1 ] s.t . R ◦M is a calibrated model . One such mapping can be obtained from the definition of calibration itself . Setting R ( p ) = P [ [ M ( X ) ] ( Y ) ≤ p ] makes R ◦M a quantile calibrated model . ( Vaicenavicius et al. , 2019 ) call an analogous mapping in the context of classification as canonical calibration mapping . Theorem 1 For any Model M : X → ( Y → [ 0 , 1 ] ) and given the canonical calibration mapping R ( p ) = P [ [ M ( X ) ] ( Y ) ≤ p ] , R ◦M is quantile calibrated Note that learning the mapping R reduces to density estimation of P [ [ M ( X ) ] ( Y ) ≤ p ] , which is a hard problem in itself . With this insight , and using the fact that the mapping is monotonically increasing , ( Kuleshov et al. , 2018 ) use Isotonic Regression to learn the mapping on a separate calibration dataset . Similar theorems hold for other notions of calibration , too .

Summary: The paper discusses a new calibration mechanism for regression models which produce better model prediction and uncertainty estimates. In Section 3, the paper first discusses some properties and drawbacks of the approach based on isotonic regression in Kuleshov et al., 2018 which uses a post-hoc calibration dataset after model fitting for calibration. Section 4 discusses a new approach based on regularization to achieve implicit regression calibration during model training instead of using a post-hoc processing approach. Section 5 is devoted to experimental results supporting the theory outlined in Sections 3 and 4.