Normalizing Flows for Calibration and Recalibration

In machine learning , due to model misspecification and overfitting , estimates of the aleatoric uncertainty are often inaccurate . One approach to fix this is isotonic regression , in which a monotonic function is fit on a validation set to map the model ’ s CDF to an optimally calibrated CDF . However , this makes it infeasible to compute additional statistics of interest on the model distribution ( such as the mean ) . In this paper , through a reframing of recalibration as MLE , we replace isotonic regression with normalizing flows . This allows us to retain the ability to compute the statistical properties of the model ( such as closed-form likelihoods , mean , correlation , etc . ) and provides an opportunity for additional capacity at the cost of possible overfitting . Most importantly , the fundamental properties of normalizing flows allow us to generalize recalibration to conditional and multivariate distributions . To aid in detecting miscalibration and measuring our success at fixing it , we use a simple extension of the calibration Q-Q plot . 1 INTRODUCTION . Recent advances in deep learning have led to models with significantly higher overall accuracy on both classification and regression tasks compared to what was achievable in the past . However , an important component in conjunction with accuracy is a model ’ s ability to accurately assess the uncertainty in its prediction . Most taxonomies classify uncertainty into three sources : approximation , aleatoric , and epistemic uncertainty ( Der Kiureghian & Ditlevsen , 2009 ) . Approximation uncertainty quantifies the error from fitting a simple model to complex data . Aleatoric uncertainty quantifies the uncertainty of the conditional distribution of the target variable given features . This uncertainty arises from hidden variables or measurement errors and can not be reduced through collecting more data under the same experimental conditions . Epistemic uncertainty quantifies the uncertainty arising from fitting a model utilizing finite data , i.e . it is inversely proportional to the density of the training examples and can be reduced by collecting data in the low density regions . These different sources of uncertainty have different techniques for handling them . Using high capacity models such as neural networks removes a large part of the approximation uncertainty . By fitting a full distribution on the target conditional on features , we can model the aleatoric uncertainty from observations . Inaccurate estimates of aleatoric uncertainty can be explained by underfitting ( insufficient complexity in the conditional distributions ) or overfitting ( models with sufficient capacity can memorize the data , leading to the distributions collapsing to deltas ) . Though epistemic uncertainty is important for the model to answer what it does not know , the focus of this paper is on improving estimates of the aleatoric uncertainty . Our approach in this paper is to handle both model fit and calibration using normalizing flows . Normalizing flows can be used in conjunction with amortized inference to improve the flexibility of the output distribution , and further , through a reframing of recalibration as maximum likelihood estimation ( MLE ) , normalizing flows can be used to handle any miscalibration found on a validation set . Further , we use a simple extension of the calibration plot from Kuleshov et al . ( 2018 ) to help with the the analysis of the calibration of a model across different regions of the data . 2 RELATED WORK . One method for handling aleatoric uncertainty is amortized inference with Gaussians ( Lakshminarayanan et al. , 2017 ; Nix & Weigend , 1994 ; Kendall & Gal , 2017 ) where a model , such as a neural network , maps from features to the parameters of a Gaussian . This approach models aleatoric uncertainty directly but suffers from approximation uncertainty as a Gaussian can not model complex targets . Another approach is Bayesian methods such as Bayesian Ridge Regression ( Tipping , 2001 ) and MC Dropout ( Gal & Ghahramani , 2016 ) . Similar to amortized inference with Gaussians , the output distribution limits the capacity of the model . Full Bayesian techniques with neural networks are often too computationally expensive in practice , and approximate methods often fail to capture the full complexity of the uncertainty ( Lakshminarayanan et al. , 2017 ) . Another family of methods uses quantile regression with non-linear techniques such as decision trees or neural networks . Some of these methods require a predefined set of quantiles ( Takeuchi et al. , 2006 ; Wen et al. , 2017 ; Rodrigues & Pereira , 2018 ; Taylor , 2000 ) . Simultaneous Quantile Regression ( SQR , Tagasovska & Lopez-Paz ( 2019 ) ) trains one model on all quantiles and is able to learn complex shaped distributions . However the training procedure requires the model to learn to be monotonic instead of being constrained to be so and is not trivial to extend to multidimensional outputs . Pearce et al . ( 2018 ) learns a finite set of quantiles by using quality metrics for predictive intervals . Normalizing flows have been used in the contexts of variational inference and generative modeling . The approaches to normalizing flows can be categorized into autoregressive methods ( Kingma et al. , 2016 ; Papamakarios et al. , 2017 ; Huang et al. , 2018 ; Cao et al. , 2019 ) , coupling layers ( Dinh et al. , 2014 ; 2016 ; Kingma & Dhariwal , 2018 ; Ho et al. , 2019 ) , residual networks ( Rezende & Mohamed , 2015 ; van den Berg et al. , 2018 ; Gopal , 2020 ) and continuous flows ( Grathwohl et al. , 2018 ) . 3 BACKGROUND . 3.1 RECALIBRATION . An important goal in modeling is to have well-calibrated distributions as this allows users of the model to understand the confidence the model places on its prediction . In other words , with a well-calibrated model , we can better ascertain the uncertainty in the model ’ s predictions and respond differently to those predictions depending on the uncertainty surrounding them . Guo et al . ( 2017 ) showed that unlike techniques used decades ago such as Bayesian Ridge Regression , modern neural network-based classifiers are very poorly calibrated . A simple variant of Platt scaling and other histogram based techniques applied to a validation set were shown to help alleviate the calibration problem where perfect calibration is defined as P ( Ŷ = Y |P̂ = p ) = p , ∀p ∈ [ 0 , 1 ] where Ŷ is a class prediction and P̂ is its predicted probability of correctness . Kuleshov et al . ( 2018 ) extended the analysis in Guo et al . ( 2017 ) to neural network-based regressors ; isotonic regression , a method for learning monotonic univariate functions , was applied to map from Fxj ( yj ) = p̂j to | { yn|Fxn ( yn ) < p̂j , n = 1 , . . . , N } |/N ( the fraction of the data where the model CDF is less than p̂ ) to improve calibration where perfect calibration is defined as P ( Y < F−1X ( p ) ) = p , ∀p ∈ [ 0 , 1 ] where FX is the predicted CDF function . Kuleshov et al . ( 2018 ) further introduced calibration error as a metric to quantitatively measure how well the quantiles are aligned : p̂j = | { yn|Fxn ( yn ) < pj , n = 1 , . . . , N } | / N cal ( y1 , . . . , yN ) = M∑ j=1 ( pj − p̂j ) 2 ( 1 ) where M is the number of quantiles that are evaluated . In this paper , we set this to 100 evenly spaced quantiles . 3.2 NORMALIZING FLOWS . A crucial component in improving uncertainty estimates in this paper is to fit normalizing flows , a generative model for density estimation using invertible functions . Suppose that we wish to formulate a joint distribution on an n-dimensional real vector x . A flowbased approach treats x as the result of a transformation g applied to an underlying vector z sampled from a base distribution pz ( z ) . The generative process for flows is defined as : z ∼ pz ( z ) x = g ( z ) where pz is often a Normal distribution and g is an invertible function . Notationally , we will use f = g−1 . Using change of variables , the log likelihood of x is log px ( x ) = log pz ( f ( x ) ) + log ∣∣∣∣det ( ∂f ( x ) ∂x ) ∣∣∣∣ To train flows ( i.e . maximize the log likelihood of data points ) , we need to be able to compute the logarithm of the absolute value of the determinant of the Jacobian of f , also called the log-determinant . To construct large normalizing flows , we can compose smaller ones as this is still invertible and the log-determinant of this composition is the sum of the individual log-determinants . An important observation is that every univariate distribution can be viewed as a flow in which the base distribution pz ( z ) is a Uniform distribution over [ 0 , 1 ] and g is the inverse CDF of the univariate distribution . This can be generalized to multivariate distributions using the chain rule . 3.3 CONDITIONAL QUAR FLOWS . In this paper , the specific normalizing flow used is Quasi-Autoregressive Residual Flows ( QuAR Flows , Gopal ( 2020 ) ) . QuAR Flows were shown to handle modeling complex distributions while having nice optimization properties and computationally efficient log-likelihoods . Other commonly used normalizing flows have drawbacks . Coupling layers do not work for one-dimensional inputs . Autoregressive flows are often conditional Gaussians which means in the one-dimensional case , the flow is simply a Gaussian . Residual Flows ( Chen et al. , 2019 ) are expressive but , though equivalent to QuAR Flows in the one-dimensional case , are computationally expensive for high-dimensional distributions . If we would like to condition a QuAR Flow given some features c , we could use a hypernetwork that maps c to the parameters of the QuAR Flow ( Figure 1a ) . However , this can be computationally expensive if the QuAR Flow is parameterized with thousands of parameters . Instead , conditional information can be incorporated by concatenating the conditions c before applying the first layer in the residual connection within a QuAR Flow ( Figure 1b ) . 4 REGRESSION USING NORMALIZING FLOWS . The goal of regression is to model the conditional probability distribution p ( y|x ) . Frequently , this task is reduced to obtaining point estimates of the mean or median by minimizing the mean squared error or mean absolute error respectively . In order to access uncertainties , we must model the full conditional distribution ; in addition to this , we can still retrieve point estimates of the mean , median , or any other distributional statistic . In this section , we discuss how normalizing flows can be used to this end as well as how recalibration can be reframed as maximum likelihood estimation ( MLE ) . 4.1 CALIBRATION WITH NORMALIZING FLOWS . One approach to modeling aleatoric uncertainty is to use amortized inference with a Normal distribution as its output distribution , conditioned on features ( Lakshminarayanan et al. , 2017 ) . However , if a Gaussian is not appropriate to fit the target , then the error in miscalibration can be attributed to approximation error , i.e . insufficient capacity to learn the distribution . A simple example of this is where we have no features to condition on ( or all the features we have contain no relevant information ) with a one-dimensional target . In Figure 2 , we can see that since the target is bimodal , the Gaussian fit does not well describe the data . To improve this model , we replace the Gaussian with a normalizing flow conditioned on our data , similar to Figure 1b . In this way , the distribution fit to the targets has sufficient capacity to learn complex distributions , including multimodal distributions ( Figure 2 ) .

Summary: The authors propose an approach to calibrate conditional distribution estimation models. The approach uses normalizing flows to transform an existing model's predictions into a prediction that better matches the empirical quantiles to the theoretical quantiles. After this remapping procedure, the authors introduce a new plot to visualize calibration. Empirical benchmarks are run on a suite of UCI datasets.