Normalizing Flows for Calibration and Recalibration

In machine learning , due to model misspecification and overfitting , estimates of the aleatoric uncertainty are often inaccurate . One approach to fix this is isotonic regression , in which a monotonic function is fit on a validation set to map the model ’ s CDF to an optimally calibrated CDF . However , this makes it infeasible to compute additional statistics of interest on the model distribution ( such as the mean ) . In this paper , through a reframing of recalibration as MLE , we replace isotonic regression with normalizing flows . This allows us to retain the ability to compute the statistical properties of the model ( such as closed-form likelihoods , mean , correlation , etc . ) and provides an opportunity for additional capacity at the cost of possible overfitting . Most importantly , the fundamental properties of normalizing flows allow us to generalize recalibration to conditional and multivariate distributions . To aid in detecting miscalibration and measuring our success at fixing it , we use a simple extension of the calibration Q-Q plot . 1 INTRODUCTION . Recent advances in deep learning have led to models with significantly higher overall accuracy on both classification and regression tasks compared to what was achievable in the past . However , an important component in conjunction with accuracy is a model ’ s ability to accurately assess the uncertainty in its prediction . Most taxonomies classify uncertainty into three sources : approximation , aleatoric , and epistemic uncertainty ( Der Kiureghian & Ditlevsen , 2009 ) . Approximation uncertainty quantifies the error from fitting a simple model to complex data . Aleatoric uncertainty quantifies the uncertainty of the conditional distribution of the target variable given features . This uncertainty arises from hidden variables or measurement errors and can not be reduced through collecting more data under the same experimental conditions . Epistemic uncertainty quantifies the uncertainty arising from fitting a model utilizing finite data , i.e . it is inversely proportional to the density of the training examples and can be reduced by collecting data in the low density regions . These different sources of uncertainty have different techniques for handling them . Using high capacity models such as neural networks removes a large part of the approximation uncertainty . By fitting a full distribution on the target conditional on features , we can model the aleatoric uncertainty from observations . Inaccurate estimates of aleatoric uncertainty can be explained by underfitting ( insufficient complexity in the conditional distributions ) or overfitting ( models with sufficient capacity can memorize the data , leading to the distributions collapsing to deltas ) . Though epistemic uncertainty is important for the model to answer what it does not know , the focus of this paper is on improving estimates of the aleatoric uncertainty . Our approach in this paper is to handle both model fit and calibration using normalizing flows . Normalizing flows can be used in conjunction with amortized inference to improve the flexibility of the output distribution , and further , through a reframing of recalibration as maximum likelihood estimation ( MLE ) , normalizing flows can be used to handle any miscalibration found on a validation set . Further , we use a simple extension of the calibration plot from Kuleshov et al . ( 2018 ) to help with the the analysis of the calibration of a model across different regions of the data . 2 RELATED WORK . One method for handling aleatoric uncertainty is amortized inference with Gaussians ( Lakshminarayanan et al. , 2017 ; Nix & Weigend , 1994 ; Kendall & Gal , 2017 ) where a model , such as a neural network , maps from features to the parameters of a Gaussian . This approach models aleatoric uncertainty directly but suffers from approximation uncertainty as a Gaussian can not model complex targets . Another approach is Bayesian methods such as Bayesian Ridge Regression ( Tipping , 2001 ) and MC Dropout ( Gal & Ghahramani , 2016 ) . Similar to amortized inference with Gaussians , the output distribution limits the capacity of the model . Full Bayesian techniques with neural networks are often too computationally expensive in practice , and approximate methods often fail to capture the full complexity of the uncertainty ( Lakshminarayanan et al. , 2017 ) . Another family of methods uses quantile regression with non-linear techniques such as decision trees or neural networks . Some of these methods require a predefined set of quantiles ( Takeuchi et al. , 2006 ; Wen et al. , 2017 ; Rodrigues & Pereira , 2018 ; Taylor , 2000 ) . Simultaneous Quantile Regression ( SQR , Tagasovska & Lopez-Paz ( 2019 ) ) trains one model on all quantiles and is able to learn complex shaped distributions . However the training procedure requires the model to learn to be monotonic instead of being constrained to be so and is not trivial to extend to multidimensional outputs . Pearce et al . ( 2018 ) learns a finite set of quantiles by using quality metrics for predictive intervals . Normalizing flows have been used in the contexts of variational inference and generative modeling . The approaches to normalizing flows can be categorized into autoregressive methods ( Kingma et al. , 2016 ; Papamakarios et al. , 2017 ; Huang et al. , 2018 ; Cao et al. , 2019 ) , coupling layers ( Dinh et al. , 2014 ; 2016 ; Kingma & Dhariwal , 2018 ; Ho et al. , 2019 ) , residual networks ( Rezende & Mohamed , 2015 ; van den Berg et al. , 2018 ; Gopal , 2020 ) and continuous flows ( Grathwohl et al. , 2018 ) . 3 BACKGROUND . 3.1 RECALIBRATION . An important goal in modeling is to have well-calibrated distributions as this allows users of the model to understand the confidence the model places on its prediction . In other words , with a well-calibrated model , we can better ascertain the uncertainty in the model ’ s predictions and respond differently to those predictions depending on the uncertainty surrounding them . Guo et al . ( 2017 ) showed that unlike techniques used decades ago such as Bayesian Ridge Regression , modern neural network-based classifiers are very poorly calibrated . A simple variant of Platt scaling and other histogram based techniques applied to a validation set were shown to help alleviate the calibration problem where perfect calibration is defined as P ( Ŷ = Y |P̂ = p ) = p , ∀p ∈ [ 0 , 1 ] where Ŷ is a class prediction and P̂ is its predicted probability of correctness . Kuleshov et al . ( 2018 ) extended the analysis in Guo et al . ( 2017 ) to neural network-based regressors ; isotonic regression , a method for learning monotonic univariate functions , was applied to map from Fxj ( yj ) = p̂j to | { yn|Fxn ( yn ) < p̂j , n = 1 , . . . , N } |/N ( the fraction of the data where the model CDF is less than p̂ ) to improve calibration where perfect calibration is defined as P ( Y < F−1X ( p ) ) = p , ∀p ∈ [ 0 , 1 ] where FX is the predicted CDF function . Kuleshov et al . ( 2018 ) further introduced calibration error as a metric to quantitatively measure how well the quantiles are aligned : p̂j = | { yn|Fxn ( yn ) < pj , n = 1 , . . . , N } | / N cal ( y1 , . . . , yN ) = M∑ j=1 ( pj − p̂j ) 2 ( 1 ) where M is the number of quantiles that are evaluated . In this paper , we set this to 100 evenly spaced quantiles . 3.2 NORMALIZING FLOWS . A crucial component in improving uncertainty estimates in this paper is to fit normalizing flows , a generative model for density estimation using invertible functions . Suppose that we wish to formulate a joint distribution on an n-dimensional real vector x . A flowbased approach treats x as the result of a transformation g applied to an underlying vector z sampled from a base distribution pz ( z ) . The generative process for flows is defined as : z ∼ pz ( z ) x = g ( z ) where pz is often a Normal distribution and g is an invertible function . Notationally , we will use f = g−1 . Using change of variables , the log likelihood of x is log px ( x ) = log pz ( f ( x ) ) + log ∣∣∣∣det ( ∂f ( x ) ∂x ) ∣∣∣∣ To train flows ( i.e . maximize the log likelihood of data points ) , we need to be able to compute the logarithm of the absolute value of the determinant of the Jacobian of f , also called the log-determinant . To construct large normalizing flows , we can compose smaller ones as this is still invertible and the log-determinant of this composition is the sum of the individual log-determinants . An important observation is that every univariate distribution can be viewed as a flow in which the base distribution pz ( z ) is a Uniform distribution over [ 0 , 1 ] and g is the inverse CDF of the univariate distribution . This can be generalized to multivariate distributions using the chain rule . 3.3 CONDITIONAL QUAR FLOWS . In this paper , the specific normalizing flow used is Quasi-Autoregressive Residual Flows ( QuAR Flows , Gopal ( 2020 ) ) . QuAR Flows were shown to handle modeling complex distributions while having nice optimization properties and computationally efficient log-likelihoods . Other commonly used normalizing flows have drawbacks . Coupling layers do not work for one-dimensional inputs . Autoregressive flows are often conditional Gaussians which means in the one-dimensional case , the flow is simply a Gaussian . Residual Flows ( Chen et al. , 2019 ) are expressive but , though equivalent to QuAR Flows in the one-dimensional case , are computationally expensive for high-dimensional distributions . If we would like to condition a QuAR Flow given some features c , we could use a hypernetwork that maps c to the parameters of the QuAR Flow ( Figure 1a ) . However , this can be computationally expensive if the QuAR Flow is parameterized with thousands of parameters . Instead , conditional information can be incorporated by concatenating the conditions c before applying the first layer in the residual connection within a QuAR Flow ( Figure 1b ) . 4 REGRESSION USING NORMALIZING FLOWS . The goal of regression is to model the conditional probability distribution p ( y|x ) . Frequently , this task is reduced to obtaining point estimates of the mean or median by minimizing the mean squared error or mean absolute error respectively . In order to access uncertainties , we must model the full conditional distribution ; in addition to this , we can still retrieve point estimates of the mean , median , or any other distributional statistic . In this section , we discuss how normalizing flows can be used to this end as well as how recalibration can be reframed as maximum likelihood estimation ( MLE ) . 4.1 CALIBRATION WITH NORMALIZING FLOWS . One approach to modeling aleatoric uncertainty is to use amortized inference with a Normal distribution as its output distribution , conditioned on features ( Lakshminarayanan et al. , 2017 ) . However , if a Gaussian is not appropriate to fit the target , then the error in miscalibration can be attributed to approximation error , i.e . insufficient capacity to learn the distribution . A simple example of this is where we have no features to condition on ( or all the features we have contain no relevant information ) with a one-dimensional target . In Figure 2 , we can see that since the target is bimodal , the Gaussian fit does not well describe the data . To improve this model , we replace the Gaussian with a normalizing flow conditioned on our data , similar to Figure 1b . In this way , the distribution fit to the targets has sufficient capacity to learn complex distributions , including multimodal distributions ( Figure 2 ) .

Summary: The paper proposes to use normalizing flows to improve estimates of aleatoric uncertainty in regression tasks. First, the paper suggests that since normalizing flows can improve the flexibility of output distribution, they can be used to mitigate issues of underfitting. Second, the paper proposes an approach that uses normalizing flows for recalibration, which allows people to still query the model’s statistical properties. The authors also introduce a plot that is useful for diagnosing calibration issues.