Normalizing Flows for Calibration and Recalibration
Authors: Achintya Gopal
Aaron Key
Venue:  (2021)
Keywords: recalibration
normalizing flows
uncertainty

Abstract: In machine learning, due to model misspecification and overfitting, estimates of the aleatoric uncertainty are often inaccurate.
One approach to fix this is isotonic regression, in which a monotonic function is fit on a validation set to map the model's CDF to an optimally calibrated CDF. However, this makes it infeasible to compute additional statistics of interest on the model distribution (such as the mean). In this paper, through a reframing of recalibration as MLE, we replace isotonic regression with normalizing flows. This allows us to retain the ability to compute the statistical properties of the model (such as closed-form likelihoods, mean, correlation, etc.) and provides an opportunity for additional capacity at the cost of possible overfitting.  Most importantly, the fundamental properties of normalizing flows allow us to generalize recalibration to conditional and multivariate distributions. To aid in detecting miscalibration and measuring our success at fixing it, we use a simple extension of the calibration Q-Q plot.

Under review as a conference paper at ICLR 2021
NORMALIZING FLOWS FOR CALIBRATION AND RE- CALIBRATION
Anonymous authors Paper under double-blind review
ABSTRACT
In machine learning, due model misspecification and overfitting, estimates the aleatoric uncertainty are often inaccurate. One approach to fix this is isotonic regression. in which monotonic function is fit on validation set map the model's CDF to an optimally calibrated CDF. However, this makes it infeasible to compute additional statistics of interest the model distribution (such as the mean). In this paper, through a reframing of recalibration as MLE, we replace isotonic regression with normalizing flows. This allows us to retain the ability to compute the statistical properties of the model (such as closed-form likelihoods, mean, correlation, etc.) and provides an opportunity for additional capacity at the cost of possible overfitting. Most importantly, the fundamental properties of normalizing flows allow us to generalize recalibration to conditional and multivariate distributions. To aid in detecting miscalibration and measuring our success at fixing it, we use a simple extension of the calibration Q-Q plot.
INTRODUCTION
Recent advances in deep learning have led to models with significantly higher overall accuracy on both classification and regression tasks compared to what was achievable in the past. However, an important component conjunction with accuracy is a model's 's ability to accurately assess the uncertainty in its prediction. Most taxonomies classify uncertainty into three sources: approximation, aleatoric, and epistemic uncertainty (Der Kiureghian & Ditlevsen 2009). Approximation uncertainty quantifies the еrror from fitting simple model to complex data. Aleatoric uncertainty quantifies the uncertainty of the conditional distribution of the target variable given features. This uncertainty arises from hidden variables or measurement errors and cannot be reduced through collecting more data under the same experimental conditions. Epistemic uncertainty quantifies the uncertainty arising from fitting model utilizing finite data, i.e. it is inversely proportional the density of the training examples and can be reduced by collecting data in the low density regions.
These different sources of uncertainty have different techniques for handling them. Using high capacity models such as neural networks removes a large part of the approximation uncertainty. By fitting full distribution on the target conditional on features, we can model the aleatoric uncertainty from observations. Inaccurate estimates of aleatoric uncertainty can be explained by underfitting (insufficient complexity in the conditional distributions) or overfitting (models with sufficient capacity can memorize the data, leading to the distributions collapsing to deltas). Though epistemic uncertainty important for the model answer what does not know, the focus of this paper is on improving estimates of the aleatoric uncertainty.
Our approach in this paper is to handle both model fit and calibration using normalizing flows. Normalizing flows can be used in conjunction with amortized inference to improve the flexibility of the output distribution, and further, through a reframing of recalibration as maximum likelihood estimation (MLE), normalizing flows can be used to handle any miscalibration found on validation set. Further, we use a simple extension of the calibration plot from Kuleshov et al. (2018) to help with the the analysis of the calibration of model across different regions of the data.

---

Under review as a conference paper at ICLR 2021
2 RELATED WORK
One method for handling aleatoric uncertainty amortized inference with Gaussians [Lkkshii narayanan et al. 2017; Nix & Weigend, 1994; Kendall & Gal, 2017) where model, such neural network, maps from features to the parameters of Gaussian. This approach models alcatoric uncertainty directly but suffers from approximation uncertainty Gaussian cannot model complex targets.
Another approach is Bayesian methods such as Bayesian Ridge Regression (Tipping 2001) and MC Dropout (Gal & Ghahramani 2016). Similar to amortized inference with Gaussians, the output distribution limits the capacity of the model. Full Bayesian techniques with neural networks are often too computationally expensive in practice, and approximate methods often fail to capture the full complexity of the uncertainty [Lassmmnnrrayaaan al. 2017).
Another family of methods uses quantile regression with non-linear techniques such as decision trees neural networks. Some of these methods require a predefined of quantiles (Takeuchi al.) 2006 Wen et al. [017 Rodrigues & Pereira 2018:: Taylor] (2000). Simultaneous Quantile Regression (SQR, Tagasovska Lopez-Pate (2019)) trains one model on all quantiles and is able learn complex shaped distributions. However the training procedure requires the model to learn to be monotonic instead of being constrained be so and not trivial to extend multidimensional outputs. Pearce al. 2018 learns a finite set of quantiles by using quality metrics for predictive intervals.
Normalizing flows have been used in the contexts of variational inference and generative modeling. The approaches to normalizing flows can be categorized into autoregressive methods (Kingma et al.
2016
Papamakarios et al.) [0177; Huang et al., 2018 Cao et al. 2019). coupling layers (Dinh et al.
2014
2016; Kingma & Dhariwal 2018 Ho et al. 2019). residual networks (Rezende & Mohamed
2013
van den Berg et al.
2018 Gopa 2020) and continuous flows (Grathwohl et al. 2018).
BACKGROUND
3.1
RECALIBRATION
An important goal in modeling is to have well-calibrated distributions as this allows users of the model understand the confidence the model places on prediction. other words. with a well-calibrated model, we can better ascertain the uncertainty in the model's 's predictions and respond differently to those predictions depending on the uncertainty surrounding them.
Guo et al. (2077) showed that unlike techniques used decades ago such Bayesian Ridge Regression, modern neural network-basee based classifiers are very poorly calibrated. simple variant of Platt scaling and other histogram based techniques applied to a validation set were shown to help alleviate the calibration problem where perfect calibration defined as
P(Ý = Y = p) =
Vp E e
where y a class prediction and P is its predicted probability of correctness.
Kuleshov al. (2018) extended the analysis in Guo al. (2017) neural setorkkbbssed regressors; isotonic regression, a method for learning monotonic univariate functions, was applied to map from Fs, (My) = pú (2 FF,. (yn) < p.j n 1...., , N} / (the fraction of the data where the model CDF is less than p) to improve calibration where perfect calibration is defined as
P(Y (p)) = p.
Vp E [0.
where Fx is the predicted CDF function.
Kuleshov al. (0018) further introduced calibration error metric to quantitatively measure how well the quantiles are aligned:
P. 12n Fa. (yn) < Pj, I,...,,NII NH/N
M Σ(; (P, - 22)
cal y yN)
j=1
where M is the number of quantiles that are evaluated. In this paper, we set this to 100 evenly spaced quantiles.
2

---

Under review as a conference paper at ICLR 2021
X
X
W2
c
NN
c
Wi
W2
(a) Hypernetwork QuAR
(b) Conditional QuAR
Figure 1: A comparison between using hypernetoor approach versus conditional approach to creating Conditional QuAR Flows to model p(X|c) where W1 and W2 parametrize the layers the residual connection.
3.2 NORMALIZING FLOWS
crucial component in improving uncertainty estimates in this paper is to normalizing flows, generative model for density estimation using invertible functions.
Suppose that we wish to formulate a joint distribution on an n--imensional real vector x. A flow- based approach treats the result of transformation applied to underlying vector sampled from a base distribution (z().. The generative process for flows defined as:
2 Pz(2) = ==)
where ps is often a Normal distribution and g is an invertible function. Notationally, we will use g Using change of variables, the log likelihood of is
lo log (f(z)) + log det (0f()) Ox
To train flows (i.e. maximize the log likelihood data points), we need to be able to compute the logarithm of the absolute value of the determinant of the Jacobian of also called the deg-determinant. To construct large normalizing flows, we can compose smaller ones as this is still invertible and the log-determinant of this composition the sum of the individual log-determinants.
An important observation that every univariate distribution can viewed as a flow which the base distribution Pe (2) is Uniform distribution over [0, and g the inverse CDF of the univariate distribution. This can be generalized to multivariate distributions using the chain rule.
3.3 CONDITIONAL QUAR FLOWS
In this paper, the specific normalizing flow used is Quasi-Autoregressive Residual Flows (QuAR Flows, Gopal (2200). QuAR Flows were shown handle modeling complex distributions while having nice optimization properties and computationally efficient loglliiiihhoos. Other commonly used normalizing flows have drawbacks. Coupling layers do work for one-dimensional inputs. Autoregressive flows are often conditional Gaussians which means in the one-dimensional case, the flow is simply a Gaussian. Residual Flows (Chen et al. 2019) are expressive but, though equivalent to QuAR Flows in the one-dimensional case, are computationally expensive for high-dimenniioal distributions.
If would like to condition QuAR Flow given some features could use hypernetwork that maps to the parameters of the QuAR Flow (Figure la). However, this can be computationally expensive if the QuAR Flow is parameterized with thousands of parameters. Instead, conditional information can be incorporated by concatenating the conditions before applying the first layer in the residual connection within QuAR Flow (Figure 1b).

---

Under review as a conference paper at ICLR 2021
4 REGRESSION USING NORMALIZING FLOWS
The goal of regression is to model the conditional probability distribution p(y]z). Frequently, this task reduced obtaining point estimates of the mean or median by minimizing the mean squared enror or mean absolute eтror respectively. In order to access uncertainties, we must model the full conditional distribution; in addition to this, we can still retrieve point estimates of the mean, median, or any other distributional statistic. In this section, we discuss how normalizing flows can be used to this end as well how recalibration can be reframed as maximum likelihood estimation (MLE).
4.1 CALIBRATION WITH NORMALIZING FLOWS
One approach to modeling aleatoric uncertainty is to use amortized inference with Normal distribu- tion as its output distribution, conditioned on features [Lakshminarayanal et al. (017). However, a Gaussian is not appropriate to fit the target, then the error in miscalibration can be attributed to approximation error, i.e. insufficient capacity to learn the distribution. A simple example of this is where we have no features to condition on (or all the features we have contain no relevant information) with one-dimensional target. In Figure 2] we can see that since the target is bimodal, the Gaussian fit does not well describe the data.
To improve this model, we replace the Gaussian with a normalizing flow conditioned on our data, similar to Figure 1b In this way, the distribution fit to the targets has sufficient capacity to Iearn complex distri- butions, including multimodal distributions (Figure 2
0.6
Gaussiaan Ft. on
Flow Olt
0.4
0.3
4.2 RECALIBRATION
WITH NORMALIZING FLOWS
To handle overfitting of the uncertainty, recali- brate the results the original model where we as- sume the original model outputs full distribution that has closed-form likelihoods. Since the only re- quirement is full distribution with a closed-form likelihood, the original model that we recalibrate can be simple as Bayesian Linear Regression model or as complicated as conditional normalizing flow. We recalibrate the original model by fitting normalizing flow on top of it. this way, the training process is the following two-step procedure:
0.1
-2
Figure 2: One-dimensional example of Gaus- sian versus Flow
1. So = arg max la EnT log ff fo (v. (x!")
Ner
2. 90 arg max yo En log Ps, (Fo (yt (o(""()))"))) )
where fe(y/zz) is the predicted pdf function given x, Fe(ulz) is the CDF of fe(ulæ), {o yfr),N is the training set and fa "y Nunt is the validation set. Though the formulation of recalibration above shows we can use any likelihood model for 9a. since isotonic regression is a method to fit a free form function the CDFs, choose use normalizing flows, specifically QuAR Flows, as these are more flexible than using autoregressive flows such as MAFs (Papamakarios et al. 2017).
After the two-step training, the generative process is:
Uniform 1)
" (=) y = F (ulzc)
where Go is the CDF of 9a. The density function of the above generative process is simply the product of the individual density functions: fo and 9g. Given this view of the generative process, one interpretation of the two-step process to first fit 0 on the training set where & is initialized so that G-1 is the identity function, and then fit & on the validation while keeping frozen. Though
4

---

Under review as a conference paper at ICLR 2021
15
15
-10
-10
(a) Recalibrated via isotonic regression
(b) Recalibrated via normalizing flows
Figure 3: Comparison plots of isotonic regression and flow recalibration on multiple outputs. The two axes different targets being modeled, and the scatter plot shows that the targets correlated. The top and right sides show the marginal histogram as well as the fit from the recalibrated distribution. The contour plot shows the two-dimensional contours of the recalibrated distribution which can be seen to not align well for isotonic regression.
and á could be trained end-to-end, similar the approach Section 4.1 we show in our experiments (Section 6.2) that there are benefits to this two-step approach.
Given this log-likelihood viewpoint of recalibration, we can trivially incorporate conditioning of the normalizing flow additional information. For example, say fe outputs and that parameterize Normal distribution, we could then condition Ia on these two outputs. If were to use many features to condition on, we would return to the first situation where we could overfit to the finite number of observations we have. We can think of the first model fo as compressing/emtee /eteeediing/mmdddiingggg the data into two dimensions, and the second model utilizing the embedding create meaningful and well-calibrated distribution on the target. Compared to isotonic regression, we can easily incorporate conditional information while retaining monotonicity as normalizing flows are monotonic by construction.
Recalibration is interesting in that it is useful for both overfitting as well as underfitting. From Guo et al. (2017), uncertainties and benefit from recalibration. On the other hand, the simple example in Figuur sufferrs from miscalibrated uncertainties due approximation uncertainty (underfittin)) and recalibration reduces the approximation error.
that multinomial classification models have sufficient capacity to overfit
One solution handling overfitting is to early-stop validation set. Whether early stopping occurs or not, our solution incorporates the information of the validation set to correct potential overfitting. However, if we try to correct for multiple sources of overfitting, then we could introduce overfitting problem. In essence, there is a tradeoff between the validation size (which in turn controls training size) and how many sources of overfitting one can try to correct after training.
4.3 MULTIVARIATE RECALIBRATION
Whereas Platt scaling (Guo et al. 2017) and isotonic regression (Kuleshov et al. 2018) have been shown to improve the calibration of distributions when we have one-dimensional outputs, scaling these solutions to high-dimensional targets is non-trivial.
Say we are modeling two-dimensional real-valued targets and use isotonic regression to calibrate the targets individually. For simplicity, say distributions. If the targets are not conditionally independent, the distribution will not well- calibrated, even after recalibration with isotonic regression (Figure 3a). However, using normalizing flows, we can compute the CDF the observed targets per output distribution and recalibrate this two-dimensional distribution (Figure 3b).
amortized inference with independent output
The general procedure of recalibrating N-dimensional distributions to fit a normalizing flow on the N-dimensional vector v; F.y.lye)).
5

---

Under review as a conference paper at ICLR 2021
1.0
1.0
0.8
0.8
Emptrical CDF
Empirical CDF
0.4
0.4
Unrer
E10.00 0.25) (4.57)
FIYs) E|0.00, 0.25) (0.01)
El0.25, 0.50) (0.43)
E10.25,,, (0.01)
0.2
EI0.0000 (0.33)
0.2
FIG E|0.50.0.73)
FiYs) E10.75, (4.94)
FIYs) Ef0.75, 1.00) (0.01)
Aggregate
Aggreçate
0.0
0.0
F(Y2lY1)
F(Y2(Y1)
(a) Recalibration via isotonic regression
(b) Recalibration normalizing flows
Figure 4: Calibration Q-Q plots for recalibration using isotonic regression and recalibration using flows for the simulated multivariate case in Figure 3a The legend contains the calibration error for each curve.
5 MULTIVARIATE RECALIBRATION EXAMPLE
In multiclass classification, often focus individual metrics such accuracy, precision, recall, F1, etc. However, exploring the accuracy of the separate classes using a confusion matrix can show the model is biased even if the performance we extend the calibration plot introduced in (Kuleshov et al., (018)) to show calibration performance on different subsets of the data, binned by either feature or prediction, e.g. Figure 4 restriction is the binming cannot be function of the target we are evaluating.
average is good. Similar to this view for classification,
The only
Say we are modeling two-dimensional real-valued targets (Y Y2) shown in Figure 3a If we fit independent Gaussians to the target, the resulting distribution will not align with the contours of the data.
We first recalibrate the distribution by fittting isotonic regression models on F(Y1) and F(Y)(11). The results for Fi) are shown in Figure 4a where we can see that, though across the full dataset the calibration eтror is zero, when we bucket by F(Yi). the calibration error achieved is both significant and strong function Y1 as is unable to account for correlations in Yi and Y2. In other words, the model is not calibrated in any given region of the data but only in aggregate.
However, we recalibrate using normalizing flows by fitting a flow on the pair (F(Y)), F(Y2)Y1)). we can see in Figure that even if we bucket by F(()), the calibration error stays close to zero Unlike isotonic regression, the flow can learn the correlation between the CDFs by being trained jointly all of them. In addition to this, had we used more complex calibration model that could model the correlations directly, we would not be able to compute the correlation after recalibrating with isotonic regression. By recalibrating with flows, we obtain result that is still distribution with a closed-form likelihood; this allows us to still be able to use the model to compute the correlation and any other distributional statisties.
In Figure 5)
we show visually the difference between recalibration with flows and with isotonic regression. The key differences being that flows can recalibrate the CDFs jointly and be conditioned on additional information c, while isotonic regression is fit on each CDF independently and is unable incorporate any additional information. The reason is that flows are monotonic by construction and can therefore handle multiple variables and conditions.
EXPERIMENTS
In this section, we analyze the performance of flows on multiple UCI datasets. Importantly, we do not aim for state-of-the- performance, nor do we expect normalizing flows to outperform isotonic

---

Under review as a conference paper at ICLR 2021
Calibration
X
Se
Pso (11 8/2 (X)
Isotonic Recalibration
Flow Recalibration
(1(1
Fo(22))1)
(v
Fo(valv1)
Iso
Iso2
9
q1
42
Pys (F()), F(v2la))lc)
Figure 5: A diagram for multivariate recalibration using the same notation as that used in Section 4.2 The first step for recalibration is always the calibration step, shown in the blue box, where any likelihood model fe can be used. On the left, isotonic regression is used to recalibrate the CDFs from fo; on the right, normalizing flows Dø are used to recalibrate the CDFs from fo conditioned on some
regression in aggregate. terms of calibration, we show that the increase in expressivity can be beneficial; in terms of recalibration, we show specific failure points of isotonic regression that binning uncovers and how normalizing flows fixes these issues.
6.1 CALIBRATION
To the effect of increased capacity in distributions, tested how well normalizing flows improve performance in distributional estimates compared to few other popular approaches for estimating uncertainties. We do not compare directly against other methods for conditional distributions such as autoregressive flows for the reason specified in [Section 3.3) in the one dimensional case, those are often the same as ConditionalGaussinn Instead of replacing the output distribution with a complex distribution created by hand, such as an explicit mixture, we rely using more expressive flows which can model mixture behavior as well as more complicated behavior such as skew and kurtosis.
Table 1: Results before and after recalibration on the UCI dataset sauval-rropulsion on the model methodology well as performance other UCI datasets are in [Appeddix B The mean squared error (MSE) and its uncertainty is computed using 20 seeds. "Calib" " is the calibration error (Equation 1} using 100 quantiles across 20 seeds instead of per seed.
Details
Before Recalib.
Isotonic Recalib.
Flow Recalib.
MSE
Calib
MSE
Calib
MSE
Calib
ConditionalFlow
0.009 + 0.004
0.45
N.A.
0.01
0.016 + 0.012
< 0.01
ConditionalGaussia
0.012 + 0.006
2.35
N.A.
0.01
0.012 + 0.006
< 0.01
Bayysianniigeeeggrsse
0.055 + 0.004
8.03
N.A.
0.01
0.48 + 0.02
< 0.01
MC Dropout
0.005 + 0.002
0.51
N.A.
N.A.
N.A.
N.A.
SQR
0.0052 + 0.0012
N.A.
N.A.
N.A.
N.A.

---

Under review as a conference paper at ICLR 2021
All results were evaluated held out test that not used for calibration recalibration. Due to limited space, the full extent of the results and details of the models are left for Appendixx A and Appendix B, To show the usefulness of the plot we introduced in |Section 5 focus the results on the UCI dataset "naval-propulsioonpaan""" Dheeru & Karra Taniskidou 2017) in Table 1 and Section 6.2
The models with consistent performance across UCI datasets in terms calibration are Bayesian Ridge Regression, Conditional Gaussian, and Conditional Flow, with Conditional Flow often per- forming well in cases where Conditional Gaussian unable to. Whereas these three models are well calibrated, the mean performance is much weaker for Bayesian Ridge Regression due the limitations of linear functions.
6.2 RECALIBRATION
Of the models we tested, we recalibrated the Bayesian Ridge Regression, Conditional Gaussian, and Conditional Flow models. MC Dropout and SQR could also be recalibrated, though we chose to as they lack closed form CDFs, meaning the technique would be approximate. In (Table 1] we see the calibration errors after recalibrating with normalizing flows are comparable to those from isotonic regression. However, after isotonic regression is applied, only the calibration error can be computed. In contrast, we can still compute ordinary distribution statistics such as the mean after applying normalizing flows. Due to this property, we can even detect cases in which an improvement calibration error coincides with a degradation mean performance, suggesting if the goal were to only get point estimates of the mean, recalibration could be detrimental. For example, in Table 1 this was the case for Bayesian Ridge Regression.
In Figure 6a whereas the calibration enoor metric shows the flow recalibrated model is well-calibrated averaged across the full dataset, we can see that see that, when bucketing by the predicted mean, the calibration error is significant.
To improve this, we recalibrate with a normalizing flow conditioned on the mean predicted by the in Figure 6b that though the aggregate performance does not
original model. In this way, change, the performance is improved across different buckets. In essence we see that we were analyzing the distribution across the dataset, would be accurate; if started look closer individual distributions, the performance is worse suggesting the quantiles of individual distributions cannot be trusted.
1.0
1.0
0.8
0.8
Empirical CDF
Empirical CDF
0.6
0.4
Calimation Eroor
Calioution
E1-10, 0.5) (0.96)
#[-10, -0.5) (0.02)
El-3.000 #TY;] (0.0.0.5)
E(-05, (0.01)
0.2
(0.28)
0.2
#tY; (0.00.00)
(0.02)
Y11 (00.5,
10.33)
st (00.5,10)
(0.02)
Apuregate
Angrequte
(0.00)
0.0
0.0
0.2
F(Y)\Y))
F(Y))Y))
(a) Unconditional Flow
(b) Flow, Conditioned on Predicted Mean
Figure 6: Calibration Q-Q plots for recalibration with an unconditional flow and recalibration with conditional flow on the dataset "naval- shaval-propulsion-paane The legend contains the calibration error for each curve.
8

---

Under review as a conference paper at ICLR 2021
1.0
1.0
0.8
0.8
Empirical CDF
Empirical CDF
FIYS) E10.00, 0.25) (6.30)
FIYs) E|0.00, 0.25) 12.0 E(0.25,00000000 (0.68)
El0.5, 0.50) (1.73)
0.2
FLY]) E70.000000 (0.28)
0.2
FIY E|0.50.0.75)
FiYs) E10.75, 1.00) 10.36)
FIY El0.75, 1.00) (0.03)
Aggreegen sggreegat
Aggreçate
0.0
0.0
F(G()))
F0g//))
(a) Recalibrated via isotonic regression
(b) Recalibrated via normalizing flows
Figure 7: Calibration Q-Q plots for recalibration with isotonic regression and recalibration with normalizing flows on the second time step on the dataset "electricity". The legend contains the calibration error for each curve.
6.3
TIME SERIES
For our time series example, we trained a model on the UCI Dataset "eelccrricty".. We predict two time steps and show that even though we used an autoregressive model for calibration (the model could learn correlated distributions), there is still some correlation not captured that recalibration can help with.
Figure 7a shows that the calibration error of Yi) when recalibrated independently with isotonic regression is significant. Figure 7b) shows that a flow trained jointly on the CDFs significantly reduces the errors. Although the calibration error of the marginal distribution for the second step is comparable hetween the two approaches, the conditional performance favors normalizing flows. A possible explanantion of the still imperfect calibration of the flow model, though better than isotonic regression, is distribution shift.
7 CONCLUSION AND FUTURE WORK
In this paper, addressed the calibration issue in modern neural networks through using higher capacity distributions via normalizing flows. We showed that normalizing flows can not only help during calibration, but also recalibrating miscalibrated distributions, even improving the fit in the underfitting case. Though isotonic regression also uses a two-step producedure, we reframe the problem MLE through which we can trivially generalize recalibration the multivariate case. Though isotonic regression could be extended such that we could compute the log likelihood implied by its change of variables, normalizing flows are more natural way to handle this with more research backing this technique, and are easily extendable to conditional and joint distributions. Interestingly, though the Conditional Flow has all it needs be well calibrated, recalibration helps the overall calibration which was similarly found for classification in Guo et al. (2017); it remains for future work to better understand why and when this is useful.
Further, we extend the plot introduced in Kuleshov et al. (2018) by showing the calibration error different subsets of the data, for example bucketed by mean prediction. Though this plot is able to show the calibration error across buckets, for future work, will explore ways to visually show any trends the miscalibration as well as whether the misalignments of the quantiles are statistically significant.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Nicola De Cao. Wilker Aziz, and Ivan Titov. Block neural autoregressive flow. In Conference on Uncertainty in Artificial Intelligence, volume 3, pp. 1723-1735. UAI, 2019. URL http: //avai.org/ua http://aaa1199/////// http://///0///or.eecooaooaaa//a/
/ 511.pdf
Ricky I. Q. Chen, Jens Behrmann, David K Duvenaud, and Joem-Henrik Jacobsen. Residual flows for invertible generative modeling. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dÁlché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. Curran Associates, Inc., 2019. URL Desidual-flowee for invert ible -
Djork-Aréé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs), arXiv e-prints, art. arXiv:1511.07258 November 2015.
Armen Der Kiureghian and Ove Ditlevsen. Aleatory or epistemic? does it matter? Structural safety, 31(2):105-112, 2009.
Dua Dheeru and Efi Karra Taniskidou. UCI machine learning repository, 2017. URL http: /archive. www.cc....uu///..
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear Independent Components
Estimation. arXiv e-prints, art. arXiv:1410.855 October 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using Real NVP. arXiv e-prints, art. urXiv:1605. 000800, May 2016.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1050-1059, New York, New York, USA, 20-22 Jun 2016. PMLR. URL
Achintya Gopal.
Quasi-Autoreegressive Residual (QuAR) Flows. arXiv:2009.04419, September 2020.
arXiv e-prints, art.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models. arXiv e-prints, art. arXiv:1810.01367, October 2018.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 Proceedings of Machine Learning Research, pp. 1321 International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL
Jonathan Ho. Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2722-2730, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http::ppp//ppoceeenng.......e v99/hol 19.htmml
Chin-Wir Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autore- gressive flows. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th Interna- tional Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Re- search, pp. 2078-2087, Stockholmsmssan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http::/proceedii ngs.ml wtt.eeeedd..ss..m..r.r...... r.press/v80. hww.0//uaaggg1a 8d.html
Guolin Ke. Qi Meng. Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie Yan Liu. Lightghm: A highly efficient gradient boosting decision tree. In
10

---

Under review as a conference paper at ICLR 2021
I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems 30, pp. 3146-3154. Curran Associates, Inc., 2017. URL hotttboostinggg deep--eeiisiie btt.::ssttnnndccci.ooooooeeeeeee highly-efficienn gradi ent cont-bowt
69077 aig--
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learn- ing for computer vision? In L. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gareett (eds.), Advances in Neural Informa- lion Processing Systems 30, pp. 5574-5584, Curran Associates, Inc., 2017. URL
do de-ww--w--oeee--o-oee--e-ee--ee- we we-need-
in ----ayyeian--dee--ee------------ www.arannngggoooooooooteeeeeaaaa
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. KisaaBianchi, and R. Gameett (eds.),
Advances in Neural Information Processing Systems 31, pp. 10215-10224. Curran Associates, Inc., 2018.
httttp/p/ppaerrrssps.cccpppppppp /22aaper/8224-00e www.ggggwwgggggattaaeee flow with- bith-inverribbe-iall hat.ee-ixx-ccoooootiooospspppppo
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 4743-4751. Curran Associates, Inc.,
2016. URL http://paeessmips.cppppse//33 www.--oooooaaaaaaaioioooooo variational
inference-wie with www.-//neeee----trrrrreee.....ee flow.pd
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2796-2804, Stockholmsmssan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL wtt.:/ss.....pprrs//ssss/ ku wuleshovi 8a.html
Balaji Lakshminarayanann Alexander Pritzel, and Charles Blundell. Simple and scal- able predictive uncertainty estimation using deep ensembles. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances Neural Information Processing Systems 30, pp. 6402-6413. Curran Associates. Inc., 2017. URL http:/ttp///poooeea//p////////// sttp:///.e.......mmmmmmaaaaaccaa prreiitceeennnetttrrrreeetttttte
David Nix and Andreas Weigend. Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 ieee international conference on neural networks (ICNN "44),
volume 1, pp. 55-60. IEEE, 1994.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In I. Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 2338-2347, URL
Curran Associates, Inc., 2017.
ive- heraraeeeeeeeoo---rraaooooeeeeae
Tim Pearce, Alexandra Brintrup, Mohamed Zaki, and Andy Neely. High-quality prediction intervals for deep learning: A distribution free, ensembled approach. volume 80 of Proceedings of Machine Learning Research, pp. 4075 4084, Stockholmsmssan, Stockholm Sweden, 10-15 Jul 2018.
PMLR.
Boris Polyak and Anatoli Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30:838-855, 07 1992. doi: 10.113770330046.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Francis Bach and David Blei (eds.). Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1530-1538, Lille, France, 07-09
Jul 2015. PMLR. URL http: / http///preeeeennn...l.cc U
Filipe Rodrigues and Francisco C. Pereira. Beyond expectation: Deep joint mean and quantile regression for spatio-temporal problems. arXiv e-prints, art. 1oXiv:8888.08777 August 2018.
11

---

Under review as a conference paper at ICLR 2021
David Salinas, Valentin Flunkert, and Jan Gasthaus. DeepAR: Probabilistic Forecasting with Autore- gressive Recurrent Networks. arXiv e-prints, art. arXiv:1704.04110 April 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Dropout: A simple way prevent neural networks from overfitting. Journal of Ma- chine Learming Research, 15(56):122--198, 2014. URL
Brivastaval4a.htm.
Natasa Tagasovska and David Lopez-Paz. Single-model uncertainties for deep learning. In Advances in Neural Information Processing Systems 32, pp. 6417-6428. Curran As- sociates, Inc., 2019. URL uunertaiities--fon for -oe---ee----rr-nnnn.o--
88 2ttp///8870--111--- 10ll----l---eeeeeeneeeeeeeeeeeee model
Ichiro Takeuchi, Quoc V Le, Timothy D Sears, and Alexander J Smola. Nonparametric quantile estimation. Journal of Machine Learning Research, 7:1231-1264, 2006.
James W Taylor. A quantile regression neural network approach to estimating the conditional density of multiperiod returns. Journal of Forecasting, 19:299-311, 2000.
Michael Tipping. Sparse bayesian learning and the relevance vector machine. Journal of machine learning research, (ilm)):211224:: 2001.
Rianne den Berg, Leonard Hasenclever, Jakub M. Tomezak, and Max Welling. Sylvester Normalizing Flows for Variational Inference. arXiv e-prints, art. 1i3i::803300649 March 2018.
Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A Multi-Horizon Quantile Recurrent Forecaster. arXiv e-prints, art. arXiv:1711.1105, November 2017.
12

---

Under review as a conference paper at ICLR 2021
Affine
C-QuAR
Affin
2
X2
scale
shift
x
NN
Figure 8: In this figure, show conditional flow model. A neural network takes X and outputs the scale and shift of an affine flow and a condition vector for QuAR Flows. The condition used in the QuAR Flow is the same for both QuAR Flows. After the first conditional affine flow, there two QuAR Flows with unconditional affine flows placed in between these two and at the end. The Conditional Flow model is equivalent to Conditional Gaussian if removed the QuAR Flows.
A MODEL TRAINING DETAILS
A.1 CALIBRATION
We compare the performance of our model and multiple other popular approaches UCI datasets (Dheeru & Karra Taniskidou, 2017): boston-housing, concrete, energy, kin8nm, pavol-propulsion- plant, power-plant, vine-quality-red and yacht. The models compared Conditional Flows to
were: Conditional Gaussian (amortized inference with Gaussians, Lakssminarayaaann et al. (2017)).
Bayesian Ridge Regression (Tipping 2001). MC Dropout (Gal & Ghahramani [2016). Simultaneous Quantile Regression (SQR) (Tagasovska & Lopez-Paz 2019). Conditional Quantile (SQR where we only learn prespecified quantiles), Quality Driven (Pearce et al. 2018), and a Gradient Boosted Decision Tree with LightGBM (Ke et al. 2017).
Figure 8 shows a representation of the Conditional Flow network we used. The QuAR Flows are conditioned on 64-dimensional representation from the condition network. Each Conditional QuAR Flow has two hidden layers in the residual connection with a hidden size 64 with ELU (Clevert et 2015) as our activation. The condition network has a single hidden layer with size 64 and ReLU as our activation. The QuAR Flows are conditioned on the 64-dimensional representation from the condition network.
For other neural network approaches, we used networks with two hidden layers with size 128 and used ReLU nonlinearity. For hyperparameter tuning all our neural networks, searched the following grid:
Learning rate: 1e-2, le-3, lc-4
Weight decay: le-3, 1e-2, le-1
Dropout Rates: 0.1 0.25, 0.5 0.75
The hyperparameters we used were taken from Tagasovska & Lopez- Paz (2019). We trained each of our neural networks for 2000 epochs with batch size 128.
For our LightGBM tree model (Ke et al. 2017). we used mean squared error as our loss to learn the mean of the output distribution. The grid of hyperparameters we searched was:
Number of Trees: 100, 1000
Maximum Depths: 7,
Learning Rates: le-1, 1e-2, 1e-3, le-4
Minimum Child Samples: 1, 20
Dropout Rates: 0.1 0.25, 0.5 0.75
To make the tree model distributional, assuming homoscedastieity, computed the standard de- viation of the predicted normal distribution by computing the standard deviation the residuals (observations minus predictions).
13

---

Under review as a conference paper at ICLR 2021
For SQR and MC Dropout, computed the estimated mean and CDF from 1000 samples. For Conditional Flows, we computed the estimated mean by using the Gauss- quadrature with 100 points to approximate the integral.
A.2 RECALIBRATION
For recalibration, we trained the recalibration model on the validation set of each of the 20 seeds. For the flow architecture, we stacked 8 QuAR Flows with affine flows in between each as at the beginning and end. Since the imput to the flow is a CDF E [0. 1], we also transform the data with a Logit Function the first layer of the flow and account for the change of variables in the loss. In addition to this, we used a dropout rate of 0.25, two hidden layers with size 64 in each residual connection, and used Polyak averaging (Polyak & Juditsky 1992) with decay of 0.999 for evaluation.
well
A.3 TIME SERIES
For our time series experiments, we tested the performance of flow recalibration on the UCI dataset "electricity". We filtered the data down to observations after 2014-01-01. We split the data by date, where the first 72% of dates are in the train set, the next 18% are in validation set, and the last 10% in the test set.
The data is given in 15 minute increments, and so similarly to Deep AR (Salinas et al. 2017), we averaged the values per hour to have hourly data. Since correlated targets, we predicted two time steps conditioned on 48 hours of data using an autoregressive model.
trying analyze the effect of
We included three numeric covariates: the time step within the sampled time series, the hour of the normalized have zero mean and one
day, and the day of the week. Each of the covariates standard deviation. In addition to this, since there are 370 different types of time series, we also added an embedding layer from 370 to embedding dimension of 20, same as that used Deep AR (Salinas et al. 2017). We concatenated this vector with the observations and covariates, leading to 27-dimensional vector.
Since all the observations are positive, we model the target with a conditional Gamma distribution (i.e. amortized inference with Gamma distribution instead of a Gaussian). Similar to Deep AR (Salinas ct al. 2017). we normalized each time series {2,,))--) 17'1 by U. = 1 -T- Ei,t- We account for the scaling in the output distribution, i.e. Gamma(k, 0 U.).
The model we used was a layer LSTM, and then two linear layers with a ReLU and Dropout (Srivastava et al.) (214)) in between were applied the hidden representation. We set the dropout rate to 0.5 and used a hidden size of 48 everywhere. For evaluation, we used Polyak averaging (Polyak & Juditsky 1992) with decay of 0.999.
For recalibration with flows, we trained on the CDFs for both time steps with a flow with the same architecture that used in Appendix A.2
B EXPERIMENTAL RESULTS
B. CALIBRATION
In Table 2, we compare Conditional Flows against multiple other approaches for estimating uncer- tainty, and measure performance using Prediction Interval Coverage Probability (PICP), Mean Squared Error (MSE), and Calibration Error.
For each model, we trained the model on 20 different seeds where each seed randomly splits the data in 72% train, 18% validation, and 10% test. We picked the best hyperparameter by choosing the hyperparameters that gave the best average validation loss across the 20 seeds where the loss used is the loss defined by the training procedure for the model, e.g. for Conditional Gaussian, we used log likelihood; for LightGBM, used MSE; for Quilin, used the Quality Driven Loss, etc.
14

---

Under review as a conference paper at ICLR 2021
Table 2: Results on percentage data captured from 0.025 to 0.975 quantiles (PICP), MSE, and calibration error with 100 buckets. The results are from minimizing the average validation loss across 20 seeds. The optimal value of PICP 0.95, and the optimal value for MSE and Calib 0.00.
bostonHousing
concrete
PICP
MSE
Calib
PICP
MSE
Calib
Conditionalllow
0.93 + 0.04
0.13 + 0.06
0.06
0.92 + 0.03
0.09 + 0.03
0.01
ConditionalCaussian
0.93 + 0.03
0.14
0.13
0.91 0.02
0.02
0.09 + 0.03
0.06
MayesiannRideRRgreessi
+ 0.03
0.08
0.34
0.95 0.03
+ 0.07
0.01
MC Dropout
0.50 0.08
0.11
2.99
0.69
0.07 0.07 + 0.03
0.89
SQR
0.73 + 0.08
0.11 0.05
0.48
0.74
0.07 + 0.03
0.47
ConditionalQuantii
0.92 + 0.05
N.A.
N.A.
0.93
N.A.
N.A.
QualityDriven
0.83 1 0.06
N.A.
N.A.
0.8310.00
N.A.
N.A.
LightGBM
1
0.12
7.74
0.63 1 0.07
0.07 1 0.02
1.25
energy
kin8nm
PICP
MSE
Calib
PICP
MSE
Calib
ConditionalFlow
0.96 1 0.02
0.0022 1 0.0006
0.12
0.952 1:: 0.006
0.081 1 0.013
0.01
ConditionnlGaussian
0.97 0.03
0.0030 + 0.0009
0.26
0.947 + 0.007
0.071 + 0.005
0.01
BayeskanRidgeRogressio
0.89 0.04
0.09 + 0.02
0.24
0.951 + 0.007
0.59 + 0.02
0.09
MC Dropout
0
0.0026 + 0.0010
0.17
0.69 0.02
0.069 = 0.005
1.14
SQR
0.97 + 0.02
0.0028 + 0.0012
0.37
0.892 1 0.010
0.072 + 0.005
0.01
ConditionalQuantii
0.98 + 0.02
N.A.
N.A.
0.943 + 0.011
N.A.
N.A.
QuivinyDriven
0.96 0.04
N.A.
N.A.
0.88 + 0.02
N.A.
N.A.
LightGBM
0.24 +
0.0013 0.0006
5.18
0.205 0.010
0.190 + 0.011
5.59
naval-propulsioon-laal
power-plant
PICP
MSE
Calib
PICP
MSE
Calib
ConditionalFlow
0.98 + 0.03
0.009 + 0.004
0.45
0.952 + 0.007
0.048 + 0.007
0.01
ConditionalGaussian
0.9995 + 0.0007
0.012 + 0.006
2.35
0.947 = 0.010
0.048 + 0.006
0.01
BayeesanniigeeRegressinn
1.0 + 0.0
0.055 + 0.004
8.03
0.961 0.007
0.072 + 0.007
0.01
MC Dropout
0.98 + 0.02
0.005 + 0.002
0.59 + 0.02
0.046 + 0.007
1.88
SQR Conditional
0.998 + 0.002
0.0052 + 0.0012
1.44
0.93 + 0.02
0.048 + 0.007
0.02
0.97 + 0.02
N.A.
N.A.
0.937 + 0.014
N.A.
N.A.
QuavityDriven
0.949 1 0.010
N.A.
N.A.
0.89 + 0.02
N.A.
N.A.
LightGBM
0.714 0.011
0.0056 1 0.0009
0.79
0.17 0.02
0.035 1 0.007
5.95
wine-quality-re
yacht
PICP
MSE
Calib
PICP
MSE
Calib
ConditionalFlow
0.90 # 0.02
0.6 + 0.2
0.04
0.95 + 0.04
0.004 + 0.003
0.27
ConditionalGiausii
0.95 + 0.02
0.59 + 0.09
0.22
0.99 + 0.02
0.004 + 0.002
0.62
BayesiannidgeRRegeessii
0.95 + 0.02
0.63 + 0.11
0.05
0.94 0.03
0.35 + 0.10
0.25
MC Dropout
0.35 0.04
+.00
4.01
0.97 0.97 0.04
0.004 0.002
0.21
SQR
0.81 + 0.09
0.59
0.15
0.9610.004
0.004 1 0.004
0.49
Conditionalluanteo
0.95 1 0.02
N.A.
N.A.
0.902
N.A.
N.A.
Quality Driven
0.84
N.A.
N.A.
0.97
N.A.
N.A.
LightGBM
0.40 0.05
0.55 0.10
2.93
0.45
0.003 + 0.003
3.04
15

---

Under review as a conference paper at ICLR 2021
Table 3: Calibration error (with 100 buckets) before and after recalibration.
BayesianRidgeReerressin
ConditionalGaaussan
ConditionalFlow
Before
Flow
Iso
Before
Flow
Iso
Before
Flow
Iso
bostonHouusing
0.34
0.02
0.02
0.13
0.04
0.02
0.06
0.02
0.01
concrete
0.01
0.01
0.01
0.06
0.02
0.02
0.01
0.02
0.03
energy
0.24
0.02
0.01
0.26
< 0.01
0.01
0.12
0.01
0.01
kin8nm
0.09
0.01
0.01
0.01
0.01
0.01
0.01
0.011
0.01
naval-propulsion plant
8.03
0.01
0.01
2.35
< 0.01
< 0.01
0.45
< 0.01
< 0.01
power-plant
0.01
< 0.01
< 0.01
< 0.01
< 0.01
< 0.01
0.01
< 0.01
< 0.01
wine-quality-red
0.05
0.02
0.01
0.22
0.04
0.02
0.04
0.01
0.01
yacht
0.25
0.05
0.07
0.62
0.03
0.03
0.27
0.02
0.03
Table 4: MSE performance before and after recalibration. We do not show the MSE for isotonic regression as the mean cannot be computed after recalibrating with isotonic regression.
BayessiaRRigeeRReeesssonn
ConditionalGaussian
Before
Flow
Before
Flow
bostonHousing
0.28 + 0.08
0.28 + 0.08
0.14 + 0.07
0.14 + 0.07
concrete
0.40 = 0.07
0.40 + 0.07
0.09 = 0.03
0.09 = 0.03
energy
0.09 = 0.02
0.09 + 0.02
0.0030 = 0.0009
0.0032 = 0.0011
kin8nm
0.59 = 0.02
4.59
0.071 = 0.005
0.072 = 0.005
naval-propulsion-laar
0.055 = 0.004
0.48
0.012 0.006
0.012
power-plant
0.072 = 0.007
0.072 + 0.007
0.048 = 0.006
0.049 = 0.006
wine-quality-red
0.63 + 0.11
0.63 0.11
0.59 0.09
0.59 0.10
yacht
0.35 + 0.10
0.36 + 0.10
0.004 - 0.002
0.004 + 0.003
Conditional
Before
Flow
bostonHousing
0.13 + 0.06
0.13 + 0.06
concrete
0.09 + 0.03
0.09 = 0.03
energy
0.0022 + 0.0006
0.0027 = 0.0007
kin8nm
0.081 + 0.013
0.081 + 0.013
naval-propulsion-planr
0.009 + 0.004
0.016 + 0.012
power-plant
0.048 + 0.007
0.048 = 0.007
wine-quality-red
0.6 + 0.2
0.64 = 0.15
yacht
0.004 + 0.003
0.008 + 0.012
are the and standard deviation of PICP and MSE across the 20 seeds. Due to the limited data, for calibration error with 100 bins, we computed it over the CDFs from all 20 seeds. We included PICP (the percentage of true observations between the quantiles 0.025 and 0.975) to compare against alternative approaches where we prespecify which quantiles we need.
Reported in Table 2
B.2 RECALIBRATION
From the above section, we recalibrate the best Bayesian Ridge Regression, Conditional Gaussian, and Conditional Flow. In Table 3 compare the calibration error before recalibration, after recalibration with isotonic regression, and after recalibration with normalizing flows.
As we can see, across all 8 UCI datasets, the calibration enrors after recalibrating with normalizing flows are comparable to those from isotonic regression. More interestingly though, improvement in calibration enror can be detrimental to mean performance.
16


Reviews:
[{"review_id": "ah_sfcB04GN", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for calibrating uncertainty estimates for regression models. It builds off a method proposed by Kuleshov et al (2018). The newly proposed method has the following steps:\n  1) Train a conditional density model for a regression outcome y given an input x on training data. The authors use conditional normalizing flows for this task.\n  2) For each set of points (x_t, y_t) in a validation set, pass an input x_t to the model in step 1 to learn an induced CDF for the output F_t(). Calculate F_t(y_t), i.e. the induced CDF evaluated at the actual output, and use another flow-based density model to learn the distribution of the CDF values. If the model in step 1 is perfectly calibrated, this density should be uniform, but in practice it seldom is.\n  3) For future data points, the composition of densities in steps 1) and 2) provide a new, uncertainty-calibrated density.\n\nThe overall problem area is interesting, important, and underexplored. There has been a lot of work on calibration estimates for classification tasks, but there are less methods for regression. Using flows here is a cool idea. Any exact density model can be used for the method above, and normalizing flows are a good solution because they are monotonic by construction. \n\nThat being said, after reading the paper, I'm not convinced that the proposed method is a significant improvement over the method by Kuleshov et al. that it's building off of. Kuleshov et al. proposes a similar 2-step process, but instead of explicitly learning a distribution over the induced CDF values, it uses an isotonic regression to calibrate the CDF. Looking at the experimental results, it seems that the isotonic recalibration performs on par with the flow recalibration in terms of test set calibration error. \n\nPut another way, why should someone use flow-based recalibration instead of isotonic recalibration? A possible answer mentioned in the paper is that flow-based recalibration can be used to compute distribution statistics, such as the mean, while isotonic recalibration cannot compute these statistics. (Side question: why can't the isotonic recalibration be used to compute the mean? It seems like isotonic recalibration explicitly transforms an inverse-CDF to another inverse-CDF. Can't distribution statistics be imputed from this transformed inverse-CDF?). \n\nEven though flow recalibration can compute distribution statistics, the MSE never improves after flow recalibration; in some instances, it gets worse. What is the benefit of having distribution statistics? On the one hand, the worsened MSE might be expected behavior. Is uncertainty calibration expected to behave like a regularizer? If so, that should be stated and discussed in the paper. If not, then of course we shouldn't expect improvements in MSE after recalibration, because we're changing the model that had the best training-set performance. This could explain the results we see. In any case, there should be some justification in the paper for a) why computing distribution statistics is important, b) whether we should expect recalibration to behave like a regularizer, c) a discussion about the tradeoff between calibration performance and model error, and d) an illustration of scenarios where distribution statistics are crucial [and e), why the Kuleshov et al method can't be used to calculate test error].\n\nAdditionally, the paper proposes a way to visualize recalibration results. To be honest, I found the CDF performance plot confusing and hard to interpret. How should we interpret the x-axis (are predictions standardized, and if not, what units are they in)? I found the standard qq-plot-like calibration graph a lot more interpretable. What does the new visualization answer? I think the new visualization should be better explained (it also didn't help that the legend in Figure 5 blocked the middle of the graph).\n\nOverall, I think the paper proposes an interesting model, but it doesn't adequately justify when/why the model should be used over the existing method. I think there could definitely be scenarios where it is useful -- I just don't think the paper has adequately and convincingly illustrated them.\n\nPros:\n- Interesting and underexplored problem area\n- Thorough experiments\n- Normalizing flows are an interesting and new model for this problem\n\nCons:\n- Doesn't justify meaningful ways the method is different from existing methods\n- Visualizations are confusing", "extra_scores": {}, "extra_text": {}}, {"review_id": "OOyYBo8xKRO", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors propose an approach to calibrate conditional distribution estimation models. The approach uses normalizing flows to transform an existing model's predictions into a prediction that better matches the empirical quantiles to the theoretical quantiles. After this remapping procedure, the authors introduce a new plot to visualize calibration. Empirical benchmarks are run on a suite of UCI datasets.\n\nReview:\n\nI'm not sure what is really that interesting here. My high-level problems:\n\n- The remapping that the authors propose is just using a normalizing flow with a simple quantile calibration. Why do we need normalizing flows for this at all? Any model can be recalibrated using any other model here. Is there some special reason for normalizing flows here?\n\n- The new plot introduced is more confusing than illuminating. I really didn't follow it at all. It is very crowded and takes a lot of inspection to understand what is going on. I suspect all of this could have been visualized much clearer by using a handful of simpler plots that are straightforward.\n\n- The benchmarks do not really compare against very competitive models. The authors choose to use a model that was only pushed to the arxiv a month ago as the baseline. Why? Then the alternative choices are strawmen: a Bayesian linear regression model, a variational dropout model, etc. There is a wealth of conditional distribution estimation literature with companion code publicly available on github (NADE, MAF, MDNs, etc). Why not use those?\n\n- Does this really help us do anything new? Is this just \"my model is 0.1% better on 8 UCI datasets than 4 other models\"? Seems like a pretty uninspiring result if that's the idea.\n\nOverall, I just don't know what to see as the big contribution here. The paper feels a little rushed and could use a slower, more methodical pace where the authors carefully think through the contribution(s) and why they're necessary. A more thorough comparison to related work is also called for.", "extra_scores": {}, "extra_text": {}}, {"review_id": "M7SWXscDUil", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to use normalizing flows to improve estimates of aleatoric uncertainty in regression tasks. First, the paper suggests that since normalizing flows can improve the flexibility of output distribution, they can be used to mitigate issues of underfitting. Second, the paper proposes an approach that uses normalizing flows for recalibration, which allows people to still query the model’s statistical properties. The authors also introduce a plot that is useful for diagnosing calibration issues. \n\nThe paper’s approach of applying normalizing flows in recalibration seems that it could be useful to the community. The supporting experimental results look reasonable. In addition, the presentation of the paper looks nice, the experiments are well documented, and the diagnosing plots seem like a helpful tool. Given these plus points, I would recommend acceptance. \n\nOne suggestion is that it feels quite obvious that since flows can represent flexible distributions, using them to model aleatoric uncertainty can reduce underfitting issues. I am not sure if it is worth stating in the paper. It seems that the interesting part of the paper is recalibration, so perhaps it might be better to focus more on that. \n\nQuestions for the authors:\n- I wonder whether using flows to recalibrate is susceptible to overfitting. \n- The paper focuses on aleatoric uncertainty. How can the approach proposed in the paper be combined with approaches for addressing epistemic uncertainty?", "extra_scores": {}, "extra_text": {}}, {"review_id": "57qyvZSVdv", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a normalizing flow approach for calibrated predictions in regression tasks. Experimental results on toy and UCI datasets demonstrate the proposed method improves model calibration.\n\nIt is not clear what the technical contributions are. Also, most of the important details are missing.\n\n**Strengths**\n- The reviewer appreciates the effort towards improved calibration models; important for reliable predictions\n\n**Weaknesses**\n\n*Lack of clarity*:\n- The paper was difficult to follow, it omits several crucial details necessary to understand the proposed method. \n\nBelow are some general questions or suggestions:\n1) While the paper's principal focus is on calibration and recalibration, it is unclear why there are claims to address aleatoric uncertainty\n2) The concept of recalibration is introduced in Section 2 as a classification problem. However, in Section 4, the focus is on regression with normalizing flows\n3)  In Figures 1 and 2, what is $X, c, W_1, W_2$? \n4) Figure 2, add labels to x-axis and y-axis\n5) Improve caption quality across all Figures and tables\n\n*CDF performance plot*:\n- The paper claims the CDF performance plot is one of the main contributions, yet it is difficult to interpret or follow. Why are calibrated CDFs uniformly distributed?\n1) What is $\\sigma$?\n2) What is **pull**?\n3) What is $\\psi$\n\n*Recalibration Normalizing flows*:\n- What are the complete formulations of the likelihoods optimized in 1) and 2)?\n-  How is normalizing flow extended to multivariate calibration?\n\n*Weak experiments*:\n- How is the *Calib*  metric computed?\n- The experiments are on toy data and small UCI datasets. Additional large scale image datasets or regression tasks would strengthen the submission.\n\n ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to recalibrate predictive models by fitting a
normalizing flow on top of the predictive model on a held out validation
set using side information. At a high level this idea has some potential,
especially in the multivariate setting, but there are several directions for
improvement:

- Comparison with a broader set of baselines as suggested by the reviewers


- Clarity on why recalibrate with a normalizing flow especially in the 1-d case


- Why not any other model with explicit density? Are there other important desiderata?


- A motivating experiment that makes the potential value clear
Decision: Reject