Quantile Regularization : Towards Implicit Calibration of Regression Models
Authors: Saiteja Utpala
Piyush Rai
Venue:  (2021)
Keywords: Calibration
Reliable Uncertainty Quantification
Probabilistic Deep Learning

Abstract: Recent works have shown that most deep learning models are often poorly  calibrated, i.e., they may produce overconfident
predictions that are wrong, implying that their uncertainty estimates are unreliable. While a  number of approaches have been proposed recently to calibrate classification models, relatively little work exists on calibrating regression models. Isotonic Regression has recently been advocated for regression calibration. We provide a detailed formal analysis of the \emph{side-effects} of Isotonic Regression when used for regression  calibration. To address this, we  recast quantile calibration as entropy estimation, and leverage this idea to construct a novel quantile regularizer, which can be used in any optimization based probabilisitc regression models. Unlike most of the existing approaches for calibrating regression models, which are based on \emph{post-hoc} processing of the model's output, and require an additional dataset, our method is trainable in an end-to-end fashion, without requiring an additional dataset. We provide empirical results demonstrating that our approach improves  calibration for regression models trained on diverse architectures that  provide uncertainty estimates, such as Dropout VI, Deep Ensembles

Under review as a conference paper at ICLR 2021
QUANTILE REGULARIZATION: TOWARDS IMPLICIT CALIBRATION OF REGRESSION MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
Deep learning models are often poorly calibrated, i.e., they may produce confident predictions that are wrong, implying that their uncertainty estimates are unreliable. While a number of approaches have been proposed recently cali- brate classification models, relatively little work exists on calibrating regression models. Isotonic Regression has recently been advocated for regression calibra- tion. We provide detailed formal analysis of the side-effects of Isotonic Regres- sion when used for regression calibration. To address these, we investigate the idea of quantile calibration (Kuleshov et al. [018)), recast it as entropy estima- tion, and leverage the new formulation to construct a novel quantile regularizer, which can be used blackbox to calibrate any probabilistic regression model. Unlike most of the existing approaches for calibrating regression models, which are based post hoc processing of the model's output, and require additional dataset, our method is trainable in an end-to-end fashion, without requiring an additional dataset. We provide empirical results demonstrating that our approach improves calibration for regression models trained on diverse architectures that provide uncertainty estimates, such Dropout VI, Deep Ensembles.
INTRODUCTION
For supervised machine learning, the notion of calibration of a learned predictive model is measure of evaluating how well model's confidence in its prediction matches with the correctness of these predictions. For example, binary classifier will be considered perfectly calibrated if, among all predictions with probability score 0.9, 90% of the predictions are correct Guo et al. (217)). Likewise, consider probabilistic regression model that produces credible interval for the predicted outputs. In this setting, the model will be considered perfectly calibrated if the 90% confidence interval contains 90% of the true test outputs (Kuleshov et al. are known to be poorly calibrated (Guo et al. 2017). raising questions on their reliability.
2018). Unfortunately, modern deep neural networks
originally first considered in meteorology literature (Brier) 1950; Murphy 1972] Gneiting & Raftery| (007)) and saw one of its first prominent usage used in the machine learning literature by (Platt et al., 1999) in context of Support Vec- tor Machines (SVM), in order to obtain probabilistic predictions from SVMs which are inherently non-probabilistie models. Recently, there has been renewed interested in calibration, especially for classification models, after it has been shown (Guo et al., 2017) that modern deep neural networks for classification are often poorly calibrated.
The notion of calibration for elassification problems
The pupular notions of calibration for classification include confidence calibration, multiclass cali- bration, classwise calibration, and confidence calibration (Kumar et al. 2019; Vaicenavicius al.
2019 Kull et al.) 2019). Most calibration methods (Platt et al., 1999. Zadrozny & Elkan, 2000 2002 Guo al. 2017 Kull et al., 2017 2019) for classification models are post hoc, where they already trained model. There has been recent work showing some of these popular post hoc methods are either themselves miscalibrated sample- innffiiieffiicint (Kumar et al. 2019) and they do not actually help the model output wall-calibrated probabilities.
learn calibration mapping R [0 I + 0, I] using additional dataset to recalibrate
An alternative to post hoc processing is to ensure that model outputs well-calibrated probabilities after model training finishes. We refer to these as implicit calibration methods. Notably, such an approach does not require additional dataset to learn the calibration mapping. While almost all

---

Under review as a conference paper at ICLR 2021
post hoc calibration methods for classification models be seen in a unified manner density estimation methods (see section 2.1) ) existing implicit calibration methods for classification models have been designed with various, often distinct, considerratonns/ppraaccee Several heuristics like Mixup (Zhng et al., 2017; Thulasidasan et al. 2019) and Label Smoothing (Szegedy et al., [016] Müller al. 2019) that were part of high performance deep networks for classification were later shown empirically to achieve calibration. (Maddox et al., 2019) show that their optimization method instrinsically improves calibration. (Pereyra et al.) 2017) found that penalizing high-confidence predictions acts as a regularizer. A more principled way of achieving implicit calibration is by minimizing a loss function that tailored for calibration (Kumar et al., 2018). This is somewhat similar in spirit to our proposed approach which aims to do it for regression models.
Among the early approaches for calibrating regression models, (Gneiiing et al., 2007) were the first to propose a framework for calibrating regression models. However, they do not provide any pro- cedure to correct miscalibrated model. Recently, (Kuleshov et al.) 2018) introduced the notion of Quantile Calibration which intuitively says that the p confidence interval predicted by model should have target variable with probability p. They use a post hoc calibration method based on Iso- tonic Regression (Fawcett & Niculescu-Mizil] 2007), which is well-known calibration technique for classification models. The difference between Isotonic Calibration in classification and Isotonic Calibration in regression is in terms of (i) the dataset on which calibration mapping is learnt; and (i)) the function with which learnt calibration mapping is pre-composed. In the former case, it is pre-composed with a probability mass function (PMF) and whereas the latter, is pre-composed with a conditional density function (CDF). Both these differences have side effects; in particular (i) the nature of recalibration dataset already satisfies monotonicity constraint, so there risk of overfitting in case of smaller calibration datsets; and (ii) composing the CDF with piecewise linear function can make the resultant CDF discontinuous and the corresponding PDF donfeifferentiable (see Sec. se..3ire for detailed discussion for side effects of the Isotonic Calibration approach). In another recent work, (Song et al., 2019) proposed much stronger notion of calibration called Distributional Calibration which guarantees that among all instances whose predicted probability density function (PDF) of the response variable has mean μ and standard deviation o, the marginal distribution of the target variable should have mean μ and standard deviation a. They too propose a post hoc recalibra- tion method based on Gaussian processes, which can be computationally expensive. Among other work, (Keren et al.) 2018), consider different setting where neural networks for classification used for regression problems and showed that temperature scaling (Hinton et al. 2015 Guo et al.) 2017) and their proposed method based on empirical prediction intervals improves calibration for regression problems as well. Again, these are post hoc methods.
Our contributions are summarized below:
1.
We analyze in detail the side effects of Isotonic Calibration for regression models. We show how using Isotonic Calibration results in truncation of the support, which will result in assigning zero likelihood fortes time. We also discuss about Isotonic Calibration re- sulting in nonsmooth PDFs, and its tendency to produce miscalibration when using small calibration datasets.
2. At test time, after composing the predicted CDF with the learned isotonic mapping, the mean prediction (point estimate) also changes. Kuleshov et al. (2018) do not acknowledge the changes in the mean estimate. While Song et al. (2019) acknowledge this issue, they use a trapezoidal approximation to remedy this. In contrast, we derive an analytical expres- sion for the updated point estimate after isotonic calibration. We also provide a different expression for updated point estimate, which reduces the time-complexity from O(m) to (1), where m is calibration dataset size.
3. In order to mitigate these shortcomings of Isotonic Calibration, we propose a simple, yet novel and general purpose, trainable loss function for quantile calibration where the smoothness of PDF/CDF is not sacrificed for well-calibrated probabilties. Our approach also eliminates the need for an additional calibration dataset.
4. We conduct extensive experiments on wide range of architectures using the proposed loss function (Quantile Regularization) and show empirically that it improves calibration on wide range of architectures that produce uncertainty estimates.
2

---

Under review as a conference paper at ICLR 2021
Targets
Architecture Specific Loes
Total
Parameters
Treining (K,)
Model
Entropy
Parameters
Differentiable Differential Entropy Estimator
CDF evaluater
Targets
Figure 1: Computation of loss in the training loop when augmented with Quantile Regularization
(QR). Parts in Red are the ones that constitute QR. Total Loss Loss Entropy
BACKGROUND AND DEFINITIONS
Before we proceed with definitions, we state the notation followed in the rest paper. X, Y denotes the input and output space respectively. X, Y denote random variables modelling inputs and outputs. P denotes probability measure f. are reserved for probability density functions (PDF) and F, for cummulative density functions (CDF). We use denote the evaluation of CDF at some particular value i.c., c = F(.) and for evaluation elements 01, d2..... an use a(1), a(2)..... ((n) for permutation s.t a(x)) I ((+-)). Also, denotes the calibration dataset size. yX. denotes the training data. Given random variables X, Y, we use (L(X||Y) to denote KL divergence between corresponding distributions
PDF at some value i.c., = f(.) Given sequence of
A probabilistic regression model can be seen conditional PDF/conditional CDE. In the rest of the paper, we express as conditional CDF X' - () - [o, 1)). So, M(x)) denotes model's predicted CDF for X' denoted Fz. In practice this is achieved by making model output parameters that parametrize the CDF, e.g., (μ, or) for Gaussian, À for Exponential, etc. the rest of the paper, consider Gaussian likelihood unless stated otherwise, because it is of the most prevalent cases. Kuleshov et al. (2018) proposed the following notion calibration of regression models, called quantile calibration. An appearling aspect of this definition is that we get reliable confidence intervals.
Definition (Quantile Calibration) Given regression model M x -2 [0, 1]) and X,
jointly distributed as P, the model M is said to be Quantile Calibrated iff
P [M(X](0) Sp =p Vp [0.1]
(1)
In words, [M()](() is cumulative density that the model predicts for random input-response pairs drawn from the joint distribution of (X, Y). It important to note that, regardless of dimensions of
X Y and support of distributions of X, Y and the likelihood model considered, [M(X)] is
random variable whose support is a subset of [0, 1] because range of any real-valued CDF is [0, 1]. So Def. is quite general and covers the case when output netor-valued, i.e., y R n >
2.1 FUNDAMENTAL THEOREM OF POST HOC CALIBRATION
The objective of post hoc calibration is to recalibrate a miscalibrated model M by learning a map- ping R [0, the definition of calibration itself. Setting R (p) IP [M(X)] < makes R M quantile cali- brated model. (Vaicenavicius et al. 2019) call an analogous mapping in the context classification canonical calibration mapping.
[0, 1] s.t. R M is calibrated model. One such mapping can be obtained from

---

Under review as a conference paper at ICLR 2021
Theorem For any Model M X +V'+ [0) and given the canonical calibration mapping R(p) = P[[M(X))(Y) S p]. R o M is quantile calibrated
Note that learning the mapping R reduces to density estimation of PP P[[M(X][()) < p]. which is hard problem in itself. With this insight, and using the fact that the mapping monotonically increasing, (Kuleshov et al., 2018) use Isotonic Regression to learn the mapping separate cali- bration dataset. Similar theorems hold for other notions of calibration, too.
2.2 ISOTONIC CALIBRATION
We next describe Isotonic Calibration which is based Isotonic Regression. Given Data {(a;, b where a, b, E R and a, S a;-1. Isotonic Regression finds new E1, Em by solving the following optimization problem
(b; e²² min e
(2)
1=1
s.t.
e1
S e2 ... S..S... em
Mair et al. Scikit-learn (Pedregosa al. (00TT) uses Pool Adjacent Violaters Algorithm (PAVA) of complexity (mm).
2009) provides a nice survey of algorithms for Isotonic Regression. In particular
Given the calibration dataset {xi, yi Her Isotonic Calibration first builds recalibration dataset as
D = { ((xx))aa)
1 Σ
M(x;)lu S (xx)(u))
and sort points in D based on |mm, m
m7
first coordinates and then applies Isotonic Regression to get an isotonic mapping R. Let Frest be test input and Fr... M(Xres)) be the CDF of its predicted output. Now, during post-hoc calibration, we get a new CDF as Gr = R F Confidence intervals can be obtained from Gz Note that the parameters that parametrize the G Gamm (after calibration) are different from F Fa (before calibration). Importantly, what this implics is that mean prediction before and after calibration changes as well. Kuleshov et al. 2018)) do not take this aspect into consideration, whereas (Song al. (019)) uses trapezoidal approximation to find the new mean, and finding which has a time complexity Om)). ). In contrast, derive analytical expression for the mean which reduces time complexity from ((m) to (11) at test-time (See Eq 4), assuming a Gaussian likelihood model.
3 ANALYSIS OF ISOTONIC CALIBRATION
Let {(x, 3:) be the calibration dataset. Let {wy )m be means and variances predicted by the learned model. Suppose we have obtained CDF values at actual outputs (y1, y2. ... ..m) " Let these be (c1, C2 ... C. Then the re-calibration dataset after sorting based on first co-ordinates
(C(1): G(2). ... ((m) ) is obtained by sorting (c1, C2. ... Cm) ascending order. Now Isotonic Calibration fits isotonic regression on the dataset D. Our entire analysis is based on next crucial observation. Since (-)- 2
would be = ((q1): ) ) (C(2)= )..... (C(m); 1)} where
1) are already increasing order, isotonie regression does not modify values. In the notation Sec. [2.2 what this means that, if we have that (a,, by) A (c(e) }) then it will be e; = 1 i.e., b, will be same as e,, So we do not even need to use Isotonic Regression and just linear interpolation on D yields the same result as Isotonic Regression.
We first give conditions under which smoothness is lost and later derive the correction one has to use after isotonic calibration. All the proofs are provided the supplementary material.
Claim Let Y be a random variable with CDF F, and let = R Ro F be its CDF after composing with mapping R obtained from isotonic regression characterized by C = (())) .(m)): there exist i - 1, i +1 E {0,m) m) AC()) -C(-11) F C(1+1) - C(a) then the CDF G is not differentiable and its corresponding probability density function g is not continuous at (c()
4

---

Under review as a conference paper at ICLR 2021
Because of this, the PDF of transformed r.v becomes discontinuous and spiky (see Fig. 2d Update likelihood inversely depends on m((()) C(i+1)) (see Eq. 10 for full expression). In many of cases, becomes so small that sometimes the updated likelihood increases by factor of 102 to 10 for single point thereby completely destroying what the average log likelihood represents. To show this, we report the maximum likelihood attained among the entire test dataset after isotonic calibration in UCI experiments (see Table Table 5
Now we derive the analytical expression for the updated mean after Isotonic Calibration assuming Guassian likelihood.
Claim Let Yuo be the transformed random variable after applying isotonic mapping R the random variable Y. Then the expectation of Yiso is as follows
f(F ((F"(((H1))) - f(
Y =
m
(c(++)) c(1)
The summation involves both recalibration dataset and test time prediction. Now we will use prop- erties of quantile functions (Lemma. D to decouple the dependency, using which summation just depends on the recalibration dataset.
Claim Let ((S) Fr (M()) and P(1) = Spco (W(3))
1 (1+1))((+1) (())())
E i 11
m
A(i+1) (())
Now 8 can be computed once, at test time given μ., model prediction, the updated mean after isotonic calibration is 14 åo. With this, the time required reduced from ((m) to O Also, the construction of recalibration dataset as suggested in (Kuleshov al., 2018) result in truncation of the support of updated r.v (see fig. (2) and see Sec. B. for more on this. One way to remedy this is use (x, oc) calibration dataset for any R So the recalibration dataset becomes (C(1) m+T m ..... (C(m)) n (1,
To illustrate all these, we use Bayesian linear regression with 256 training examples generated from y 3r where NO, 1). and E [-4,4] 4) randomly sampled and calibration dataset of size 32.
4 QUANTILE REGULARIZATION
In quantile calibration, we want P [[M(()](Y) S p] = Vp E [0, 1]. Our method is based on the idea that both right and left hand sides can be viewed as CDFs. Let R(p) PP [[[(()](() < p] and Sp) = Here Rt can be seen as the CDF of [MY while S can be seen as the CDF of Uni Quantile calibration mandates these two CDFs to be equal. So, for perfectly calibrated quantile model M, we have that [M(X][(Y) be uniform distribution. We seek to penalize the model when this deviates from uniform distribution which gives calibration loss that can be used as a regularizer with any regression loss, achieving highly desirable calibration during training itself. We name above proposed procedure as Quantile Regularization, hereafter denoted as QR.
We use the KL divergence as the distance measure. Note that the KL divergence between any distri- bution on [0, 1] and the uniform distribution is negative of the differential entropy, which provides a very intuitive interpretation for the Quantile Regularization (QR). Essentially, improve calibration while training. QR maximizes the differential entropy of [M)()]( i... the predicted cumulative density of target value. We formalize the statement for completeness below.
Claim Let M be any regression model. Then M is perfectly quantile calibrated iff
KL ( (IM(X))(()II)) =0
(5)
5

---

Under review as a conference paper at ICLR 2021
bytani PIE
33
(a) Training Data
(b) Learnt Isotonic Mapping
645
Tuacaaiin safter
035
1.9
028
035
332
053
028
0.08
Predicted PDF
(d) PDF after Isotonic Calibration
Figure 2: Fig. shows 256 training data points, Fig. 2b shows the calibration mapping learnt using Isotonic Calibration with 32 samples. Fig. [c shows predicted PDF at test time before applying Isotonic Calibration and truncation and shift in mean that happens if the mapping in Fig. 2b is applicd It also shows that such truncation can affect the support. Fig. shows the resulting PDF after Isotonic Calibration
4.1 DIFFERENTIAL ENTROPY ESTIMATION
We need the differential entropy estimator for deriving calibration loss function. There is rich literature for entropy estimation. brief overview about non-parametric entropy estimation can be found in (Beirlant & Dudewiez 1997). We use sample-spacing entropy estimation originally proposed in (Vasicek] 1976). A l spacing of random variable is defined as the amount of probability mass between ordered samples that are k 1 samples apart. Let S be one-dimensional n.v. with CDF F and assume we are given samples {s.) S and & s.t. < k S n. Sample spacing entropy estimation based on the following observation k-spacings random variables
Es (()(41)) F(s())
k
n ++
There are many formulations of sample spacing entropy estimators but use the one in Learned Miller et al. (2003); Equation 8 which is
n
A(S)
log
(8((4)) - -(1))
(6)
12
1
k
4.2 CALIBRATION LOSS FUNCTION
In our case, the random variable is [M(X)[(Y). Given training data with input-output pairs (24, y:), need to get samples [M(x;)](ss). compute the expression H(S), and maximize it. Note that, we want to make this part of the training loop to achieve implicit calibration. To do so, we need ordered samples compute the Entropy in Eq. 6 However, inherently, sorting not differentiable

---

Under review as a conference paper at ICLR 2021
operation. We therefore use NeuralSort (Grover al. 2019) differentiable relaxation to sorting. We summarize our Quantile Regularization algorithm below
Algorithm Quantile Regularization
Precondition: (x,, 3i) are n i.i.d training instances and (Ai; MODELw (x:) and DIFFSORT is any
differentiable relaxation sorting operation.
function CALIBRATION Loss FUNCTION(Y, p.
2:
3:
for <- m do
4:
Φ; t 2 (1;; 0.)
D. Φ: CDF
5:
C; f P; [y:]
6:
end for
7:
s
+ DIFFSORT(C)
8:
k < vn
9:
log
(sith 8:)
10: return e
11:
12: end function
Assume that X. is the training data. Let (Jw, aw) MODEL., (X), where denotes the param- eters of the model, E(y, fw, ow) denotes the architecture specific loss and CL be calibration loss computed by Algorithm The overall loss function can be written as follows, where the is the hyperparameter that controls the effect of QR.
C(X, y, kww ow) = Ely, HAw, Ow) - L x CL fow ow)
4.3 DEGENERATE BUT PERFECTLY QUANTILE CALIBRATED MODEL.
Every notion of calibration has examples of models such that it is perfectly calibrated according to that specific notion but is far from the ideal model. We prove below that Quantile Calibration is no different. Specifically, we give conditions under which a model is quantile calibrated and then use it to build a model that is degenerate in sense that the model does not depend on the inputs while predicting output, but is perfectly quantile calibrated model.
Claim Let f be the marginal distribution Y Y and F- M be the model's predicted cumu- lative distribution for E X and fo be the corresponding predicted probability density then, if following holds, M Quantile Calibrated
Vx, Vp
Now if we set f= Ja Vx. the above condition easily holds. So, model that outputs marginal distribution of Y for every input E X. is perfectly quantile calibrated. Note that this ob- servation is general that it does not require Gaussian likelihood, for it to true. As a sim- ple example, consider f(x
NN 5z, 1x, 4). then the marginal distribution of y is f(v) 1 (5 1)/(z/00, 4)dx = Noy15.0, 5.42.5 + 12) (2 401) so the model which predicts (,)00, regardless of input is perfectly quantile calibrated, but the true model is (y, 1). So, a model can predict good confidence intervals despite being far from ideal. There- fore, need model to be both well-calibrated and sharp.
EXPERIMENTS
We evaluate our approach on various regression datasets terms of the calibration error as well as other standard metries, such as nott-mean-squared-eron (RMSE) and negative log- oog-liieldoood (NLL.).

---

Under review as a conference paper at ICLR 2021
Hetcrosccdastic Dropout VI
Dataset
Calibration Erron ())
RMSE
NLL
base
QR
base
QR
basc
QR
Air Foil
13.15 + 1.92
9.11
1,89
3.63 -0.05
4.05 + 0.05
2.70 = 0.01
2.83 + 0.01
Boston Housing
21.35 + 4.89
19.96
+ 3.36
4.59 =0.23
4.58
0.12
3.23 0.05
3.16 0.05
Concrete Strength
25.78 1: 2.01
15.90
1: 3.72
8.74 1: 0.18
9.11
3.61 - 0.03
3.66 1 0.02
Fish Toxicity
3.23 0.39
0.39
3.09 0.79
0.92
0.01
0.94
1.24 0.01
1.27 0.01
Kin8nm
7.23 0.69
5.41
0.09
0.00
0.00
-0.87 0.01
-0.72 0.01
Protein Structure
2.79 0 0.22
1.22 0.30
4.63 0.02
4.01
2,89 = 0.01
2.94
Red Wine
3.72 0.29
1.67 0.44
0.65 0.00
0.00
0.98 0.01
0,97 0.01
White Wine
4.30 0.48
0.00
0.01
1.11 0.01
Yacht Hydrodynamics
29.52 1: 3.36
2.70
3,55
4.20
2.28 0.03
2.51 1 0.02
Year Prediction MSD
5.16 NA
0.77 + NA
9.08 NA
9.32 NA
3 NA
3.48 NA
Table 1: Base Model is Dropout- VI model without Quantile Regularization and QR is when Base Model is trained with Quantile Regularization. NLL stands for negative log likelihood. RMSE stands for Root Mean Square Error. Bold represents there no overlap over I std interval.
5.1 12 QUANTILE CALIBRATION ERROR
Given any model M X - ( 1)), we define the l2 calibration error follows. Let us
choose M equidistant points {p,}-- (0, with Pas = 1. Given a test set {z,, y/ // whose predictions are F2 F(xn). ) the M-bin estimator of above integral gives us the calibration metric used (Kuleshov et al. 2018).
M
N
CE(F) = [ (P[M((X)(()) spl - P)
(F))))))
spl -Pl-pp
(8)
M
5.2
UCI DATA EXPERIMENTS
We consider two architectures Dropout VI Gal 2016) and Deep (ansebbles(.akshmmnarya al., (017). The dataset sizes ranges from 308 to 515345 and input feature dimensions ranges from 6 to 91. Every dataset, except Year Prediction MSD, is divided into 5 splits whereas for Year Prediction MSD there is a single split where we train on 463715 points and test on 51630 points. This experiment is repeated 3 times and averages reported except for Year Prediction MSD. We use 2 hidden layer network with 128 umits with ReLU activation. and trained with Adam Optimizer with learning rate of 10 Results are presented in Table | and 2 We use L = for Dropout-Vi VI and = 5 for Deep Ensembles. Spacing value is choosen as A - for all datasets except Year Prediction MSD, for which use h 3 /n for both architectures where is the batch size. See Sec. C.3 and Sec. C.4 for detailed experiments about how values of L, influence calibration enror and RMSE. The code and link the datasets be found here: https: //qithub. com/ www/oocc.mccccccoooooo.o.www.... www.mmccccmoooooooo ra-h--rr-22 20r/08
5.3 MONOCULAR DEPTH ESTIMATION
Now we consider problem Monocular Depth Estimation. We use architecture present Jégou et al. (2017) which combines principles of Dense-nets and U-net for Dense prediction tasks like semantic segmenation and depth estimation. We use Make3d Dataset (Saxena et al.) (2005) which has 400 scenes for training and 134 scenes for testing. We use 57 and 103 layer Neural Network, which denoted as FC-DenseNet577, FC-DenseNer103 in Jégou et al. (2017). We set L k /n where is batchsize. Also, use pooling layers cummulative density values to make sure the locality exploited and to decrease the computational time, while computing calibration loss. Results presents in Table. 3 and additional details and experiments in Sec. C.6
and
5.4 DISCUSSION
Both, in case Dropout VI and Deep Ensembles, calibration error improves when trained with quantile regularizer. Table indicates Quantile Regularization effective in large scale architec-
8

---

Under review as a conference paper at ICLR 2021
Deep Ensemble with Adversarial Training
Dataset
Calibration Erron(%)
RMSE
NLL
base
QR
QR
base
QR
Air Foil
24.61 + 2.31
16.17 + 3,91
3.13 + 0.04
3.34 + 0.08
2.95 = 0.04
2,59 + 0.02
Boston Housing
37.76 + 4.55
25.25 + 2.95
4.83 + 0.1
4.49 - 0.11
3,89 = 0.16
Concrete Strength
37.27 1: 2.94
22.76 1: 3.32
9.02 1 0.10
8.48
0.19
4.82 10.05
3.76 1 0.07
Fish Toxicity
3.26 1 0.62
1.67 1.67:10.30 0.38
0.92 1 0.00
0.92
0.00
1.57 1.00
1.31 1 0.00
Kin8nm
0.56 0.36
1.25 0.03
0.07 0.00
0.07 0.00
-1.35
-1.25 0.01
Protein Structure
2.48 = 0.17
1.62 1: 0.08
3.05
4.00
2.60 0.00
2.73 = 0.00
Red Wine
8.66 = 0.38
2.28 0.17
0.00
1.2 0.00
1.05 = 0.00
White Wine
8.30 0.75
0.76
0.76
164 0.00
1.14 1 0.00
Yacht Hydrodynamics
24.18 1 6.64
4.16
0.17
3 0.20
2.66 + 0.08
Year Prediction MSD
1.94 NA
0.51
8.68 NA
8.68 NA
3.34 NA
3.36 NA
Table 2: Base Model is Deep Ensemble without Quantile Regularization and QR when Base
Model is trained with Quantile Regularization. NLL stands for negative log likelihood. RMSE stands for Root Mean Square Error. Bold represents there no overlap over I std interval
FC-DenseNet57
FC-DenseNet103
Model
Calibration Error(%)]
RMSE 4
Calibration Error(%) |
RMSE |
Base
17.61
12.30
17.11
12.02
QR (ours) 12.43
13.55
10.94
12.60
Table 3: Base represents model trained without Quantile Regularization and QR represents Base model trained with Quantile Regularization.
tures as well. Tables 45 indicate QR-trained model works better than base in post-hoc setting too. Also in some cases, Isotonic Calibration increases the calibration error (e.g.. for dropout VI, see Conerete Strength, Yacht Hydrodynamics and for Deep Ensembles Boston Housing and Yacht Hydrodynamics). We believe this due to two reasons
1. Unlike isotonic calibration for classification calibration, recalibration dataset case of regression calibration already satisfies monotonicity constraint (see Sec. 33 So Isotonic calibration in case of regression calibration has much weaker regularization properties and hence overfits
2. Isotonic calibration is ponpparamttriic method that requires large amount of data work well. This could be another reason for the poor performance of isotonic regression some cases.
For large calibration datasets like protein structure (calibration dataset size m 36, 584) and Year Prediction MSD ( calibration dataset size m 463715), Isotonic Regression offers better improve- ments. Although it can work well in such setting, PDF after calibration becomes unusable. QR not only works well on small datasets but also works well on very large datasets, like Year predic- tion MSD (nearly 60 % improvement on both architectures). There is small increase in RMSE and NLL in models trained with QR, but in many cases it is negligible. A possible justification for this is that just because the model well calibrated doesn't 't necessarily mean is close to the true model. Being wellOcalibrated and being close true model are two separate things (see Sec. 4.3
CONCLUSION
We have proposed black-box calibration loss function that be used with any probabilistic regression model. Unlike current methods for quantile calibration, our method is implicit in nature and does not require additional calibration dataset and, importantly, the smoothness of the PDF is not lost. We conduct experiments to show effectiveness of our proposed method.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jan Beirlant and Edward Dudewicz. Nonparametric entropy estimation: An overview. Interna- tional Journal of Mathematical and Statistical Sciences, 6(1):17-39, 1997.
Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 18,1), 1950.
Tom Fawcett and Alexandru Niculescu-Mizil, Pav and the roc convex hull. Machine Learning, 68 (1):97-106, 2007.
Yarin Gal. Uncertainty in deep learning. University of Cambridge, 1:3, 2016.
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359-378, 2007.
Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series (Statistical Methodology). 69(2): 243-268, 2007.
Aditya Grover, Eric Wang, Aaron Zweig, and Stefano Ermon. Stochastic optimization of sorting networks via continuous relaxations. arXiv preprint arXiv:1903.000 088550 2019.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning-Volun Volume 70, pp. 1321-1330. JMLR. org, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1rXiv:1503.025// 2015.
Simon Jégou, Michal Drozdzal, David Vazquez, Adriana Romero, and Yoshua Bengio. The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. In Proceed- ings of the IEEE conference on computer vision and pattern recognition workshops, pp. 11-19, 2017.
Gil Keren, Nicholas Cummins, and Bjöm Schuller. Calibrated prediction intervals for neural net- work regressors. IEEE Access, 1.54033-540411 2018.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. arXiv preprint arXiv::
2018.
Meelis Kull, Telmo Silva Filho, and Peter Flach. Beta calibration: well-founded and easily imple- mented improvement on logistic calibration for binary classifiers. In Artificial Intelligence and Statistics, pp. 623-631, 2017.
Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. In Advances in Neural Information Processing Systems, pp. 12295-12305, 2019.
Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified uncertainty calibration. In Advances in Neural Information Processing Systems, pp. 3787-3798, 2019.
Aviral Kumar, Sunita Sarawagi, and Uijwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In International Conference on Machine Learning, pp. 2805- 2814, 2018
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predic- tive uncertainty estimation using deep ensembles. In Advances in neural information processing systems, pp. 6402-6413, 2017.
Erik G Leaamed-Miller et al. Ica using spacings estimates entropy. Journal of machine learning research, 1 2003.
10

---

Under review as a conference paper at ICLR 2021
Wesley J Maddox. Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems, pp. 13132-13143, 2019.
Patrick Mair, Kurt Hornik, and Jan de Leeuw. Isotone optimization in r: pool-adjacent--vollloos algorithm (pava) and active set methods. Journal of statistical software, 32(5):1-24, 2009.
Rafael Müller, Simon Korbblith, and Geoffrey E Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems, pp. 119444694000 2019.
Allan H Murphy. Scalar and vector partitions of the probability score: Part i. two-state situation. Journal of Applied Meteorology, ((())273-282. 1972.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Couraaeeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Gabriel Pereyra, George Tucker, Jan Chorowski, Eukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.0653 2017.
John Platt al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
Ashutosh Saxena, Sung Chung, and Andrew Ng.
Learning depth from single monocular images. Advances in neural information processing systems, 18:1161-1168, 2005.
Hao Song, Tom Diethe, Meelis Kull, and Peter Flach. Distribution calibration for regression. arXiv preprint 100.:::99000002333 2019.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826 2016.
Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tammoy Bhattacharya, and Sarah Micha- lak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. In Advances Neural Information Processing Systems, pp. 13888- 13899, 2019.
Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas B Schön. Evaluating model calibration in classification. arXiv preprint
arXiv:1902.06977, 2019.
Oldrich Vasicek. A test for normality based on sample entropy. Journal of the Royal Statistical Society. Series B (Methodologicall) 38(1)554-59, 1976. ISSN 00359246. URL http::/wwwt
jstor. sttbor..rr//ttaaee/288835
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Icml, volume 1, pp. 609-616. Citeseer, 2001.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass proba- bility estimates. In Proceedings of the eighth ACM SIGKDD international conference Knowl- edge discovery and data mining, pp. 694 699, 2002.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 1710:1711009411 09412, 2017.
11

---

Under review as a conference paper at ICLR 2021
APPENDIX
A PROOFS
A.1 PROOF THEOREM 1
Theorem For any Model M X - (Y - [0, and given canonical calibration mapping
R(p) = P[[M(X)](X) & pl. o M is quantile calibrated
Proof: To show that Ro M is quantile calibrated, need show that () =p Vp E [0, 1] since we are assuming that R (p) is invertible function, which gives us that it is surjective. Vp [0, 1]
An equivalent way of showing this is that [P[ (R M) [X][Y] s R(D) R (p)
P (Ro M][X][M] & R(p)
IP R- ((R2((RR (M((M)])))))) s (
R is strictly increasing
= -P (M[X)) s
R(())
By definition
A.2 PROOF OF CLAIM 1,2,3
R F be its CDF after composing Claim Let Y be random variable with CDF F, and let G = with mapping R obtained from isotonic regression characterized by = (c ... '(2) there exist i - 1, i1 E (0.m) AC()) ((i1) + ((4+)) C(s) then the CDF is not differentiable and its corresponding probability density function is not continuous at (c(s)
((m)) }- If
Proof: First, G can be expressed as follows
F(x).
-00 ~00 <<<
(c)()
mC(1)
F(x) ((1)
1
(-)()))) < < F
m(((2) C(1))
F(x) C(2)
G(x)
2
F-((()) < x F- (c(s)
m((33) ((2)
F(x) C(m-1)
m-1
F ((m-m))))
<I F F
C(m) G(n-1)
Let FF We will show that G not differentialle at Similarly show that it is not differentiable at the other m switching points.
The left derivative is as follows
F(x)
F(a)
lim G(x) G(a)
m.c(1)
m.C(1)
F(x) F(a) F'(a) mc(1)
lim
lim mc(1)
T. a
m-
T a
I a
12

---

Under review as a conference paper at ICLR 2021
The right derivative is as follows
F(x) ((1)
+ 1 1
G(z) G(a)
m(((2) ((1))
lim x-aa
lim x-aa
x a
F(x) F(a)
F'(a) m.(((2) c(1))
lim x-a
m. m.(((() ((L))
I
Hence G is not differentiable.
Although the CDF is not differentiable at only a finite number of points, we can still get the PDF by piece-wise differentiation.
[(x)
-00 <<< F
mc(a)
f(x)
F (()())) < <x F
m(c(2) G()) -()))
f(x)
g(x)
((2) < < (
(10)
m(c(s) -(())
f(x)
F F (c(m-))) < < F FF
m2((C() C(m-1))
Now consider for any 1, + 1 m0 s. 6--1 # c++1 C.. Let (c: then
f(x)
f(a)
lim g(x) lim
m(c(s) 9(-1))
m(c(i) C(i-1))
f(x)
f(a)
lim g(x) xmaat
lim
x-aa
m m((++1) - ((9) m(((+1) (())
Since the right limit and left limit do not coincide and by construction of point we have that limit does not exist and therefore g(x) not continuous
Note that, most of times, the hypothesis is satisfied, so the smoothness is lost.
Claim Let Yiso be transformed random variable after applying isotonic mapping R random variable Y. Then the expectation of Yuo is as follows
m S(F-1 (((+)))-
[V = p
n
(G(+1) c(())
Proof: Assume that, before transformation, the random variable is distributed X ~ NNH, so exp -(x 2
f(x)
2
202
13

---

Under review as a conference paper at ICLR 2021
I. z.g(z))r
1
(a(t))
f(x) dx
2.
(l(c(i+1) 9(i)
(-1()
(o(tt)
-(x -(x- [1)
I. V2T.02
exp
2.0² dr
n(C(++1) - c())
F (-1())))
1
(((++1))
xx p.)
(x M H1)
exp
dx
n(c(i1)) c())
F-1()())
V
2.0²
1
1
((((41))
-(x 11
(x- 11)
exp
dæ
m(c(+1) - -(())
F-((e())
V2T
2.02
(((++1))
(x 11)²
exp
dx
using linearity
(011+1)-((1)
(-(())))
V2T.022
2.0²
-(x - 11)
F-1 (1611))
exp
m(c(++1) (())
12T.022
2.0²
(=f))(c()
m(c(++1) (()) F(F F(F (((
using def of edf
1=0
m
2 f(F (c(+t))) f(F ((F (c(o))
1 F(F (c(+1))) ((F'((c))
+
(c(++1 - -(t))
m
(G(+1) c(3)
1=0
f(F f(F- (c(4a))) f f(F-f(((e (c())
(G(+1) ((3))
1=0
Lemma Let Jpor Fy F-1 be density, distribution, and quantile functions, respectively, of the normal distribution with mean and std a. Then
fuo F-1 n.
(Fpo (3o))
00 Suo.00 (yo)
Proof: We use the following three properties of normally distributed random variables
1. Fg.(()) Fo,1(2) *=
2. fn.a (v) = fo, (1=
3. F (p) a. a.Fott (p) n
14

---

Under review as a conference paper at ICLR 2021
fu.o Frdd (Foo Fo (vo)) = SM.
Find [o..
1 140 ))
by using (1)
Ou
= -p.
P, 1
M4 )) 11
by using (3)
00
-Sp
+ /1
F F(x)
0
y/
Jo.1
00
by using (2)
yo
14
f0.1
0g
Jo
1
fu,1
Mul and Div by
00
00
10o.00 (16)
by using (2)
Claim 3
m
(1+1)((++) (i)((i)
E EYel = 12
m
C(++1)
C(3)
i=0
Proof: We first re-sustituttee c(s)
(M(+1)), then using the above claim, substituting
that S(F (c(o)) = ((F=
lim f(x) =
x -00
m
f(F (18-((9(11)) ((F
11
m
(9(+1) - -(0)
f(F (CC--((PPaaaa (1 - f(F (Fa (1(++1)))
(c(i+)) - -(1)
(1)P(1)
m ((+1)P((+1) (i)P(i)
by using ((F=1 (c(o))) 0, ((0) =
m
C(1)
G(++) C())
(1+1)P(++1 (()
( = 0, = 0, c()) =
((++) - c())
1=0
A.3 PROOFS OF CLAIM 4 AND CLAIM 5
Claim Let M be any regression model. Then M is perfectly quantile calibrated iff
(L(1MX))(Y)IIU) =
Proof:
Claim 5 Let fy.o be the marginal distribution of Y and (Fyn.o. = (M]n= = be the model's
predicted cumulative distribution for X, then if f P (p) = Spaia (Frz. (v)) Va, Vp
[0, 1), we have that M is Quantile Calibrated.
15

---

Under review as a conference paper at ICLR 2021
Proof:
1x MMMM X=x (0)-
| dp
P [M[oiMl =x) sr] -fx (x(x)dd
[ dp
p P
M p] fx((x)d
[x dp
d P p((r ( (x)xddr
/ dp
d
F Fi
mo Wi)] fx(x)dz
[ Sn
1
(F Maiaz (p)]
Speia (F fx (x)dr
1x
Since we have that M [XX][[]]Uniformmmpp 1] we can conclude that M is Perfectly Quantile Calibrated.
B IMPLEMENTATION DETAILS
B.1 CALCULATING NEGATIVE LOG LIKELIHOOD AFTER ISOTONIC CALIBRATION
Note that we have analytical expression for the updated density function g (see Eq. 10 Given a test input Fres, predicted CDF F, corresponding PDF f and target Yeea, now we describe the procedure to calculate likelihood for yhest after isotonic calibration i.e., (Btest) Recall that ((s) 's are ordered x-coordinates of recalibration dataset. Now since F (cx-)) I x F (c()) implies that 1)--1) S Fxx) < C(s) as monotonic. We can then do binary search for finding the correct s.t 1(1-1) & F(z) c() and scaling appropriately. This summarized below
μl, MODEL(SIes)
2: Fy Fn.a (Shess)
fy -fuo
4: C = [0.0, C(1), ((I), C(2).
C(m)
5: i BINACHSEARCH(C, Fÿ)
if Fy S C(m) then
fz 7: f n=(co 7
else
9: f
10:
end if
11: RETURN fulo
B.2 CALCULATION OF TRUNCATION POINT
The above algorithm clearly elucidates why there is possibility of assigning likelihood at test time after Isotonic Calibration as discussed in Sec. 3 F F(iee)) > C(m) then we have that g(tee)) = 0. So truncation point is Ytra = FF - (C(m))) Then the support of random variable is reduced from (-00, +o0) to (-00, Mmmn] so for every point in (8mmn 2o) model assigns zero likelihood, which is extremely undesirable. A simple way circumvent this proposed in the discussion in Sec.
16

---

Under review as a conference paper at ICLR 2021
C EXPERIMENTS
C.I BAYESIAN LINEAR REGRESSION
For Bayesian Linear Regressionm we sklearn's (Pedregosa al. 2011 Bayesian Ridge Regres-
sion implementation.
C.2 UCI-EXPERIMENTS
C.2.1 EXPERIMENTAL RESULTS AFTER POST-HOC CALIBRATION
Heteroscedastic Dropout
Dataset
Calibration Error(%)
RMSE
Maximum Likelihood
base+iso
QR + iso
iso
QR+iso
base +iso
QR+iso
Air Foil
15.96 + 2.08
10.00 +: 1.73
3.61 + 0.06
4.03 + 0.06
917.33
412.33
Boston Housing
29.80 + 4.91
23.26 5.11
4.52 + 0.19
4.55 0.09
115.08
6748.73
Concrete Strength
34.41 + 3.06
21.54 6.37
8.70 = 0.09
9.18
238.10
52.91
Fish Toxicity
1.39
0.92 + 0.01
0.94
2044.42
661.17
Kin8nm
0,26
0.18
+
0.00
29798.21
33123.68
Protein Structure
3.08
0.05
4.63 0.02
4.00
423.61
299.68
Red Wine
3.08 + 0.63
1.71+ 0.43
0.65 + 0.00
0.65
27011.34
973.92
White Wine
4.24
2,92 + 0.55
0.73 + 0.00
0.00
2523.27
9664.37
Yacht Hydrodynamics
12.13 +: 3.54
23.65 10.96
3.87 + 0.22
668.81
1020.40
Year Prediction MSD
0.04 + NA
0.02 + NA
9.00 = NA 9.02 + NA
15.64
15.64
Table 4: base+iso is DropoutVI without Quantile Regularization and after isotonic calibration and
QR is when Base Model trained with Quantile Regularization and after isotonic calibration. RMSE stands for Root Mean Square Error. Maximum Likelihood represents maximum of the like- lihoods among test time points
Deep Ensemble with Adversarial Training
Dataset
Calibration Error(%)
RMSE
Maximum Likelihood
base+iso
QR + iso
base + iso
QR+iso
base +iso
QR+iso
Air Foil
38.26 3.62
21.0 4.77
3.12 + 0.03
3.34 : 0.07
873.89
560.88
Boston Housing
52.05 2.40
30.81 3.77
4.83 + 0.13
4.47 0.10
164.63
221.09
Concrete Strength
50.10
31.11
2.44
9.00 + 0.08
8.53 : 0.15
24956.94
13536.36
Fish Toxicity
0.34
1.60 0.17
0.92 - 0.00
0.92 1 0.00
15522.65
10141.10
Kin8nm
0.10
0.52 0.02
0.07 0.00
27399.41
83444.29
Protein Structure
0.07
0.04 0.00
4.33 +
1723.68
1733.38
Red Wine
1
2.38 0.42
0.70 0.00
0.65 + 0.00
30918.93
19673.26
White Wine
19.40 + 0.45
5.22 0.32
0.77 + 0.00
0.73 + 0.00
78359.57
30925.28
Yacht Hydrodynamics
81.48 + 10.89
44.50 1 4.62
3,47 + 0.18
3.73 1 0.17
149.63
196.68
Year Prediction MSD
0.07 + 0.00
0.05 + NA
8.68 + NA
8.68 4 NA 5.65
6.18
Table 5: base+iso is Deep Ensemble without Quantile Regularization and after isotonic calibration and QR+iso is when Base Model is trained with Quantile Regularization and after isotonic calibra- tion. RMSE stands for Root Mean Square Error. Maximum Likelihood represents maximum of the likelihoods among test time points
17

---

Under review as a conference paper at ICLR 2021
C.2.2 CALIBRATION PLOTS
GCLB
(a) Airfoil
(b) Boston
(c) Concrete
(d) White Wine
Figure 3: Dashed line (y=x) indicates perfect Calibration. The closer to dashed line, the better
18

---

Under review as a conference paper at ICLR 2021
(a) Airfoil
(b) Boston
SCLB
Red
(d) Yacht Hydrodynamies
Figure 4: Dashed line (y=x) indicates perfect Calibration. The closer to dashed line, the better
19

---

Under review as a conference paper at ICLR 2021
C.3 VARYING QUANTILE REGULARIZATION PARAMETER
In the following sections, we show how quantile regularization parameter L affects both calibration and Root Mean Square Error (RMSE) for dropout-VI Sec. and for deep ensembles in
Sec. C.3.2
We do so by fixing spacing value to k vn, where n. is batch-size and varying L
C.3.1
FOR DROPOUT
24
On X-axis have Calibration Error (%). On Y-axis have the QR-reg parameter Each curve with same color represents calibration error for a particular dataset as we vary QR- reg parameter L for (1, 2,,2,3,
Figure 5:
Spacing value is set k vn. Here model is Dropout. Datasets are divided into two groups based on the scale of Calibration Error, which useful for viewing the plots. On left, plot for (airfoil, boston, concrete, yacht, kin8nm). On right, plot is for (fish, red, white, protein).
4,5] 3,,
Enden
1g pENmANLL
0king parmate
Figure 6: On X-axis we have Root Mean Square Error (RMSE). On Y-axis we have that QR-reg pa- rameter Each curve with same color represents RMSE for a particular dataset as we vary QR-reg parameter for {1. 2, 3, 4, 4.5) Spacing value is set to h = vn. Here the model is Dropout. Datasets are divided into two groups based on the seale of RMSE which useful for view- ing the plots. On left, plot is for (airfoil, boston, concrete, yacht, protein]. On right, plot is for (fish, red, white, kin8nm).
20

---

Under review as a conference paper at ICLR 2021
C.3.2 FOR DEEP ENSEMBLES
06.ng puRTTESe
Figure 7: On X-axis we have Calibration Error (%). On Y-axis we have the QR-reg parameter L. Each curve with same color represents calibration error for a particular dataset as we vary QR-reg parameter L for {1, 3, 4, 5] Spacing value is set to k = v m. Here model is Deep Ensemble. Datasets are divided into two groups based on the scale of Calibration Error which useful for viewing the plots. On left, plot is for (airfoil, boston, concrete, yacht, kin8nm). On right, plot is for {fish, red, white, protein).
Figure 8: On X-axis we have Root Mean Square Error (RMSE). On Y-axis we have the QR-reg parameter L. Each curve with same color represents RMSE for particular dataset as we vary QR-reg parameter L for {1, 2, 4, 5). Spacing value is set to /m. Here the model Deep Ensemble. Datasets are divided into two groups based on the scale of RMSE, which is useful for viewing the plots. On left, plot for (airfoil, boston, concrete, yacht, protein }. On right, plot is for (fish, kin8nm, red, )
21

---

Under review as a conference paper at ICLR 2021
C.4 VARYING SPACING VALUE k
In the following sections, we show how spacing value k affects both calibration error and Root Mean Square Error (RMSE) for dropout-Vi VI in Sec. [C...Jand for deep ensembles in Sec. C.4.2 We do so by fixed Quantile Regularization parameter L and varying spacing to different multiplies of n..
C.4.1 DROPOUT-V VI
Figure 9: On X-axis have Cailibration Error (%). On Y-axis have the spacing multi- plier Resultant Spacing value for spacing multiplier t is h = t.yn. Each curve with same color represents Cailibration Error (%) for particular dataset as vary spacing multiplier / for (1,2,3, 4, Here the model Dropout. Quantile Regularization parameter is fixed at L = 1. Datasets are divided into two groups based on the scale of Cailibration Error (%), which useful for viewing the plots. On left, plot is for {airfoil, boston, concrete, yacht }. On right, plot is for (fish kin8nm, red, white.protein.year)..
3.8
Figure 10: On X-axis we have Root Mean Square Error (RMSE). On Y-axis we have the spacing multiplier& t. Resultant Spacing value for spacing multiplier t is k = t.vn. Each curve with same color represents RMSE for particular dataset as we vary spacing multiplier for {1, 3, 4, 5} Here the model is Dropout. Quantile Regularization parameter is fixed at L = 1. Datasets are divided into two groups based on the scale of RMSE, which useful for viewing the plots. On left, plot for {airfoil, boston, concrete, yacht, pratein.year). On right, plot is for {fish, red, white, kin8nm).
22

---

Under review as a conference paper at ICLR 2021
C.4.2 DEEP ENSEMBLES
18aceg marglert
Figure 11: On X-axis we have Cailibration Error (%). On Y-axis we have the spacing multi- plier Resultant Spacing value for spacing multiplier is k t.vn. Each curve with same color represents Cailibration Error (%) for particular dataset as we vary spacing multiplier t for (1,2,3,4, 2, Here the model is Deep Ensemble. Quantile Regularization parameter fixed at L= 5. Datasets are divided into two groups based on the scale of Cailibration Error (%). which is useful for viewing the plots. On left, plot is for {airfoil, boston, concrete, yacht }. On right, plot is for {fish kin8nm, red, shite..rotein..earr.
3.4
Figure 12: On X-axis we have Root Mean Square Error (RMSE). On Y-axis we have the spacing multiplier f. Resultant Spacing value for spacing multiplier is l t.vn. Each curve with same color represents RMSE for particular dataset as vary spacing multiplier t for {1, 2, 5, Here the model is Deep Ensemble. Quantile Regularization parameter fixed at Datasets divided into two groups based the scale of RMSE, which is useful for viewing the plots. On left, plot is for {airfoil, boston, concrete, yacht, protein.year). On right, plot for (fish, red, white, kin8nm)
5.
23

---

Under review as a conference paper at ICLR 2021
C.5 ISOTONIC CALIBRATION WITH K-FOLD CROSS VALIDATION
Note that, in the UCI experiments, we used the training data as was done in Kuleshov et al. (2018). In this sections this we consider Isotonic Calibration with K-fold cross validation. It is summarized as below. FM denotes the final model after recalibration that is used at test time.
1: Initialize K Models, M.. M
Get K Training data sets, TrtinData [1] ..... TrainData(A]]
3: Get K calibration data sets, CalibrationDatase// II. CaltbrationDatsse(A K
5: for each do
6:
M = TRAIN M[i], TraianaaaDDatt(i)
7: R = GET_IS _IO (M[i), CatiaaationDataset(i]]
8:
end for
K
FM
R o M
Heteroscedastic Drooput VT
Dataset
Single Model
5-fold Cross Validation
base
Iso
Base with 5-fold
Iso with 5-fold
Air Foil
13.15 + 1.92
15.96
+ 2.08
9.74 + 0.51
11.93 + 1.05
Boston Housing
21.35 + 4.89
29.80
4.91
22.45 = 0.45
20.69 = 1.54
Concrete Strength
25.78 1 2.01
14.41::30
22.68 1 2.04
26.09 1 0.83
Fish Toxicity
3.23 1 0.39
1.39
1: 0.13
3.83 1 0.23
1.45 1 0.20
Red Wine
3.72 - 0.29
3.08
0.63
2.84 0.24
2.01 + 0.15
White Wine
4.30
+ 0.48
3.12 + 0.21
3.07 + 0.21
Yacht Hydrodynamics
29.52 3.36
34.88 + 1.59
14.83 + 0.68
Table 6: Base Model is Dropout-VI model without Quantile Regularization and QR is when Base Model is trained with Quantile Regularization. '5 fold cross validation' column considers the pro- cedure described above.
Deep Ensemble with Adversarial Training
Dataset
Single Model
f-fold Cross Validation
base
Iso
Base with 5-fold
Iso with 5-fold
Air Foil
24.61 = 2.31
38.26 + 3.62
15.74 + 0.89
18.35 + 1.15
Boston Housing
37.76
4.55
52.05 + 2.40
33.74 + 2.50
22.17 + 1.26
Concrete Strength
37.27
2.94
50.10
2.26
32.67 + 1.53
31.21 + 0.77
Fish Toxicity
3.26 + 0.62
6.31 0.34
1.62 + 0.35
1.24 1.05
Red Wine
8.66 + 0.38
18.04 0.34
5.90 0.26
1.60
= 0.10
White Wine
8.30 + 0.75
19.40 + 0.45
0.17 6.55 + 0.17
3.52 - 0.18
Yacht Hydrodynamics
24.18 6.64
81.48 + 10.89
23.84 1.56
21.03 + 1.71
Table 7: Base Model is Deep Ensemble without Quantile Regularization and QR when Base Model is trained with Quantile Regularization. '5 fold cross validation' column considers the pro- cedure described above.
24

---

Under review as a conference paper at ICLR 2021
C.6 MONOCULAR DEPTH ESTIMATION
We scaled down the images to 115 x 153. We used batch size of 4 and learning rate of 5e 5 and trained it for 1500 epochs with adam optimizer with step decay of 4.
C.6.1 USAGE OF POOLING LAYERS
Instead of directly using Quantile Regularization suggested in Alg. added average pooling layer before computing entropy over cumulative density values. This is justified in principle because average of CDF functions again valid CDF. Note that this is serves purposes.
1. It exploits locality for better uncertainty propagation
2. It reduces the computational resources for computing loss
C.6.2 POST-HoC CALIBRATION RESULTS
FC-DenseNet57
FC-DenseNet103
Model
Calibration (
RMSE +
Calibration Error(%) + RMSE +
Base after Iso QR after Iso
1.39
12.01
11.02 0.99
12.26
0.87
13.91
12.67
Table 8: 'Base after Iso' represents Base model when composed with Isotonic Mapping. 'QR after Iso' represents QR model when composed with Isotonic mapping.
C.6.3 CALIBRATION PLOTS
Calibration plot for FC-DenseNet 57
Caibration plot for FC-DenseNet 103
Figure 13: Base Represents FC-DenseNet57 in Fig. 13a and QR represents FC-DenseNet 57 trained with Quantile Regularization. Base Represents FC- 1C3 in Fig. 13d QR represents FC- DenseNet103 trained with Quantile Regularization. Ideal line is x line which represents perfect calibration. The More closer the curve to diagonal line the better.
D DATASETS DIMENSION
We consider following datasets www.oodaaaaaaminnttaateeeaaaeeee ing 15),,13), Concrete Strength (1030,8),Fist Toxicity (908, Kin88m) (8192, 9), Protein Structure (45730, 10), Red Wine (1599, 12), White Wine (4898, 12), Yacht Hydrodynamics (308,6), year prediction MSD (515345.91)
AirFoil (1503,6), Bouston Hous-
25

---

Under review as a conference paper at ICLR 2021
(a) Calibration plot for FC-DenseNet 57
(b) Caibration plot for FC-DenseNet 103
Figure 14: Base+iso Represents FC-DenseNet57 in Fig. 14a after isotonic claibration and QR rep- resents FC-DenseNet 57 trained with Quantile Regularization after isotonic calibration. Base Rep- resents FC-DenseNet103 in Fig. 14b] and QR represents FC-DenseNet103 trained with Quantile Regularization. Ideal line is y line which represents perfect calibration. The More closer the curve to diagonal line the better.
26


Reviews:
[{"review_id": "r-iaTl4DaGw", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The manuscript discusses the side-effects (or drawbacks) of Isotonic regression and proposes an alternative approach for calibration in regression problems. The authors demonstrate the limitiation of Isotonoc regression such as nonsmooth PDFs and truncation of support under some constructions of the calibration dataset (line 2 in page 4) on a simple linear regression problem (last paragraph before section 4). On the light of these observations, they propose quantile regression (QR) which does not require an additional dataset.\n\nThe discussions on the shortcomings of isotonoc regression are valuable. However, it is not clear why the proposed approach QR should address these limitations. The main motivation of QR is based on pushing KL between the predicted cumulative density of the target value M(X)(Y) and the uniform distribution to 0. First, KL is not a symmetric metric so the authors should clarify why they use KL(M(X)(Y) || U) instead of KL(U || M(X)(Y)). Second, uniform distribution has a bounded support and Gaussian distribution, which is the main focus of the manuscript, has unbounded support. As a result, KL(M(X)(Y) || U) can suffer from division by 0. Third, I would like to see if the limitations of Isotonic Calibration discussed in Figure 1 are resolved by this approach.\n\nMy other concerns/questions about the manuscript are below:\n\n- I do not follow how truncation points in Fig 1b-c were computed. More discussion in text on these figures would be helpful.\n- In the first paragraph of section 2, the authors mention that they only consider Gaussian likelihood but in conclusion they claim that they propose a black-box calibration loss function. This confusion should be clarified. Indeed, Gaussian model was used in most of the claims so my question is how the claims in section 3 would be like if the true data distribution is not Gaussian.\n- I do not follow the relevance of the statement \"This way the time required is reduced from O(m) to O(1)\" following Claim 3 on page 4 to the main approach.\n- Is expectation in equation in section 4.1 over i ?\n- Why was m in Algorithm 1 set to square root of n?\n- I am surprised that the datasets picked in Table 1 have no overlap with the ones in Kuleshov et al., 2018.\n- In discussion section, the authors mention that Isotonic Regression suffers in case of smaller calibration datasets. Kuleshov et al., 2018 state that K-fold cross-validation as an alternative approach to using a separate calibration set. I wonder if such approach would be helpful for small datasets and such addition to discussion would be helpful.\n- Lemma 1 in supplementary material was not discussed in main text.\n\n===========================================================================\nUpdate after Authors' response:\n\n- I thank authors on their detailed response and clarifications. I increased my score based on their response.", "extra_scores": {}, "extra_text": {}}, {"review_id": "h1hPiRwH1Jw", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a novel method to produce probabilistic models whose predictive uncertainty is quantile-calibrated, without requiring the use of a calibration dataset separate from the training dataset, and with sometimes improved calibration error. \n\nThere are three major technical ideas in the paper: \n- sec4.0 interpreting the definition of quantile calibration as an equality between two CDF, of which one is of the uniform distribution\n- sec4.1 estimating differential entropy using sample spacing\n- sec4.2 overcoming the need for ordering in the recalibration set with NeuralSort\n\nThese ideas support the entire algorithm and have merit. The approach is novel, improves over the baseline defined by Kuleshov 2018. The advantage of not requiring a calibration set is definitively important, especially in the case of small datasets of course. This lends the paper applicability. I can figure this method making its way into mainstream libraries such as scikit-learn.\n\nThe experiments are sound. However, the table reporting experimental results is unconvincing due to inaccuracies: bold (which is not defined in the caption, hopefully not to escape contradiction) seems to be used when the average result is better, *irrespective of standard deviation/error bars*, contrary to standard practice, where a method is not said to be \"better\" when it does not beat the baseline with non-overlapping mean +/- one standard deviation intervals. The results on max likelihood are insufficient to support the claim made before claim 2 (on a purported weakness of isotonic regression), because eg in the case of table 1, only half of the UCI datasets exhibit a base+iso max likelihood which is definitely larger than the QR+iso case.\n\nThe exposition has a good structure, especially sec3 is useful in pointing out weaknesses of isotonic calibration. Sec4 relatively clearly exposes the main ideas behind the approach. Sec5, the experiment description, should be improved. At a finer level of detail, the text is not well edited, and is probably hard to access for somebody not familiar with the matter; despite knowing the background literature well, I found myself having to guess and resolve ambiguities too often to call this a \"clear exposition\".\n\n# Major errors, suggestions\n- sec2+2.2: define X in the introduction of sec2: why can we order x's in sec2.2 all of a sudden? I didn't expect there to necessarily be a total order relation on X. Does the entire paper only apply when X can be ordered?\n- claim 3: what is F when indexed with $\\mu, \\sigma$ ? is it the Gaussian CDF, errf ? is it the same as $\\Phi$ in Algo 1 (undefined there as well)?\n- eq between eq5 and eq6 : this is probably wrong, should be m/(n+1) (also cf Learned-Miller 2003 eq 4)\n- eq6 should reference Learned-Miller 2003 eq 6 (the paper is long, you should reference the exact place to allow the reader to check)\n- sec4.2 I disagree with your argument for using the sample spacing entropy estimator: \"one has to use beta kernel\": you seem to imply that there are only these two alternatives, and that the latter is unfeasible; you cannot make such a strong claim without proving that there are no other methods.\n- Algo 1 line 4 what is $\\Phi$ ? ; line 8 where does this come from (mentioned nowhere else)? refer to Learned-Miller 2003 ?\n- text below Algo 1: so is CL the same as $\\hat{H}$ in eq6? why use a new notation?\n- table 1: the second column is repeated between top and bottom rows, with one error: the second ocurrence of (kin8nm, QR) is wrong (maybe incorrectly copied while editing the table, is equal to (kin8nm, QR+iso)) -- are the rest of the results reliable?\n- table 1: how is max likelihood obtained?\n- table 1 and 2: The text does not explain what the case without $\\lambda$ stands for -- this seems important to analyse the results however.\n- sec5.3 The meaning of \"monotonicity constraint\" is unclear -- what is the argument?\n- sec5.3 last sentence: Why is this a reason for a reduction in RMSE, NLL ? This is unclear.\n\n# Minor errors, typos and language issues\n\nThe text numerous syntax errors and typos. In a few instances, they obscure the meaning of a few sentences, which is not acceptable. \n\n- \"confidence interval\" is used inaccurately throughout the text\n- sec1 \"three main notions of calibration\", but I see only 1) post hoc and 2) implicit methods\n- sec3: basic interpolation: you mean linear interpolation?\n- claim 2: what is f ?\n- fig 1 is useful, but definitely needs labels on all axes.\n- table 1 and 2: why duplicate two columns between top and bottom rows ?\n- table 1, time: seconds?\n- usage used\n- after it has shown that\n- intervals improves\n- assigning zero likelihood: to what ?\n- wide range of architectures: duplicated\n- call...classification *as* canonical\n- 2.0 last sentence: rewrite\n- O(m),.\n- set *as*\n- per say -> per se ??\n- in many of cases\n- after claim 3: remedy is to use: do you mean \"is to add\" ?\n- the model this r.v. \n- fig 1: (A) instead of (a)\n- which is model is quantile-calibrated\n- is general that it does not\n- sec5.3: \"but again it is negligible\": why \"again\" ?\n- reference Gal 2016 is incomplete", "extra_scores": {}, "extra_text": {}}, {"review_id": "PVSp9_q1D4o", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper discusses a new calibration mechanism for regression models which produce better model prediction and uncertainty estimates. In Section 3, the paper first discusses some properties and drawbacks of the approach based on isotonic regression in Kuleshov et al., 2018 which uses a post-hoc calibration dataset after model fitting for calibration. Section 4 discusses a new approach based on regularization to achieve implicit regression calibration during model training instead of using a post-hoc processing approach. Section 5 is devoted to experimental results supporting the theory outlined in Sections 3 and 4.\n\nReview: I think the approach to achieve implicit regression calibration by adding a suitable regularization to the loss function objective is novel and a good contribution. Also the experimental results can be reproduced as the code has been included as part of the submission.\n\nBut I do have certain concerns on the results in Section 3 and the overall presentation. I am a bit confused by the discussion in Section 3. The proof of claims 1 and 2 seems to define the c.d.f. $F$ based on the assumption that $P(y|\\boldsymbol{x})$ is homoscedastic, while the discussion at the beginning of Section 3 assumes a heteroscedastic model with different $\\sigma_k$’s for different datapoints in the calibration dataset. I think the proofs can still hold with a slight modification by assuming the c.d.f. $F$ is for a standard Gaussian distribution by suitably accounting for different $\\mu_k$’s and $\\sigma_k$’s but, in general, I think the proofs and results can benefit from precise definitions and assumptions of the various quantities like the c.d.f. $F$, p.d.f. $f$ and how they depend on data $\\boldsymbol{x},y$.\n\nAs for the overall presentation, I think the paper can benefit from adding concise introductions to differential entropy estimation and the NeuralSort method from Grover et al., 2019 which are important concepts in the implicit calibration method discussed in Section 4. The material in section 4 seems to one of the major contributions of the paper and as such I think more space should be devoted to it. The optimization of loss function equation 7 is not discussed as also the selection and sensitivity of the hyperparameter $\\lambda$ in the loss function. Also it will be easier if the various mathematical symbols are cleanly defined before use in the main paper. For example, the symbol $F$ seems to be have different definitions in Section 3 and Section 4?\n\nIn the experimental section, I am missing the description of the QR + iso model. Also I will like to see more discussions on the increase in RMSE in Tables 1 and 2 for the QR model which is attributed to the loss of sharpness in the degenerate case outlined in Section 4.3 in the paper. For example, the RMSE seems to be impacted more for the Dropout VI model compared to the deep ensemble model. What factors impact the decrease in sharpness and how sensitive is the decrease of sharpness to the choice of hyperparameter $\\lambda$? \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4tyZRo2rsZh", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors consider the problem of learning quantile calibrated regressions model. A probabilistic regression model is a model that, given an input, outputs a distribution over possible scalar values. A quantile calibrated model is one is such that, for all quantiles $p$, the probability over $X, Y$ that the model predicts $Y$ is in the $p$th quantile is equal to $p$. Machine learning models are not usually calibrated after standard training, so the authors consider a regularization approach to improve the calibration of the model during training.\n\nThe authors first discuss an existing approach from (Kuleshov et al., 2018) where we first learn a calibration function $R(p)$ that describes the probability over $X, Y$ that the model on $X$ assigns a quantile $p$ to $Y$. If we had this function, then we can compose it with the model to get accurate quantiles. Kuleshov et al. uses isotonic regression to fit the function $R$.\n\nThe authors improve using isotonic regression for calibration when the probabilistic regression model outputs Gaussian distributions. Specifically they show that the mean calculation of the calibrated model can be done in constant time, rather than in O(calibration set) size.\n\nThey then argue that isotonic regression has some disadvantages, specifically (1) it requires an additional step of training on a hold out dataset and (2) it creates PDF distributions that can be spiky and noisy.\n\nThey then note that if the calibration function $R$ is uniform then the probabilistic regression function is already calibrated. They then directly regularize the probabilistic regression function with a KL-divergence term between $R$ and a uniform distribution. They estimate KL-divergence by using a differentiable entropy estimator. This avoids the two-stage calibration function and allows us to directly use the learned probabilistic regression function, which may have a more convenient form than the compose regression function.\n\nThey experimentally show using UCI datasets and small neural networks that their approach leads to a more calibrated model, potentially at the expense of accuracy. \n\nOverall I found the paper interesting and the main idea principled with decent experimental results. I would have liked to see experiments on larger scale networks and more challenging datasets, as they may behave differently. Additionally I would have liked to see how accuracy and calibration trade off based on the penalty parameter.\n\nCan you explain why adding isotropic regression in the experiments doesn't always improve the calibration over base?\n\nAlso, could you use a nearest neighbor entropy estimator rather than a sorting-based one (see, e.g., Gai et al., 2016, https://arxiv.org/pdf/1604.03006.pdf)? This is directly differentiable and would avoid needing a relaxation to sorting.\n\nThe paper could be improved for clarity. There are many seperate parts to the paper (results improving Isotonic calibration, issues with Isotonic calibration, KL-regularization, degenerate calibrated models). I think the paper can be improved by making these sections more self-contained and by removing results that are not the main focus of the paper (such as degenerate calibrated models).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a novel method for calibrating probabilistic regression models without requiring a held-out calibration set. The technical advances are interesting, and the experimental results look promising. The authors made a number of improvements based on the reviews, and the authors have done a good job with reproducibility of the experiments. Nevertheless, it ultimately does not meet the bar for acceptance. In revising the article for a future submission, I would encourage the authors to emphasize the observation in Section 4.3 that overall we want models that are both "well-calibrated and sharp". This helps motivate the method and will help the reader interpret the results. 
Decision: Reject