Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification
Authors: Youngseog Chung
Willie Neiswanger
Ian Char
Jeff Schneider
Venue: NeurIPS 2021 Poster (2021)
Keywords: uncertainty quantification
calibration
pinball loss
quantile regression

Abstract: Among the many ways of quantifying uncertainty in a regression setting, specifying the full quantile function is attractive, as quantiles are amenable to interpretation and evaluation. A model that predicts the true conditional quantiles for each input, at all quantile levels, presents a correct and efficient representation of the underlying uncertainty. To achieve this, many current quantile-based methods focus on optimizing the pinball loss. However, this loss restricts the scope of applicable regression models, limits the ability to target many desirable properties (e.g. calibration, sharpness, centered intervals), and may produce poor conditional quantiles. In this work, we develop new quantile methods that address these shortcomings. In particular, we propose methods that can apply to any class of regression model, select an explicit balance between calibration and sharpness, optimize for calibration of centered intervals, and produce more accurate conditional quantiles. We provide a thorough experimental evaluation of our methods, which includes a high dimensional uncertainty quantification task in nuclear fusion.

Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification
Youngseog Chung Machine Learning Department Carnegie Mellon University Pittsburgh, PA 15213 youngsec@cs. cmu edu
Willie Neiswanger
Department of Computer Science
Stanford University Stanford, CA 94305
neiswanger&cs. stanfotd..deeeooomm
Ian Char
Jeff Schneider
Machine Learning Department Carnegie Mellon University
Robotics Institute
Carnegie Mellon University
Pittsburgh, PA 15213 icharêcs. cmu edu
Pittsburgh, PA 15213 schneide@cs. cmu edu
Abstract
Among the many ways of quantifying uncertainty in regression setting, specifying the full quantile function is attractive, as quantiles are amenable to interpretation and evaluation. A model that predicts the true conditional quantiles for each input, at all quantile levels, presents a correct and efficient representation of the underly- ing uncertainty. To achieve this, many current quantile-batt based methods focus on optimizing the pinball loss. However, this loss restricts the scope of applicable regression models, limits the ability target many desirable properties (e.g. cali- bration, sharpness, centered intervals), and may produce poor conditional quantiles. In this work, develop new quantile methods that address these shortcomings. In particular, we propose methods that can apply to any class of regression model, select explicit balance between calibration and sharpness, optimize for calibra- tion of centered intervals, and produce more accurate conditional quantiles. We provide thorough experimental evaluation our methods, which includes high dimensional uncertainty quantification task in nuclear fusion.
Introduction
Uncertainty quantification (UQ) in machine learning typically refers to the task of quantifying the confidence of a given prediction. This measure of certainty can be crucial in a variety of downstream applications, including Bayesian optimization [36, 45, 54], model-based reinforcement learning [64, 42, 10, 18]. and in high-stakes predictions where mistakes incur large costs [52, 61].
While the common goal of UQ is to describe predictive distributions over outputs for given inputs, the representation of the distributional prediction varies across methods. For example, some methods assume parametric distribution and return parameter estimates [65, 14, 37), while others return density function estimates, as is common in Bayesian methods [41, 38, 24, 3, 33, 50]. Alternatively, many methods represent predictive uncertainty with quantile estimates [15, 53, 58, 49, 27].
Quantiles provide an attractive representation for uncertainty because they can be used to model complex distributions without parametric assumptions, are interpretable with units in the target output space, allow for easy construction of prediction intervals, can used to efficiently sample from the predictive distribution via inverse transform sampling [22, 32]. Learning the quantile for single
Code is available at https://ggithe wtt.aai..cooooooooeeeeeee hwt..//wwwwgeeeoccggg//////////e
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

quantile level is a well studied problem in quantile regression (QR) [30, 32], which typically involves optimizing the so-called pinball loss, tilted transformation of the absolute value function. Given target y/ prediction D, and quantile level E (0, 1), the pinball loss Pr is defined as
Pr (v. 1) (ū u) (L(v S 0) -)
(I)
By training for all quantiles simultancously, recent works have made concrete steps in incorporating QR methods to form competitive UQ methods which output the full predictive distribution [51, 58].
In this work, we highlight some limitations of the pinball loss and propose several methods to address these shortcomings. Specifically, we explore the following:
Model agnostic QR. Optimizing the pinball loss often restricts the choice of model family for which we can provide UQ We propose an algorithm learn all quantiles simultaneously by utilizing methods from conditional density estimation. This algorithm is agnostic to model class and can be applied to any regression model.
Explicitly balancing calibration and sharpness. While the pinball loss, as proper scoring rule, targets both calibration and sharpness, the balance between these two quantities is made implicitly, which may result in poor optimization objective. We propose tunable loss function that targets calibration and sharpness separately, and allows the end-user to set an explicir balance.
Centered intervals. In practice, we often desire uncertainty predictions made with centered intervals, which are not targeted via the pinball loss. We propose an alternative loss function that is better suited for this goal.
Encouraging individual calibration. Perfect quantile forecasts will satisfy individual calibration (Eq. 2), which is a much stricter condition than the more-commone used notion of average calibration (Eq. 3). We introduce training procedure that aims to improve quantile predictions beyond average calibration, and demonstrate its efficacy via adversarial group calibration (Eq. 4).
We proceed by first describing methods of assessing the quality of predictive UQ and the pitfalls of optimizing the pinball loss in Section Drawing motivation from this, we then present our proposed methods in Section 3. In Section 4, we demonstrate our methods experimentally, where we model predictive uncertainty on benchmark datasets, and on a high-dimensional, real-world uncertainty estimation task in the area of nuclear fusion.
Preliminaries and Background
We first lay out the notation, terminology, and class of models considered in this paper. Then we provide an overview of evaluation metrics in UQ and demonstrate how the pinball loss may be inadequate both as an evaluation metric and as an optimization objective.
2.1 Notation
Bold upper case letters X, Y denote random variables, lower case letters x, y, denote their values, and calligraphic upper case letters X, y denote sets of possible values. We input feature vector and y E y to denote the corresponding target. Additionally, we consider the regression setting where y C E and X C I", We use Fxx 1y Fy to denote the true cumulative distribution of the subscript random variable. For any x E X. we assume there exists a true conditional distribution Fv over )i, and we assume (x()) denotes the true pd quantile of this distribution,
E X' to denote the
iee. F = p. Any estimates of the true functions F. Qp will be denoted with a hat, F, Q We will specifically refer to any family of estimates for Qp. with p E (0, 1), as a "quantile model", denoted Q : X' x (0, 1) + y. Unless otherwise noted, we will always consider the conditional problem of estimating quantities in the target space y, conditioned on a value z E X.
(Qp(z))
2.2 Assessing the Quality of Predictive UQ
While various metrics have been proposed assess the quality of UQ, there has been a great deal of
recent focus on the notions of calibration and sharpness [15, 13, 65, 60, 55, 35, 21, 20]. We introduce calibration here, but for more thorough treatment, see Zhao et al. [65]. Broadly speaking, calibration in the regression setting requires that the probability of observing the target random variable below predicted 4 quantile equal to the expected probability p. for all p E (0, 1). We refer to the former quantity the observed probability and denote it pobs (p), for expected probability p, which we

---

2x10-1
3x10
Valls: 250
50n
Call
Ex: 420
10.9
Cel Gelbraton alon Losa
Calbration Error
0.25
0.20
Shargess 0225
0.075
SQR
Call
True Starsteed
Call Vil Eillpp (6.10)
Son val Ep (350)
0.15
10.4
0.10
0.025
6x10-
0.05
0.000
200
400
500
800
200
400
600
800
200
460
600
800
Train Epoch
Train Epoch
Train Epoch
(a) Test Loss Curves
(b) Test Calibration
(c) Test Sharpness
Figure I: (a) Test loss continues to decrease until the validated epoch. (b-c) At the validated epoch, SQR (optimizes pinball loss) is highly miscalibrated while sharper than the true sharpness level. Cali (optimizes proposed calibration loss) is better calibrated while less sharp than the true sharpness.
will write as p when it is clear from context. Calibration requires (p) = p. Vp E (0. 1). From this generic statement, we can describe different notions of calibration based on how yobs is defined.
A model is individually calibrated if it outputs the true conditional quantiles, i.c. @x(x) @z(x). In this case, we define the observed probability to be
obs Pendo (p. x) == FYl1z (Q)())), Ae Vp (0,
In words, this requires that the probability of observing y below the quantile prediction is equal to at each point x E X, individually. If we can verify this property for all x E X, then by definition, we will know the quantile output is correct and precisely the true conditional quantile. However, individual calibration is typically unverifiable with finite datasets in the assumption-less case [65].
relaxed condition is average calibration. In this case, we define the observed probability to be
Pang (p) Ez~xx Fr (Q (x)),, Vp (0, 1),
i.e. the probability of observing the target below the quantile prediction, averaged over Ex, is equal to p. Average calibration is often referred to simply as "calibration" [13, 35). Given dataset D = ((x,, yt) | we can estimate pany (p) with ( =
+ E T(y S @x(x:)). Note that if our quantile estimate achieves average calibration then Paig + N + cc, Vp E (0, 1). The degree of error in average calibration is commonly measured by expected calibration error [60, 13, 21]. ECE(D, Q m ΣØ=
(D,p)
Pats (D.p,) - -Pyl. where ~ 1n).
It may be possible to have uninformative, yet average calibrated model. For example, quantile predictions that match the true marginal quantiles of Fy will be average calibrated, but will hardly be useful since they do not depend the input x. Therefore, the notion of sharpness is also considered, which quantifies the concentration of distributional predictions [20]. For example, for non-parametric predictions, the width of centered 95% prediction interval is often used as a measure of sharpness. There generally exists tradcoff between average calibration and sharpness [20, 46).
Recent works have suggested notion of calibration stronger than average calibration, called ad- versarial group calibration [65]. This stems from the notion of group calibration [23, 29), which prescribes measurable subsets S; C X s.t. PaoBy (x: E S,) > 0, i = 1,..., k and requires the predictions to be average calibrated within each subset. Adversarial group calibration then requires average calibration for any subset of X with non-zero measure. Denote Xs as random variable that is conditioned on being in the set S. For adversarial group calibration, the observed probability is
pobs (p) Ex-Fxy (y)x(((())) Vp 10, VS C X s.t. PE-E, S) > 0.
(4)
With finite dataset, we can measure a proxy of adversarial group calibration by measuring the average calibration within all subsets of the dataset with sufficiently many points.
Intuitively, individual calibration inspects the discrepancy between pahs and p for individual inputs x E X. adversarial group calibration relaxes this by inspecting any subset X' with non-zero measure, and average calibration relaxes this further by considering the full distribution of X.
One alternative family of evaluation metries proper scoring rules [19]. Proper scoring rules are summary statisties of overall performance of distributional prediction and consider both calibration
3

---

and sharpness jointly [20]. For example, negative log- log-likelihod (NLL) is proper scoring rule that commonly used with density predictions [14, 48, 37J. For quantile predictions, one proper score is the check score, which is identical to the pinball loss. Since proper scoring rules consider both calibration and sharpness together single value, they can serve as optimization objectives for UQ. For example, optimizing the pinball loss is the traditional method in quantile regression [31], and many recent quantile-based UQ methods focus on optimizing this objective [51, 58, 8, 63].
In this work. however, we note that the balance between calibration and sharpness implied by the pinball loss is arbitrary and depends on the expressivity of the model class-and with highly expressive models, this balance can be heavily skewed towards sharpness. In their seminal work on probabilistic forecasts, Gneiting and Raftery [19] contend that the goal of probabilistic forecasting is to "maximize the sharpness of the predictive distribution subject to calibration", i.e. calibration should be first achieved and then sharpness optimized. We show that common machine learning methods that use the pinball loss objective may in fact lead to an arbitrary and miscalibrated UQ-
Proposition 1. Consider a finite dataset D. the pinball loss Pr (Eq. 1) and a quantile model f x (0,1) + y that is average calibrated on D, i.e. EC = O. Then there always exists another quantile model g : X (0,1) + V, such that, for any quantile level E (0, (0,1), g has lower pinball loss than f on D, i.e. Ei=1 Pr (ys. (x.)) < EN EE Pr fr but worse average calibration than f. i.e. ECE(D, g) > fC)
Proof: The proof is given in Appendix A.1.
This proposition essentially states how the pinball loss can become detached from calibration, and we show its practical ramifications via a synthetic example in Figure 1 (experiment details in Appendix E. We first note in Figure (a) and (b) that even while the pinball loss decreases on the test set, test calibration worsens (while sharpness improves). Further, at the best validation epoch, optimizing the pinball loss converges to a solution that is sharper than the true noise level. Note that a UQ that is sharper than the true noise level will never be calibrated (meanwhile, a less sharp prediction can still be calibrated, e.g. the marginal distribution Fy). While this may seem like an issue that can simply be addressed with regularization, we demonstrate Appendix E.2 how that is not the case. These pitfalls motivate our methods in Section 3.
3 Methods
We propose four methods that aim to produce an improved quantile model. The first is a model- agnostic procedure that relies conditional density estimation (Section 3.1). To address settings where density estimation may be difficult, we then propose two loss functions to optimize with differentiable models: the combined calibration loss (Section 3.2), which directly optimizes calibra- tion and sharpness, and the interval score (Section 3.3), which is a proper scoring rule for centered intervals. Finally, propose group batching method (Section 3.4) that can be applied to the batch optimization procedure for any loss function (c.g. combined calibration loss, interval score, and even pinball loss) to induce better convergence towards adversarial group calibration.
3.1 Utilizing Conditional Density Estimation for Model Agnostic QR
One drawback of many existing quantile- UQ methods is that their training procedure requires differentiable models. In fact, most UQ methods require a specific class of models because of their modeling structure or their loss objective (e.g. Gaussian processes [50], dropout [17], latent variable models [33], simultancous pinball loss [58], and NLL-based losses [37]). This model restriction can be especially unfavorable in practical settings. A domain expert with an established point prediction model and compute infrastructure may want to add UQ without much additional overhead.
To address these issues, we can consider the following model- agnostic procedure. Instead of optimiz- ing a designated loss function, we can consider splitting the given problem into two parts; estimate conditional quantiles directly from data, then regress onto these estimates. The benefit of this method that, granted we can estimate the conditional quantiles accurately, we can use any regression model to regress onto these quantile estimates. Further, this regression task directly targets the goal of producing the true conditional quantiles (i.e. individual calibration). This procedure, which we refer to as Model Agnostic QR (MAQR), is outlined in Algorithm 1.

---

Algorithm MAQR
1: Input: Train data (Ii, 1/s j=1 trained re-
Algorithm 2 CONDQUANTLESSSTIMATTS
gression model f(z)
1: Input: Dataset ( & point index &
2: Calculate residuals
yi f(z)), E
[N]
2: Ex,xx f {e dist(Tk; S dN, [N]}
[N], and denote the residual dataset R
{ri; G i=1
3: Initialize D Ø
3: Construct empirical CDF with Ek,dy to
produce 1F Elax :<>> E [0, 1]
4: for 1 to N do
CONDQUANTILESESTIMA 5: D; &
4: Initialize D -- 13
TORS(R, i) (Algorithm 2)
5: for each Cj in Ex do
6: + DUD;
6: Pk. f FE (Ez)
7: end for
7: + DU (TK, PR.
8: Use D to fit a regression model 9
8: end for
9: (I. p) e
9: Output: D
9: Output: 9
MAQR is based on the key assumption that nearby points in X' will have similar conditional dis- tributions, i.e. if zjj Vg then Fy|z, Fylz. Given this smoothness assumption, neighboring points to estimate the conditional density at each locality over X. with locality deter- mined by the hyperparameter dy (Algorithm 2, line 2). We then construct an empirical CDF with the group of neighboring points, and conditional quantile estimates are produced with this empirical CDF. These estimates are collected into (Algorithm line 6), which ultimately used as the training set for the quantile model ÿ.
group
In practice, we perform these steps with residuals, by first estimating a mean function f (Algorithm 1, line 1). This practical choice stems from existing works conditional density estimation, which suggests that having conditional mean in the data provides benefits in terms of lower asymptotic mean squared error the conditional density predictions [26]. Further, this demonstrates how MAQR can be readily applied in the application setting where an accurate point prediction model often already exist.
Algorithm is a specific implementation more general model-agnostic algorithm, in which we directly estimate conditional quantiles from the data with tools from conditional density estimation. We note that using KDEs for conditional density estimation is a well studied problem with theoretical guarantees [25, 26, 57]. In the case the distance in X' is measured using uniform kernel with mild assumptions on the bandwidth, Algorithm falls under the guarantees stated by Stute et al. [57].
Theorem [57]. Assume y C R, C R", dist(z;, x.)) = ==, IjJoo, and that FL is constructed using the procedure given in line 5 of Algorithu I (i.e. D; = x). Further assume that, N co, dy + 0 and that ΣN21 exp(-pNd's) < oc, Vp > 0. Then, as N - oc. for almost all x E X, sup, Fu (c) FEZ (c) + with probability
This theorem states that in the limit of data, for almost all x E X, the CDF estimate FE will converge uniformly to the true CDF F =z= with probability 1. The dataset, D. will therefore be populated with good estimates of the conditional quantile and quantile level pair for x. In Appendix B, we state the general form of Algorithm and also demonstrate how the algorithm is model agnostic. Through experiments in Section 4, we will show empirically that utilizing these density estimates sidesteps the issues inherent the pinball loss and produces much higher quality quantile predictions.
3.2 Explicitly balancing calibration and sharpness with the combined calibration loss
While MAQR can produce strong results, its performance can suffer in digh-dimensional settings, where nonparametric conditional density estimation methods falter. Neural networks (NNs) have shown good performance in high dimensional settings, given their high capacity to approximate complex functions and recent advances in fast gradient-based optimization. We therefore propose loss-based approach estimating conditional quantiles for NNs and other differentiable models.
5

---

Drawing motivation from the arbitrary balance between calibration and sharpness that pinball loss implicitly provides, we propose objectives separately for calibration and sharpness, Then, we combine the two objectives into single loss function that provides an explicit balance between calibration and sharpness that can be chosen by the end user.
We first consider calibration of quantile prediction, Q2 E y for quantile level p E (0, (). Here, we omit conditioning on x: for clarity. For this prediction to be average calibrated, exactly a proportion of the true density should lie below Qp- i.e. Pagz P(Y I Qp) p. While calibration (e.g. paae pp is non-differentiable objective, by inducing a truncated distribution based the current level of calibration, we can construct the following calibration objective, which is minimized if and only if the prediction is average calibrated:
C(Op. p) = 1p p} E
0pl L 2p1 P(Y Qp)
(5)
+ 16 > p} E - Y 11u, Y] P(O, > Y), where py = P(Y & Qp).
The empirical calibration objective, C(D, Qp p), is then defined as follows:
N
C(D.Q.P) = [(Paos < <p}
[vi - (x(o)) > Qx((s))
N
(6)
+ Pe
> p)
Σ -.) m1
N
Note I: Intuition of the calibration objective. For any given p. consider the case when the quantile estimate Qp is below the true ndh quantile Qp. Since Qpp < Qp - Py p. this implies that too much data density lies above Qp. In this case, C(pp. p) reduces to EE Q,1Y > @pl * PP > Qp). @p is pulled higher with the expectation of the truncated distribution that places Qp at the lower bound of the support. In the opposite case, when Q > Qp @, is pulled lower by the same logic.
Note 2: Is the proposed calibration objective a proper scoring rule? Strictly speaking, the calibration objective is a non-decomposable function, hence deviates from the standard convention of proper scoring rules [19], which can be "decomposed" into scores for individual examples (I., y4). This simply arises from the fact that measuring average calibration (i.e. Paty) is non-decomposable. Proper scoring rules are defined such that an optimum of the expected score (or risk, if we consider the score loss function) occurs the true distribution quantity. While an example level loss score does not exist due to non-decomposabilityy we can still show the (expectation-level) score (i.e. C(Qp. p)) is minimized by the true distribution and hence enjoys the optimum property of proper scoring rules.
Proposition 2. For any quantile level p (0, 1). the true quantile function Op minimizes the calibration objective, C(Qp.P) p)- Further, on a finite dataset D, the empirical calibration objective, C(D, Qp. p), is minimized by an average calibrated solution on D, i.e. when Pary (D.p) p) = p.
Proof: The proof is given in Appendix A.2.
Note 3: Non-zero gradients for miscalibrated predictions Qp- We can further show that for a miscali- brated quantile prediction, the gradients of C are always non-zero. When P < p. ac p)/a@r -PY )p) < 0. Thus increasing Qp decreases the objective C. Similarly, when Py > p. = P(Y < Q > 0, and an analogous argument follows (proof in Appendix A.3).
As discussed in Section 2.2, average calibration by itself is not a sufficient condition for meaningful UQ hence we also desire sharp quantile models, with more-concentrated (less dispersed) distributions. We can induce this property in quantile predictions by predicting the (1 - 2h quantile Q1- alongside each prediction (p())) and penalizing the width between the quantile predictions:
P(Qp.P) p) = TE [ 2--
(7)
The empirical sharpness objective, P(D, Qp. p). is then defined as follows:
N Q1)p(z)) (x::)) (p s 0.5) P(D, N Σ (x)) Q1)((.)) (p> 0.5).
(8)

---

It is important to note that the true underlying distribution will not have 0 sharpness if there significant noise, and sharpness should be optimized subject to calibration. Therefore, we should only penalize sharpness when the data suggests our quantiles are too dispersed, i.e. when pany (p) Pary (1 p) the observed coverage between the pair of quantiles (xs) and (1-)(z:).
is greater than 12p 1 the expected coverage.
Combining the calibration and sharpness terms, we have the combined calibration loss
L(D, L(D, = (1 A)C(D, + XP(D, Qp.
The hyperparameter À E [0. 1] sets the explicit balance between calibration and sharpness. Note that setting À= O may not always be desirable, since optimizing C(D, Qp: alone may converge to quantiles of the marginal distribution, Fy Further, in certain downstream applications that utilize UQ. sharper prediction, even at the cost of worse calibration, may result in higher utility, and À can be tuned according to the utility function of the application. In our experiments, we tune À by cross-validating with adversarial group calibration as it is the strictest notion of calibration that can be estimated with a finite dataset. Since we learn a quantile model that outputs the conditional quantile estimates for all probabilities, our training objective is Ep- C(D, Qp.P). p).
3.3 Encouraging calibration of centered intervals with the interval score
The combined calibration loss (Eq. 9) optimizes average calibration, which targets observed probabili- ties below a quantile. In many applications, however, we may desire a centered prediction interval (PI) which requires a pair of quantile predictions. A centered 95% PI, for example, is a pair of quantile predictions at quantile levels 0.025 and 0.975. Hence, for the average calibration of the nd centered interval, we want 1 + 5 - Pare (0.5 5- 5) (the PI's observed probability, a.k.a. prediction interval coverage probubility (PICP) [58, 28, 491) to be equal to the expected probability p. While can modify the objective in Eq. 9 to adhere to this altered goal, here we propose simultaneously optimizing the interval score (or Winkler score) [19, 62] for all expected probabilities E (0, 1), and bring to light proper scoring rule that has largely been neglected for the purpose of learning quan- tiles. While some previous works utilize the interval score to evaluate interval predictions [7, 4, 1, 40), to the best of our knowledge, no previous work has focused simultaneously optimizing it and shown a thorough experimental evaluation as we provide in Section 4.
For a point (Ir y). if we denote a (1 - a) centered PI as û, i.c. i= Q4 (z) and à = Qi interval score is defined as S. S., ûr y) (ū i) + 20 y) < (v i) >> a]. We show in Appendix A.4 that the minimum of the expectation of the interval score is attained at the true conditional quantiles, i = Qg (.), â = Qi (.). We train our quantile model for all centered intervals (and hence all quantile levels) simultaneously by setting our loss as Eo-Dni((.1))aa
4 (x), the
3.4 Inducing adversarial group calibration with group batching
The calibration loss (Section 3.2) and the interval score (Section 3.3) optimize for the average calibration of quantiles and centered intervals, respectively. To get closer to individual calibration, one condition we can additionally require is adversarial group calibration. Since adversarial group calibration requires average calibration over any subset of non-zero measure not fully observable with finite datasets for all subset sizes. However, for any subset in with enough datapoints, we can still estimate average calibration over the subset. Hence, can apply optimization objectives onto appropriately large subsets to induce adversarial group calibration.
the domain, this is
In practice, this involves constructing subsets within the domain and taking gradient steps based on the loss over each subset. In naive implementations of stochastic gradient descent, a random batch is drawn uniformly from the training dataset D, and gradient step is taken according to the loss over this batch. This is also the case in SQR [58]. The uniform draw of the batch will tend to preserve Fx (the marginal distribution of X), hence optimizing average calibration over this batch will only induce average calibration of the model.
Instead, deliberately grouping the datapoints based on input features, and then batching and taking gradient steps based on these batches, induces better adversarial group calibration. We find in our experiments that adversarial group calibration improves significantly with simple implementations of group batching, and in Section 4.2, we show through an ablation study that group batching can improve average calibration and adversarial group calibration of SQR as well.
7

---

SQR
PATC
Intereal
Cali
MAQR
Concrete
9.3 L517. L.)
0.2.0.5 0.8.
3.7 A 0.8018.1 10.6)
5.6 5.1108
13)
5.3 0.4.0.160000 104
26
0.210
22
10:10111111
Wine
4.2 122122
1.3 1.0.)
10.3
5.0 5.
4.2 26.00 26.00
2.7 27100091103
UCI
Yachit
9:10090000001
10.=
0.6 1.0)
0.3 0.4)
6.8 6.8=2121=00
Naval
L.63.550 0.4)
2.2
1.7
4366)
2.9
102)
2.3 2
Energy
9.8 1.8.2 101))
10.4 0.5 102)
4.8
0.9)
5.8 1.4,36 0.33
3.5 1.032 0.1)
Boston
9.0.0. 0.7)
17=1312310.7)
6.9
0.5)
1.
KinNume
4.4 0.1(14 0.2)
1.4.0.017.:000 0.5)
2. 0.4 1.1.910000 0.5))
3.5 0.3,
0.7]
1.8 0.4
Enengy
50%
2x
2
Hieev
3x
2x-30-
2x
10-1
W6-8
2.0
1.0
12:0
0.5
1.2
1.0
CrAD 3x
4x
0.0
0.3
4x
bx
0.3
Group Stote
Group 3uc
Group
Group
Figure 2: UCI Experiments. (Top Table): Average calibration (measured by ECE) and sharpness in parentheses. The best mean ECE for each dataset has been bolded and the best mean sharpness has been underlined (all values multiplied by 100 for readability). (Bottom Figure): Adversarial group calibration for the first 5 UCI datasets (full set of results in Appendix D.1). Group size refers to proportion test dataset size.
SQR
mPAIC
Interval
Cali
aminor
5.8 0.9(1.6 0.0)
13.5 0.015.3 0.0)
3.6 0.7(3.7 + +.1))
2.9 0.2 0.0)
betan
3.1
0..1
5.1 0.1
3.4 0.2)
dasdenest
0.5...
8.1.0.(13.770..
2,9+0.3,33330... 0...
0.
0.3(2.4
4.4=0.(5.40...
0.2 0.1)
Fusion
kappa
14810319010.4)
42.0466.1000
3.610.2(44
1261:0.02.70.01 0.11
3.220.222.0.11
2.9
0.8.3.3
(77047722033)
0.6 3.2.0.26200.)
0.3)
tribot
4.6 0.4(2.4 1.11)
1 0.7)
3.9 0.5(6.0 1 0.2)
4.6
0.500 0.2)
tritup
5.6 0.7(2.7 1011)
1.910.7 1.77)
3.7 0.8(6.5 104)
2.5
3.3
volumee
5.8 1 1.2(0.9 1.00)
16.9 1 1.19.9 L.4)
3.6 10.1)
2.8 1
0..00.1221111101 (1210.0)
desdenest
kappa
Cabraton Eror Sropp
20
508
10
Inava
6
4x:0--
2x
3x
1x
3x110-1
Group Stet
Group Suc
CHAMp Sec
Group Stot
Group Sux
Figure 3: Fusion Experiments. (Top Table): Average calibration (measured by ECE) and sharpness in parentheses. The best mean ECE for each dataset has been bolded and the best mean sharpness has been underlined (all values multiplied by 100 for readability). (Bottom Figure): Adversarial group calibration for the first fusion datasets (full set of results in Appendix D.2). Group size refers to proportion of test dataset size.
To summarize, the main idea we introduce here with group batching is that, only taking uniform batches from the training set (thus only drawing batches which preserve Fx) can be detrimental when optimizing for calibration. Thus, additionally drawing batches based on deliberate groupings within the training set (thus, batches which do not preserve Fx) can help to induce a stronger notion of calibration (i.e. adversarial group calibration) in the model than average calibration. This concept is quite general and allows for variations in implementations when constructing the groups. In Appendix C.3 we provide details on how we implemented group batching for our experiments and ablation study.
Experiments
We demonstrate the performances of our proposed methods on the standard 8 UCI datasets [2], and on a real-world problem in nuclear fusion. assess the predictions, we use all metrics from Section 2.2 that can be estimated with finite test dataset: 1) average calibration vs sharpness, 2) adversarial group calibration, 3) centered interval calibration, 4) check score, and 5) interval score. The results for the first two of these metrics are displayed in the main body of the paper, and the results for the
8

---

Cali
SQR
Random Batch
Group Batch
Random Batch
Group Batch
Boston
9.7 1.3.10.2 = 0.7)
8.5 166=15 15.9 0.6)
10.9 + 1.0 = 1.1)
9.8 + 1.2.9.5.
Concrete
6.6 0.9 = 2.3)
0.610. + 1.5)
1.$1.3.000 = 0.6)
7.1 = 0.9.
Energy
9.2=0.3 0.11 0.1)
0.3.6
0..2
6.9 1.144
Kin8nm
3.3 1 0.4)
3.5 1 0.3 1.7))
1. 1.11)
3.9 0.4,11.. 0.2)
W Wine
4.4 0.5/2.5 1 0.8)
4.2 = 0.4[26.0 1 10.8
4.5 1 0.4[29.7 1..5)
4.0 1 0.4(28.5 0.8)
Yacht.
11.1 1.1.1.33 1.8 0.1)
8.3 1 0.6 0.4)
9.0 1 0.9[0.9 0.1)
8.9 1 0.9(2.3 1022)
Power
0.22(1. 1 L.3)
2.0 0.1(13.1 1 L.1))
2.5 0.3(14.0 1 0.5)
2.9 1 0.5(13.6 - 0.8)
Naval
2.8 1 0.2
1 3.1)
2.4 - 0.3(50.6 8.6)
8.6 1.6(366 1 0.1)
5.3 1 0.8(660 1.55)
Figure 4: Group Batching Ablation: Average Calibration and Sharpness. The table shows mean ECE and sharpness (in parentheses) and their standard error with and without group batching. The best mean ECE for each dataset has been bolded and the best mean sharpness has been underlined for Cali and SQR separately. All values have been multiplied by 100 for readability.
other three metrics are shown in Appendix D due to space restrictions. We describe how each of these evaluation metrics are calculated in Appendix C.4.
We provide comparisons against current state-of of-tte--rttttt UQ methods for which computing the above metrics is tractable. SQR [58] is an NN model that optimizes the pinball loss for batch of random quantile levels p ~ Uni 1). mPAIC [65] is an extension of probabilistic neural networks [37, 47] that optimizes a combination of the standard Gaussian NLL and loss that induces individual calibration. While additional quantile and PI based UQ methods exist, they are mostly designed to output a single quantile level or interval coverage level, which makes measuring calibration extremely expensive (training up to 100 models separately). For these methods, we provide a comparison on simplified task of predicting the 95% PI in Appendix E.3. Lastly, could also consider the recalibration algorithm of Kuleshov et al. [35], which not a standalone UQ method, but a post-hoc refinement step that can be applied on top of other methods. We discuss recalibration results in Appendix E.4. All results report the mean and error across 5 trials. Error bars and shaded bands in plots indicate #1 1 standard error.
4.1 UCI and Fusion Experiments
UCI datasets: We evaluate 5 methods on the 8 UCI benchmark datasets: three proposed algorithms- MAQR, Cali (combined calibration loss), and Interval (interval score) and 2 alternative algorithms- SQR and mPAIC. Appendix C.1 includes more details on the experiment setup and hyperparameters.
Fusion datasets:
We further evaluate the methods on high-dimensional task from nuclear fusion:
quantifying uncertainty in plasma dynamics. Recently, there has been increasing interest in applying machine learning to prediction and control tasks in the area of nuclear fusion [11, 9, 433. The plasma data in this paper was recorded from the DIII-D tokamak, a nuclear fusion device operated by General Atomies [39]. Plasma dynamics during fusion reactions are highly stochastic and running live fusion experiments is costly. Hence, practitioners use this dataset to learn a dynamics model of the system for various purposes, such as controller learning to optimize reaction efficiency and stability [16. 5, 6]. There are 10 scalar target signals to model in this dataset (full list in Appendix C.2). each describing& a particular aspect of the current state of plasma. For each signal, the input features are 468 dimensional. We do not apply MAQR on this dataset because of the high computational costs and statistical challenge associated with nonparametric density estimation in high dimensions. Appendix C.2 provides more details on the dataset, experiment setup, and hyperparameters.
Figure 2 displays average calirrationssaappess for all 8 UCI datasets and adversarial group calibration for the first 5 UCI datasets (alphabetical order). MAQR produces the best average calibrated models on 7 of the 8 UCI datasets, and adversarial group calibration also indicates that MAQR tends to achieve the lowest calibration error across any random subgroup of any size with more than one point (6 out of datasets). Excluding MAQR, Interval and Cali achieve competitive average calibration and adversarial group calibration on 7 out of 8 UCI datasets. Notably, SQR tends to produce the sharpest predictions across all datasets (often at the cost of worse calibration), which is not surprising following the discussion in Section 2.2. Lastly, we also note that Interval and Cali tend to less brittle compared to SQR and mPAIC, which incur major failures in some cases (c.g. mPAIC in Kin8nm, SQR in Naval).
Analysis of Results:
9

---

The fusion experiment results display similar pattern (Figure 3). Interval and Cali achieve competi- tive average calibration and adversarial group calibration on 9 out of 10 fusion datasets, and SQR produces the sharpest UQ at the cost worse average and adversarial group calibration (except on betan). We also observe that mPAIC performs the poorest on the fusion datasets, with least calibrated and least sharp predictions. It is generally known that plasma dynamics display complex stochasticity [16, 34], and hence we suspect mPAIC's performance degrades significantly because assumes a Gaussian output and is trained according to the Gaussian likelihood.
Only subset of the plots and metrics are shown in the main paper due to space restrictions. Readers are encouraged to see the full set of results in Appendix D, but we summarize the main findings here. The check score, interval score, and centered interval calibration the tables in Appendix D.1 also rank MAQR's prediction as best on the UCI datasets. This is surprssing since SQR and Interval train NNs of the same capacity to explicitly minimize the check and interval scores, respectively. This indicates the distribution predicted by MAQR is fundamentally different as it utilizes direct estimates of the conditional distribution, while the other methods all optimize a specific loss function. The centered interval calibration metric on the fusion datasets (Appendix D.2) indicates that Cali and Interval both produce competitive centered PI's (9 out of 10 fusion datasets), suggesting that both methods have generally estimated the conditional quantiles better than the alternative algorithms.
4.2 Ablation Study: Effect of Group Batching
We provide an ablation study on the effect of group batching (experiment details in Appendix F.I). Figure 4 displays how group batching affects average aalirration-sharpness of Cali and SQR all 8 UCI datasets, Average calibration improved on out of 8 datasets for Cali, and on 7 out of 8 datasets on SQR. While sharpness tends to worsen with group batching, it's important to note that less sharp (i.e. more dispersed) predictions are desirable if there is high noise in the true data distribution. In Appendix F.2. we show that group batching also improves adversarial group calibration, suggesting that the underlying data distribution truly does have high noise. This study indicates that deliberately taking batches during training that do not follow Fx can improve UQ performance.
Discussion
In this paper, we have proposed four methods to improve quantile estimates for calibrated uncertainty quantification in regression. We assert that the pinball loss may not be an adequate objective to optimize in order achieve calibration, and that our proposed methods provide better means of learning calibrated conditional quantiles. We have also extended the scope of regression models on which quantile-based UQ can be applied by developing model-agnostic method. This can be of practical interest to users that have specific training infrastructure or precxisting regression procedures, since these procedures can be leveraged to quantify uncertainty without much additional overhead.
By providing an extensive evaluation with suite of metrics, also aim show that there is not one single metric that captures full information about the performance of UQ procedure. Rather, a holistic review of multiple types of metrics sheds light on different aspects of performance. This point has motivated us to additionally develop and make publicly available Uncertainty Toolbox [12], a Python library for evaluation, visualization and recalibration of predictive uncertainty, which includes the full suite of metrics discussed in this work.
In certain applications, users may also be concerned with modeling epistemic uncertainty. This is somewhat orthogonal to the goals of this paper; nevertheless, we provide a discussion on how bootstrapped ensembles of our methods can incorporate epistemic uncertainty in Appendix G.
Acknowledgments
This work was funded in part by DOE grant number DF-SC0021414 and DF-AC02-76SFO0SIS.
Willie Neiswanger was supported in part by NSF (# ##1651566 (551) ONR (N000141912145), AFOSR (FA9550191007) 10024), ARO (W91INF-21---0125), DOE (DE-ACU2-76SFO00155 and Sloan Fellowship.
Ian Char was also supported by NSF grant DGE174501 Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
10

---

References
[1] Ross Askanazi, Francis Dicbold, Frank Schorfheide, and Minchul Shin. On the comparison of interval forecasts. Journal of Time Series Analysis, 39(6):5339900 2018.
[2] Arthur Asuncion and David Newman. Uci machine learning repository, 2007.
[3] Charles Blundell, Julien Comebiie, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. arXiv preprint arXiv: 05424, 2015.
[4] VE Bowman, DS Silk, U Dalrymple, and DC Woods. Uncertainty quantification for epidemi- ological forecasts of covid-19 through combinations of model predictions. arXiv preprint arXiv:2006. 10714, 2020.
[5] MD Boyer, KG Erickson, BA Grierson, DC Pace, JT Scoville, J Rauch, BJ Crowley, JR Ferron, SR Haskey, DA Humphreys, al. Feedback control of stored energy and rotation with variable beam energy and perveance on diii-d. Nuclear Fusion, 59(7):076004, 2019.
[6] MD Boyer, Kaye, and K Erickson. Real-time capable modeling of neutral beam injection on nstx-u using neural networks. Nuclear Fusion, 59650000 2019.
[7] Johannes Bracher, Evan L Ray, Tilmann Gneiting, and Nicholas G Reich. Evaluating epidemic forecasts in an interval format. PLoS computational biology, 7(2))e100861s 2021.
[8] Alex J Cannon. Non-crossing nonlinear regression quantiles by monotone composite quantile regression neural network, with application to rainfall extremes. Stochastic environmental research and risk assessment, 32(11))3207-322, 2018.
[9] Ian Char, Youngseog Chung, Willie Neiswanger, Kirthevasan Kandasamy, Andrew O Nelson, Mark Boyer, Egemen Kolemen, and Jeff Schneider. Offline contextual bayesian optimization. Advances in Neural Information Processing Systems, 32:4627-4638, 2019.
[10] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pages 4755-4765, 2018.
[11] Youngseog Chung, Ian Char, Willie Neiswanger, Kirthevasan Kandasamy, Andrew Oakleigh Nelson, Mark D Boyer, Egemen Kolemen, and Jeff Schneider. Offline contextual bayesian optimization for nuclear fusion. arXiv preprint MxXi::000100199: 2020.
[12] Youngseog Chung, Ian Char, Han Guo, Jeff Schneider, and Willie Neiswanger. box: open-source library for assessing, visualizing, and improving uncertainty quantification. arXiv preprint 1XXi:221091002550 2021.
Uncertainty tool-
[13] Peng Cui, Wenbo Hu, and Jun Zhu. Calibrated reliable regression using maximum mean discrepancy. Advances in Neural Information Processing Systems, 33, 2020.
[14] Nicki S Detlefsen, Martin Jørgensen, and Søren Hauberg. Reliable training and estimation of variance networks. arXiv preprint 1906.02090, 2019.
[15] Matteo Fasiolo, Simon N Wood, Margaux Zaffran, Raphaël Nedellec, and Yannig Goude. Fast calibrated additive quantile regression. Journal of the American Statistical Association, pages
1-11, 2020.
[16] Yichen Fu, David Eldon, Keith Erickson, Kornee Kleijwegt, Leonard Lupin-Jimenez, Mark D Boyer, Nick Eidietis, Nathaniel Barbour, Olivier Izacard, and Egemen Kolemen. Machine learning control for disruption and tearing mode avoidance. Physics of Plasmas, 27(2)022250, 2020.
[17] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059, 2016.
[18] Javier Garcia and Fernando Fernández. Safe exploration of state and action spaces in reinforce- ment learning. Journal of Artificial Intelligence Research, 45:515-564, 2012.

---

[19] Tilmann Gneiting and Adrian E Raftery. Strietly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):35-3778 2007.
[20] Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69 20:243-268, 2007.
[21] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1321-1330. JMLR. org, 2017.
[22] Lingxin Hao, Daniel Q Naiman, and Daniel Q Naiman. Quantile regression. Number 149. Sage, 2007.
[23] Ursula Hébert-Johnson, Michael Kim, Omer Reingold, and Guy N Rothblum. Calibration for the (computationally-ddenifaaaa masses. arXiv preprinl urXiv:/711.08511 2017.
[24] José Miguel Hernández- Lobato and Ryan Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In International Conference on Machine Learning, pages 1861-1869, 2015.
[25] Michael P Holmes, Alexander G Gray, and Charles Lee Isbell. Fast nonparametric conditional density estimation. arXiv preprint
2012.
[26] Rob Hyndman, David M Bashtannyk, and Gary K Grunwald. Estimating and visualizing conditional densities. Journal of Computational and Graphical Statistics, 344:3315-336, 1996.
[27] Soyoung Jeon, Christopher Paciorek, and Michael F Wehner. Quantile-based bias correction and uncertainty quantification of extreme event attribution statements. Weather and Climate Extremes, 12:24-3 32, 2016.
[28] HM Dipu Kabir, Abbas Khosravi, Mohammad Anwar Hosen, and Saeid Nahavandi. Neural network-based uncertainty quantification: survey of methodologies and applications. IEEE accesy, 6:36218-36243 2018.
[29] Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent trade-offs in the fair determination of risk scores. arXiv preprint
2016.
[30] Koenker. Quantile Regression (Econometric Society Monographs; No. 38). Cambridge univer- sity press, 2005.
[31] Roger Koenker and Gilbert Bassett Jr. Regression quantiles. Econometrica: journal of the Econometric Society, pages 33-50, 1978.
[32] Roger Koenker and Kevin F Hallock. Quantile regression. Journal of economic perspectives, 15(4):143-156, 2001.
[33] Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
[34] Grzegorz Kowal, Diego A Falceta-Gonqalves, Alex Lazarian, and Ethan T Vishniac. Kelvin- helmholtz versus tearing instability: What drives turbulence in stochastic reconnection? The Astrophysical Journal, 892(1):50, 2020.
[35] Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. arXiv preprint 1807.000) 2018.
[36] Harold J Kushner. A new method of locating the maximum point of arbitrary multipeak curve in the presence of noise. 1964.
[37] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances neural information processing systems, pages 6402-6413, 2017. 2017.
12

---

[38] Jeremiah Liu, John Paisley, Marianthi-Amn Anna Kioumourtzoglon, and Brent Coull. Accurate uncer tainty estimation and decomposition in ensemble learning. In Advances in Neural Information Processing Systems, pages 8950-8961, 2019.
[39] James L Luxon. A design retrospective of the diii-d tokamak. Nuclear Fusion, 42(5):614, 2002.
[40] Katarzyna Maciejowska, Jakub Nowotarski, and Rafal Weron. Probabilistic forecasting of electricity spot prices using factor quantile regression averaging. International Journal of Forecasting, 32(3): 2016.
[41] Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems, pages 13153-13164, 2019.
[42] Ali Malik, Volodymyr Kuleshov, Jiaming Song, Danny Nemer, Harlan Seymour, and Stefano Ermon. Calibrated model- based deep reinforcement learning, arXiv preprint 1rXiv:1906.083/2, 2019.
[43] Viraj Mehta, Ian Char, Willie Neiswanger, Youngseog Chung, Andrew Oakleigh Nelson, Mark D Boyer, Egemen Kolemen, and Jeff Schneider. Neural dynamical systems: Balancing structure and flexibility in physical prediction. arXiv preprint arXiv:2006. 12682, 2020.
[44] Nicolai Meinshausen. Quantile regression forests. Journal of Machine Learning Research, 7 (Jun):983-99)) 2006.
[45] Jonas Mockus, Vytautas Tiesis, and Antanas Zilinskas. The application of bayesian methods for seeking the extremum. Towards global optimization, 2(117-129):2, 1978.
[46] Allan H Murphy. A new vector partition of the probability score. Journal of applied Meteorology, 12(4):595,600, 1973.
[47] David A Nix and Andreas S Weigend. Estimating the mean and variance of the target proba- bility distribution. In Proceedings 1994 ieee international conference neural networks (ICNN'94) volume 1, pages 55-60. IEEE, 1994.
[48] Tim Pearce, Felix Leibfried, Alexandra Brintrup, Mohamed Zaki, and Andy Neely. Uncertainty in neural networks: Approximately bayesian ensembling. arXiv preprint arXiv: 180.0550 2018.
[49] Tim Pearce, Mohamed Zaki, Alexandra Brintrup, and Andy Neely. High-quality predic- tion intervals for deep learning: A distribution-free, ensembled approach. arXiv preprint arXiv:/802.077 0000007167, 2018.
[50] Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer School on Machine Learning, pages 63-71. Springer, 2003.
[51] Filipe Rodrigues and Francisco C Pereira. Beyond expectation: deep joint mean and quantile regression for spatiotemporal problems. IEEE Transactions on Neural Networks and Learning Systems, 2020.
[52] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence 1(5):206-21) 2256 2019.
[53] Tárik S Salem, Helge Langseth, and Heri Ramampiaro. Prediction intervals: Split normal mixture from qurvity-driven deep ensembles. In Conference on Uncertainty in Artificial Intelligence, pages 1179-1187. PMLR, 2020.
[54] Bobak Shahriari. Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104 (1):148-11) 175, 2015.
[55] Hao Song, Tom Diethe, Meelis Kull, and Peter Flach. Distribution calibration for regression. In International Conference Machine Learning, pages 5897-5906. PMLR, 2019.
13

---

[56] Ingo Steinwart and Andreas Christmann. Estimating conditional quantiles with the help of the pinball loss. Bernoulli, 17(1):211-22,, 2011.
[57] Winfried Stute et al. On almost sure convergence conditional empirical distribution functions. The Annals of Probability, 14(3):891-901, 1986.
[58] Natasa Tagasovska and David Lopez-Paz. Single-model uncertainties for deep learning. In Advances in Neural Information Processing Systems, pages 6414-6425, 2019.
[59] Ichiro Takeuchi, Quoc Le, Timothy D Sears, and Alexander J Smola. Nonparametric quantile estimation. Journal of machine learning research, 7(uu):1231-1264, 2006.
[60] Kevin Tran, Willie Neiswanger, Junwoong Yoon, Qingyang Zhang, Eric Xing, and Zachary W Ulissi. Methods for comparing uncertainty quantifications for material property predictions. Machine Learning: Science and Technology, (2):0250066 2020.
[61] Rebecca Wexler. When a computer program keeps you in jail: How computers are harming criminal justice. New York Times, 13, 2017.
[62] Robert L Winkler. A decision approach to interval estimation. Journal of the American Statistical Association, 67(337):187-191, 1972.
[63] Qifa Xu, Kai Deng, Cuixia Jiang, Fang Sun, and Xue Huang. Composite quantile regression neural network with applications. Expert Systems with Applications, 76: 129-139, 2017.
[64] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005. 13239, 2020.
[65] Shengjia Zhao, Tengyu Ma, and Stefano Ermon. Individual calibration with randomized forecasting, arXiv preprint arXiv:2006. 10288, 2020.
14


Reviews:
[{"review_id": "Uw-2gxrDEsw", "reviewer": "Reviewer_58KW", "summary": "The paper presents four alternatives to training quantile regressors with the vanilla pinball loss:\n\n(1) a specific conditional density estimator, from which the quantiles are derived\n\n(2) a new loss function that balance calibration and sharpness\n\n(3) centered intervals\n\n(4) group calibration to enforce conditional coverage\n\nExperiments are shown to compare these methods with SQR, a neural network that optimizes the pinball loss.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and the methodology is sound. In particular, I think that the new loss function (point 2) is very interesting, as well as the idea of group calibration (point 4).\n\nThe idea of deriving quantile estimators from conditional density methods (CD; point 1) has been around for some time. Thus, I miss comparisons of the conditional density estimator the authors propose to other conditional density estimators available in the literature. This is particularly relevant because, in almost all UCI datasets, MAQR was the best performing method, but it is also the only CD-based method. Thus, it is not clear to me that the good performance comes from using this specific conditional density estimator or if it is because of the fact that the problem of estimating the quantiles was being tackled in another way (namely estimating the whole distribution). \n\nI also have two concerns with the CD method:\n\n- The same data is being used twice: (i) to fit the regression and (ii) to model the distribution of the residuals. I wonder if regression methods that lead to overfitting (e.g., a neural network that interpolates the training points) would harm the quality of the estimator.\n\n- The methods depend on the Euclidean distance between the features, which not good in general (especially if isotropic): It is often the case that some covariates are independent of the Y conditional on the remaining ones. This is especially true in high-dimensions. Thus, I don't agree with the authors that all nonparametric conditional densities fail in high-dims (some methods do variable selection, for instance), but instead that this specific method is not appropriate.\n\n\nI also miss comparisons to other quantile estimators that don't directly target the pinball loss, such as quantile regression forests.\n\n\nMinor comment:\n\n- The results on the appendix for the Fusion datasets indicate that the Check score is minimized by SQR, so I don't really with the last statement from Section 4.1.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nYes, limitations have been addressed.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "KO9kdLZ_a6", "reviewer": "Reviewer_4TMr", "summary": "The authors propose a new quantile method called MAQR for uncertainty quantification. This method is based on estimating empirical CDFs for residuals locally and therefore suffers from the curse of dimensionality. To address the issue, the authors further propose an NNs based solution, where the proposed loss consists of a convex combination of two terms. The first term encourages average calibration and the second term encourages sharpness. Thus allowing the user to control the balance between the two. The paper also provides a theoretical result stating that the first term is indeed minimized by the true quantile function. The authors also propose a version of the method for estimating centered intervals as well.\n\nThe paper is concluded with a rather extensive empirical evaluation.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed methods are interesting and can be of interest to both researchers and practitioners interested in uncertainty quantification. The proposed methods are novel to the best of my knowledge - although I am not completely up to date with the literature.  The paper is dense, but overall well-written. However, some important definitions are not stated anywhere in the paper (see comments below). The claims of the paper are reasonably justified using a combination of theoretically motivated arguments and an extensive empirical study. The authors are comparing and evaluating the methods using several evaluation metrics. The main paper is also accompanied by a rather thorough appendix providing the details of the theoretical results as well as the experimental settings. \n\n\n- It is not entirely clear to me, why eq. (6) is a good way to encourage sharpness. In my opinion, the authors should elaborate on this.\n\n- Line 252: What is meant by: \"all expected probabilities p ∈ (0, 1)\" in line 252. Can the authors elaborate on that?\n\n- What is the exact definition of sharpness used in the paper? Sharpness is both for motivating the work as well as an evaluation metric, but the authors do not provide an explicit definition.\n\n- The MAQR method has a hyperparameter controlling bandwidth or locality, $d_N$, and one might suspect that the results can be quite sensitive to perturbations in this parameter. It would be nice if the authors would include a small discussion of the robustness of the methods.\n\n- The objective in eq. (7) has the hyperparameter lambda balancing sharpness and calibration. It is state that lambda is estimated using cross-validation based on the adversarial group loss. I am curious to know what were the optimal values for the different experiments? \n\n- The y-labels are missing for Figures 7 and 13 in the appendix.\n\n# After rebuttal\nI've updated my score from 6 to 7", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable", "review_text": "", "extra_scores": {"ethical_concerns": "Not applicable", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "E7jCQNt3sh", "reviewer": "Reviewer_ePFY", "summary": "This work is motivated by some shortcomings of the classical pinball loss used to learn quantile models, like the required differentiability of the models and miscalibration issues.\nThe pinball loss is also not naturally suited to build centered intervals.\nThis work proposes alternatives ways for learning quantile models, addressing these shortcomings.\nFour methods are proposed:\n- Model Agnostic QR: it is a generic method based on conditional density estimation and standard regression. Therefore it is not particularly tied to differentiable models and doesn't use the pinball loss.\n- Combined calibration loss: a loss to be used instead of the pinball one, with the advantage of explicitly balancing  sharpness and calibration of the predictions.\n- Centered intervals: an alternative loss based on the Winkler score is proposed to train models providing centered intervals.\n- Group batching: a heuristic replacing uniform sampling for batch-based based methods, aiming at better individual calibration.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## General aspects\n\n**Originality** \n\nAs far as I know, Proposition 1 is a new result that shows that the pinball loss doesn't guarantee calibration on a finite dataset.\nFour methods are proposed, with different degree of originality:\n- MAQR is a new quantile method leveraging existing conditional density methods like KDE and standard (mean) regression methods. \n- The combined calibration loss combines two new objectives addressing sharpness and calibration respectively. A new theoretical result shows that calibration objective is well founded (it is minimized by the true quantile function).\n- The centered interval  method leverages the existing interval (or Winkler) score. The novelty is in using it as training loss---instead of an evaluation metric---and in averaging it over all interval widths.\n- The group batching is a heuristic that consists in using some non-uniform batching criterion with the goal of approximating adversarial group calibration that is closer to the ideal individual calibration.\n\n\n**Quality**\n\nThe theoretical results are useful and the proofs seem correct.  \nThe experiments properly illustrate the benefits of the methods proposed.\n\nThe methods are interesting. The two first methods (MAQR and combined calibration loss) are treated in more depth and rigor  than the last two. In particular, the group batching method (Section 3.4) is vaguely described (some elements are given in the appendix but still in an informal way).\n\n\nI found the relationship with regularization not properly discussed. In particular, the assertion \"While this may seem like an issue that can simply be addressed with regularization, we demonstrate in Appendix F.2 how that is not the case.\" is too strong, since appendix F.2 shows  it  experimentally for SQR only, which has the particularity of taking the quantile level as input.\nA discussion of previous work relating regularization and calibration would be desirable :\n\n*I. Takeuchi, Q. V. Le, T. D. Sears, and A. J. Smola. Nonparametric quantile estimation. JMLR, 2006.*\n\n*I. Steinwart et al. Estimating conditional quantiles with the help of the pinball loss. Bernoulli, 2011*\n\nIt would be also interesting to compare with the following recent work that addresses the coverage problem:\n\n*Romano, Y., Patterson, E., & Candes, E. (2019). Conformalized quantile regression. Advances in Neural Information Processing Systems, 32, 3543-3553.*\n\n\n**Clarity**\n\nThe paper is well organized and well motivated. The group batching method not properly described, with some important elements being described in the appendix only. Algorithms 1 and 2 seem to contain some errors that make them confusing (see detail below).\n\n\n**Significance**\n \nThe methods and the theoretical properties presented in this work should be of interest for the statistical and machine learning communities working on quantile regression and uncertainty quantification in general, and for practitioners requiring predictions with uncertainty estimation.\n\n## Specific comments about the methods\n\n**MAQR**\n\nI think I got the idea from the text, but Algorithms 1,2 and 3 are confusing to me.\nThe idea seems to be (please correct me if I'm wrong):\n- Learn a standard regression model $\\hat{f}:x\\rightarrow y$, compute residuals $\\epsilon=y-\\hat{f}(x)$\n- Learn an empirical conditional CDF of the residual $\\epsilon$ for each $x_k$ using its neighbors (KDE)\n- Use it to compute the cumulative density $p$ of each neighbor $x$ and build a new observation $\\\\{x, p, \\epsilon\\\\}$\n- Learn a regression model $\\hat{g}: x,p \\rightarrow \\epsilon$ , which can be interpreted as quantile regression model since it takes the quantile level $p$ as input. \n- Use $\\hat{f}+\\hat{g}$\n\nAlgorithms 1 and 2:\nFor i=1 in Algo 1, the dataset D that is passed to Algo 2 seems to be empty.\nThe dataset of pairs that is taken as argument of Algo 2 seems to be the union of sets returned by itself on previous calls, but it is confusing that Algo 2 returns a set made of triplets .\n\nAlgorithm 1: why does it output k = 1, …, m ? What is m?\n\nLine 6 : why the preceding \" D: \" ?   (also in Algo 2) \n\nMaybe I am wrong because of the previous confusions, but it seems to me that the dataset that is used to train $\\hat{g}$ can be larger than the original dataset (because of the overlapping neighborhoods). This should be discussed.\n\nAlgorithm 3 looks simpler and it is not obvious how Algo1+2 is an instance of it.\nLine 5 of Algo 3: what does \" KDE estimate of Q(xi; pk)\" mean? Q is not a density.\n\nThe statement \"residuals … which produces more accurate empirical CDFs\" should be justified.\n\n\n**Combined Calibration loss method**\n\nThe empirical loss should be introduced and explained in the main text. \nIt would be interesting to report the values of lambda selected by the cross-validation procedure to see what kind of tradeoff is best for the considered datasets.\n\n\n**Centered Interval method**\n\nExpectation over alpha uniform on [0,1] can make  $\\hat{l}>\\hat{u}$  (lower bound > upper bound). Does it make sense?\n\nThis loss also combines sharpness and calibration by taking  into account the width of the interval $u-l$ and penalizing the points outside the interval. In the light of the previous section, I would expect some discussion on how these objectives are weighted.\n\n\n**Group batching**\n\n This is an interesting heuristic to add on previous methods and other batch based methods, by modifying how batches are built. Nevertheless, I found it not properly described. It is essentially described by the following sentence in the main text:\n\"Instead, deliberately grouping the datapoints based on input features, and then batching and taking\ngradient steps based on these groups, induces better adversarial group calibration.\"\nAppendix gives more information but remains very informal.\n\nMoreover: in appendix (lines 692-694), it is said that uniformly sampled batches are alternated with the adversarial ones without justification.\n\n\n## Minor comments\n\n\"Proper scoring rules\" should be formally defined in Section 2.\n\nTheorem 1: the random variable E deserves a more explicit definition.\n\nEq. 5 $\\hat{p_p}$ seems a duplicate notation for $p^{obs}_{avg}$.\n\nLine 215: occur -> occurs\n\nLine 347: \"While sharpness tends to increase with group batching, higher sharpness is desirable if\nthere is high noise in the true data distribution. \"\nI think this refers to the width of the 95% centered interval: higher = less sharp.\nHowever, \"higher sharpness\" sounds like the opposite.\n\nLine 592:  I found the following notation confusing :\n\n$$\\hat{p}^{obs}_{avg}(D,\\tau) \\text{ of } g - \\tau$$ \n\nMaybe it would be clearer to define $\\hat{p}^{obs}_{avg}(D,\\tau)$ of $g$ and $f$ before, using different symbols.\n\nLast part of the proof of Prop 1 (generalization to any tau ): The step from line 592 to 593 is clear since tau is part of the set {p_j} defining ECE and therefore the inequality compares  terms of each ECE. \nlines 599-601: you should explain more explicitly how you get to the same inequality between ECEs when tau is not part of {p_j} .\n\nLine 637: state -> stated\nLine 810: shrink -> shrinks\n\n\n## Summary of suggestions\n\nI would be glad to increase my score if the following aspects are addressed :\n- a discussion on previous work relating pinball loss, calibration and regularization is added\n- the description of Algorithms 1,2,3 is improved\n- the group batching method is more precisely described \n\n## UPDATE AFTER REBUTTAL\n\nMy suggestions have been addressed. Therefore I increase my score to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed the limitations of MAQR related to high-dimensional data. \nIt would be desirable to discuss limitations of the other methods (i.e. combined calibration loss, centered intervals and group batching).\n\nThe potential negative societal impact is adequately addressed in the appendix.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "BJoOCtuulp-", "reviewer": "Reviewer_ed7u", "summary": "This paper studies procedures for quantifying uncertainty in a regression setup, focussing on the full quantile function as a form of interpretable output. The main contributions of the paper include 1) exhibiting that optimizing the pinball loss can lead to poor quantile functions, and 2) proposing new quantile methods which work in an agnostic manner.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem studied in the paper seems quite relevant in the context of current large-scale deep learning models being mis-calibrated, for regression as well as classification tasks. \n- Overall, I enjoyed reading this paper and it is in general well-written. With the limited space though, I would urge the authors to go into a bit more detail about some of the proposed methods rather than listing all of them in Section 3.\n- While the MAQR method seemed to perform well on the UCI datasets, it was not tried on the fusion dataset due to compute issues. I would be interested in seeing some simulation experiments for problems with varying samples (n) and dimensions (d). This would allow us to understand the breakdown point of the MAQR method in some more detail and provide a comparison with the other baselines. \n- Since one of the key contributions of the paper is proposing new methods, I am a bit worried about the choice of the benchmark datasets from UCI. In particular, why were these particular set of 8 datasets  chosen from all the available ones? Can the authors provide evaluation on some other larger scale datasets?\n- Choice of network architecture: the results were obtained with a single two-layer ReLU network. This seems quite limiting, and it would be interesting to see consistent trends across a variety of depths (even going to larger depths to showcase trends). Given the particular choice, I am not particularly convinced that the method works well in the extreme over-parameterized regime.\n- Algorithm 1 (MAQR) uses residuals instead of directly working with the y values. Is there an advantage in doing so? Can you perhaps performa an ablation over these two choices to showcase why this particular form was chosen?\n\n\nThe authors have addressed my concerns and I have updated my rating accordingly.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors provide a discussion on this and relate their work to the literature on fairness in ML.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this study, the authors are concerned with the problem of estimating the full quantile function of the responses in a regression setting. This task can be typically realized by performing quantile regression on all quantiles. However, the authors pointed out a drawback of pinball loss, which is usually used in quantile regression, and a new method that can overcome this drawback is proposed. Both the problem and the setting discussed in this paper are important, and the theoretical validity of the proposed method and the demonstration of its performance through experiments are properly done. All reviewers found the paper useful for the ML community. The authors are encouraged to consider updating the paper based on the reviewers' comments and suggestions.
Decision: Accept (Poster)