Calibration by Distribution Matching: Trainable Kernel Calibration Metrics
Authors: Charles Thomas Marx
Sofian Zalouk
Stefano Ermon
Venue: NeurIPS 2023 poster (2023)
Keywords: Uncertainty Quantification
Calibration
Decision Making
Probabilistic Forecasting
TL;DR: We introduce a unifying framework for calibration metrics as distribution matching objectives.

Abstract: Calibration ensures that probabilistic forecasts meaningfully capture uncertainty by requiring that predicted probabilities align with empirical frequencies. However, many existing calibration methods are specialized for post-hoc recalibration, which can worsen the sharpness of forecasts. Drawing on the insight that calibration can be viewed as a distribution matching task, we introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression. These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization. Furthermore, we provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions. Our empirical evaluation demonstrates that employing these metrics as regularizers enhances calibration, sharpness, and decision-making across a range of regression and classification tasks, outperforming methods relying solely on post-hoc recalibration.

Calibration by Distribution Matching: Trainable Kernel Calibration Metrics
Charles Marx*
Sofian Zalouk" Stanford University szalouk@stanforddoo
Stefano Ermon Stanford University ermonêcs. stanford, edu
Stanford University ctmarxθcs. stanford edu
Abstract
Calibration ensures that probabilistic forecasts meaningfully capture uncertainty by requiring that predicted probabilities align with empirical frequencies. However, many existing calibration methods are specialized for post-hoc recalibration, which can worsen the sharpness of forecasts. Drawing on the insight that calibration can be viewed distribution matching task. introduce kernel-based calibration metrics that unify and generalize popular forms calibration for both classification and regression. These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization. Furthermore, provide intuitive mechanisms tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions. Our empirical evaluation demonstrates that employing these metrics regularizers enhances calibration, sharpness, and decision-making across a range of regression and classification tasks, outperforming methods relying solely post-hoc recalibration.
Introduction
Probabilistic forecasts are valuable tools for capturing uncertainty about an outcome of interest. In practice, the exact outcome distribution is often impossible to recover [5]], leading to overconfident forecasts [5]. Calibration provides statistical guarantees on the forecasts, ensuring that the predicted probabilities align with the true likelihood of events. For example, consider weather forecast that assigns an 80% probability of rain tomorrow. well-calibrated forecast would imply that, on average, it rains on 80% of the days with such predictions. By appropriately quantifying uncertainty, calibration empowers decision-maace to efficiently allocate resources and mitigate risks, even when the outcome distribution cannot be recovered exactly.
Many forms of calibration have been proposed in both classification 42 23 53
21 and regression to ensure that uncertainty estimates match true frequencies. From methodological perspective, the literature on enforcing calibration is fractured; many algorithms to enforce calibration specialized for particular form of calibration [39 29 52.
[0 [7] or are designed exclusively for post-hoe calibration [27 (23 Although post-hoc methods are effective at enforcing calibration, they often lead to degradation in the predictive power, i.e. sharpness, of the forecaster [26). This motivates the need for versatile metrics that can enforce various forms calibration during training.
In this work, we introduce unified framework wherein existing notions of calibration are presented as distribution matching constraints. To enforee distribution matching, we use the Maximum Mean Discrepancy (MMD), giving us unbiased estimates of miscalibration that are amenable to gradient- based optimization. We frame these metrics as regularizers and optimize them alongside proper scoring rules. This allows us to enforce calibration while preserving sharpness.
"Equal Contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Framing calibration as distribution matching [38 allows us to use the Maximum Mean Discrepancy (MMD) 14 to give unbiased estimates of miscalibration that are amenable gradient-bus based optimization. An advantage of this approach is that we express existing calibration measures by varying the kernel in the MMD metric. Moreover, we can express new notions of calibration specific to downstream decision task. This allows us to prioritize errors that impact decision-making [52] Our contributions can be summarized as follows:
We propose framework based on distribution matching to unify many notions of calibration.
We provide differentiable training objectives to incentivize a wide variety of popular forms of calibration during empirical risk minimization in supervised learning. We frame these metrics as regularizers that can be optimized alongside proper scoring rules. This allows us to enforce calibration without degrading sharpness.
We show how to define and enforce new calibration metrics tailored to specific downstream decision problems. When these metrics are successfully optimized, decision makers can accurately estimate the decision loss on unlabeled data.
We perform an empirical study across 10 tabular prediction tasks (5 regression and 5 classification) and find that using our metrics as regularizers improves calibration, sharpness, and decision makiin relative to existing recalibration methods.
2 Related Work
Calibration
Our work builds on the literature on calibrated forecasting 31
13
Recently, 17
many forms of calibration have been proposed in classification [5]
25
24
32
35
33
10
28
36 127, regression (23 42
39 51, and beyond [47 [22] Algorithms to enforce calibration are often designed specifically for one form of calibration. For example, significant attention has been devoted to effectively estimating the Expected Calibration Error (ECE) to calibrate confidence levels in binary classification [19 44 4 15 35). Our work aims to supplement this work by providing general calibration metrics that can be used as training objectives. These methods can be used in concert with post-hoc calibration, to encourage calibration during model training and correct any remaining miscalibration after training is complete.
53
Decision Calibration An interesting line of recent work focuses calibrating predictions for downstream decision problems [39] [52.. Zhao et al. [52) shows that decision calibration can be achieved when the set of possible actions is finite. We show cases in which this result be extended to infinite action spaces by leveraging low-dimensional sufficient statistic of the decision problem. Whereas Zhao al. (52) compute calibration by approximating worst-case decision problem with linear classifier, we use the kernel mean embedding form of an MMD objective, making it simple to calibrate with respect to a restricted set of loss functions by adjusting the kernel.
Motivated by issues arising from the histogram binning of the Expected Calibration Error, recent stream of research uses kernels define smooth calibration metrics. Zhang et al. (50) and Popordanoska et al. (37 use kernel density estimates to estimate calibration in classification settings. In contrast, our work applies to regression and takes distribution matching perspective. Widmann et al. [47] and Kumar et al. [26] are the most similar existing works our own, and also introduce MMD-based calibration metries. Our work differs from Widmann et al. [46] due to our focus on trainability and optimization, connections to existing forms of calibration, and applications
Kernel- -Based Calibration Metrics
fecision-making. Kumar et al. 226 can be seen as special case of our method applied to top-label calibration in multiclass classification (see Table T. Our work extends the literature on MMD- based calibration metries, showing how to express and optimize 11 existing forms of calibration by varying the kernel in the MMD objective (see Tables and 2
3 Calibration is Distribution Matching
We consider the regression setting, where, given a feature vector & E Rdd, we predict label y E y C R. We are given access to i.i.d. examples (/),....... (I,, yn) distributed according to some true but unknown cumulative distribution function (edf) P over X' x y).. lower case p denotes the probability density function (pdf) or probability mass function (pmf), when it exists. We use

---

Table 1: We express popular forms of calibration in terms of distribution matching, where P is the true distribution and Q is the forecast. The forecaster achieves calibration if the forecast variable and target variable equal in distribution, given the conditioning variable. Here, 62 (Q)xx) is a Bayes optimal action under yylx. yo is a fixed label in ) [0, 1] is threshold, and ((X) is feature mapping. See Appendix for additional details.
Calibration Objective
Forecast Variable
Target Variable
Conditioning Variable
Quantile Calibration
23
Qyxx(()
Pvix(Y) Prix(T) Y
Threshold Calibration
39
Qrix(Y)
1 [Qrix(uo) & a)
Marginal Calibration
12
Decision Calibration
152
Y
(Q
Group Calibration
36
Y
Group(X)
Distribution Calibration
42
Y
Qrix
Individual Calibration 55
Y
X
Local Calibration 27
Y
o(X)
subscripts for marginal and conditional distributions, so 2r (.) is the conditional edf for Y given X = x and (-) is the marginal pdf for Y. When the input X is random, we write Pyzx as the cdf-valued random variable that takes value Py| upon the event that x.
Our goal is to learn forecaster Q that, given feature vector æ, forecasts edf Q+)x(+) over V A forecaster implicitly defines a joint distribution over X' x y by combining the forecasted conditional pdf with the marginal pdf for the features, i.e. g(x, v) px (x)4y (v). The forecaster's joint cdf (xx, y) is defined by integrating the pdf 9(z y) over the appropriate region of X x V. In the same way as (X, Y) ~ a sample from the true distribution, we write a sample from Q as (X, P) Q. where ~ Px and Ý a label drawn from the forecast (Ý = x) ~ Qy
An ideal forecaster perfectly recovers the true probabilities so that Q P. However, in practice we often only observe each feature vector x once, making perfect forecasts unattainable [51]. Instead, calibration enforces that forecasts are accurate on average, leaving us with tractable task that is strictly weaker than perfect forecasting. Each form of calibration requires the forecasts to respect particular law of probability. For example, if Y is binary, distribution calibration states that on examples for which the forecast qy 1y 1) = e, we should observe that Y with frequency In regression, quantile calibration 23 states that y should exceed each quantile e E [o, 1] of the forecast with frequency Calibration can also be enforced within subgroups of the data (i.e. group calibration (36)). ) smooth neighborhoods the input space (i.e. local calibration 27 and even for single feature vector (i.e. individual calibration (51) by using randomization.
Calibration can be expressed as a conditional distribution matching problem, requiring that certain variables associated with the true distribution match their counterparts from the forecasts.
Lemma 3.1 (informal). Quantile calibration equivalent distribution matching between Qyix (Y) and Py Pyzx(Y), Distribution calibration is equivalent to distribution matching between and Y given Qrix. Individual calibration is equivalent to distribution matching between Y and given X.
The choice of random variables that we match in distribution determines the form of calibration recovered. We show how to express many popular forms of calibration in terms of distribution matching in Table []] and include additional details on these correspondences in Appendix A In Section
4 we describe how to express and estimate these forms calibration using kernel-based calibration metrics. In Section 5 we show how to tailor these metries to downstream decision problems. Finally, in Section 6 we apply these metrics as regularizers and explore the empirical effects calibration, sharpness, and decision- decision-making.
4 Calibration Metrics as Trainable Regularizers
In this section, we introduce framework to enforce calibration using distribution matching. We quantify calibration using integral probability metrics which admit unbiased estimates from data

---

and are amenable to gradient- based optimization. This allows us to optimize calibration alongside a standard training objective, achieving calibration without degrading sharpness. Jointly optimizing calibration and sharpness has been shown to outperform only enforcing calibration post-hoe 226
Integral probability metrics (IPMs) are flexible set of metrics to quantify the difference between two distributions (30) in terms of family of witness functions. Since we need to perform distribution matching conditionally, we consider witness functions over both the label Y, as well as a conditioning variable Z that assumes the same distribution under P and Q
D(F,P.Q) sup Ep [f 2 EQ 15(P) 2) fEF
(I)
Different choices for the witness functions F and the conditioning variable Z measure different notions of calibration. Similar to Widmann et al. [47]. we focus on the case where F is the unit ball of the Maximum Mean Discrepancy [4]. denoted MMD(F, P. Q). MMD can either be expressed in terms of the canonical feature map d(y. 2) E H or the reproducing kernel k(((,, 2). (ū, 2²)) = (Ø(y. 2), Φ(T. 2r for H:
a Reproducing Kernel Hilbert Space (RKHS) H. so that the IPM corresponds
(P,,)) = Er (0 2) - Ee (6( 2) 2
= Ep [Y 2), (Y', Z')]] + EQ [A(((,, 2). (Y', z²))] 2E (e(()))) (P,, 2)))
Given dataset D consisting of n ii.d pairs (zi, 3/:) P. we can generate forecasted labels
Di ~ Qy|z. The conditioning variables 2j are computed from (x,, y/) according to the expressions
given in Table I The plug-in MMD estimator is unbiased and takes the form [1]
2 MMD (F, ninn
1
78
his
1)
jti
where bıy is a one-sample U statistic:
hy == k((v,, 2),, (b)) + k k(OOt, 2),, (0), zj)) - (((1 21), (o), - A((Us, 2)); (Dr, 24)
The variance of this estimator can be reduced by marginalizing out the simulated randomness used to sample 9i ~ Qy This can be done either analytically, or empirically by resampling the forecasted labels multiple times per example. We find that empirically marginalizing out the simulated randomness improves training stability in practice. The full training objective combines a proper scoring rule (we use negative likeliiil)ooo)) with an MMD estimate to incentivize calibration. To train, we divide the dataset D into batches D, C {1.....n} n large enough so that the MMD estimate on batch has low enough variance to provide useful supervision (we find that D3 size 64 suffices in practice). Given family of predictive models (Qo : E Q) (e.g., a neural network), we optimize the following training objective on each batch:
min PEB
log 9r|0, (y:) A A.-D (F, Do, Qo)
DE
In order differentiate MMD (F, Qo) with respect to 8 we require that the kernel k is dif- ferentiable and the samples can be expressed as a differentiable function of the model parameters (e.g., using reparameterization trick). For parameterizations that do not admit exact differentiable sampling, we can apply a differentiable relaxation (e.g.. if Qa is a mixture distribution, we can use a Gumbel-Softmax approximation). In principle, it is possible remove the need for differentiable sampling altogether by optimizing our MMD objective using Monte-Carlo RL techniques (c.g., REIN- FORCE 48 ). However, this is beyond the scope of our work. Note that regularizing for calibration at training time does not give calibration guarantees, unless one can guarantee the objective will be optimized -ft-lee In this sense, regularization and post-hoc calibration are complementary: regularization mitigates the trade-off between calibration and sharpness while post-hoc methods can provide finite-sample calibration guarantees (c.g.. 45 29 regularization and post-hoc calibration together (see Table
For this reason, we suggest using
On the Choice of the Kernel When the kernel k is universal, the MMD metric is zero if and only Y and y are equal in distribution, given Z [4]. Universality preserved when we decompose the kernel over y x Z into pointwise product of two universal kernels, one over y and one over Z [43).

---

Table Popular forms of calibration expressed as distribution matching for a classification problem with classes } = (1..... ............ m.. Recall that gv)x (-) is the forecasted pmf. The forecaster is calibrated the forecast variable and target variable are equal in distribution, given the conditioning variable. For top-label calibration, we use Y* = arg max ςyl (v), which is random due to X. The marginal calibration condition expresses m distribution matching constraints, one for each y E y.
Calibration Objective
Forecast Variable
Target Variable
Conditioning Variable
Canonical Calibration
Eq 1]
Y
Y
q qr x (Y*) gv Vy
Top-label Calibration
Eq 2]
1[Ý Y-}
1(Y Y*}
Marginal Calibration
44 Eq 3]
1(Y = y}
1(Y = y)
Lemma 4.1. Let l be the function defined by the pointwise product k((,, 2), (v.z)) ky Ly. of (2, z²) where by and k2 are universal kernels over spaces y and Z respectively. Then, under mild conditions, k is universal kernel over y x Z.
Theorem 4] for a proof. In our experiments, we explore using a universal RBF kernel over x 2, since satisfies the mild conditions needed for Lemma 4.1 To enforce decision calibration, we also consider a kernel by that is intentionally not universal. and instead only measures differences between P and Q that are relevant for decision-making (see Section 55. Furthermore, to measure Quantile Calibration and Threshold Calibration, we must match the distributions of the probability integral transforms Py|x (Y) and Qyix (Y) instead of Y and Y. Thus, replace ky with universal kernel over (0, 1]. the codomain of the probability integral transform.
See 443
4.1 Calibration Metrics for Classification
Thus far, we have focused estimating calibration metrics in the regression setting. We now describe how our framework applies to classification.
In the classification setting, given feature vector E X = Rd we predict label y E y {1....,m)) m.. As before, we are given access to n i.i.d. examples (.1, 91 w1)..... (In, ya) distributed according to some unknown distribution P over X' x y. Our goal is to learn a forecaster Q that, given a feature vector z, forecasts a probability mass function (pmf) (-)=(-) over labels V.
We express popular notions of calibration used in classification (i.c., canonical, top-label, marginal) in terms of distribution matching in Table 2 See Appendix |A.2 for additional details on calibration and distribution matthhing in classification. We use the same unbiased plug-in estimator for MMD shown in Equation 3 In the classification setting, we can analytically marginalize out the randomness origi- nating from the forecast when computing the MMD estimate. After performing this marginalization, the sample U-statistic hij is is given by
hij k((vi, zz), (Wi 21)) + Σ Σ
ΣV VEJ A'E
aivv)as 2.). (,)= - Σ (.)))(((,, zi), (Mi #;)),
VEY
where the conditioning variables 2, are computed from (aq, v1). as detailed in Table Here, qx()) = |=, (1) is the predicted probability of the label Y = given features 24. When the forecast and target variables depend on variables other than Y (c.g.. Y* depends on qy in top-label calibration), we add those variables as inputs to the kernel. Note that the plug-in MMD estimator with this U-statistic is differentiable in terms of the predicted label probabilities, and that it can be computed in O(n² ?m?) time.
5 Calibration and Decision-Making
Some of the most practically useful results of calibration are the guarantees provided for downstream decision-making [52]. Universal kemelss guarantee distribution matching in principle, but in some cases require many samples. However, if we have information about the decision problem in advance, we can design calibration metrics that are only sensitive to errors relevant to decision-making. In this section, we show how to design such metries and show that they enable us to (1) accurately evaluate the loss associated with downstream actions and (2) choose actions that are in a sense optimal.
5

---

We begin by introducing a general decision problem. For each example, a decision maker chooses an action a E A and incurs a loss f(a. y) that depends on the outcome. Specifically, given information (such as the forecast), a decision policy d 2 + A assigns an action 8(z). The set of all such policies is denoted ((2). The Bayes optimal policy under label distribution Py is denoted 57 ( (Py) == arg aE Ep [é(a. Y)]. Decision calibration (52) provides two important guarantees:
Eq [e(a, Y) 2 - =| - Ep [t(a, Y) 2 - =l, Va A,
(loss estimation)
Ep (Q( Y) Z = =| S Ep [e(ō(z), Y) 2 = =], Vz E 2, 2
(no regret)
Loss estimation says that the expected loss of each action is equal under the true distribution and the forecast. This means we can estimate the loss on unlabeled data. No regret says that the Bayes optimal action according to the forecast 6 (Qyl=) achieves expected loss less than or equal to any decision policy 8 A(2) that depends only on z. If we condition on the forecast by choosing z = Yr then a decision-maker who only has access to the forecast is incentivized to act as if the forecast correct. Additionally,
consider cases where the loss function is not known in advance. For example, when a vendor releases foundation model to serve a large population of users, it is likely that different users have distinct loss functions. Thus, we also show settings in which we can achieve accurate loss estimation and no regret for all loss functions & included in family of losses C.
A General Recipe for Decision Calibration We now describe approach to measure de- cision calibration using our kernel-based metrics. For each action a AA and loss & E C. want to measure the discrepancy between the expectation of the loss {(a, Y) under P and Q. To achieve this, we define the feature map that computes the losses of all actions (ly) = (E(a, (...A.E) This assigns to P and Q the mean embeddings μP Ep [(((a, ()), and uQ = Eg [(E(a, D.EAE)) respectively. Letting F be the unit ball of the Hilbert space associ- ated with this feature map, the mean embedding form of MMD gives us that (F,
||pp- (PP-~QQ7 2 = EmeA Erec (Ep [[t(a, Y)) - Eg [l(a, Po1)) When either A or is infinite, the corresponding sum is replaced by an integral. This metric measures whether each action has the same expected loss marginally under the true distribution and the forecasts. Thus, accurate loss estimation is only guaranteed marginally and the no regret result is relative to constant decision functions that choose the same action for all examples.
Next, we refine the decision calibration results to hold conditionally on variable Z. Recall that the kernel over y given by (u. = (ń(y), d(y')). We lift this kernel into y 2 by defining k((v, z), (v',=") = ky y ]k=(2,²)) where k- is a universal kernel. The MMD metric corresponding to this kernel measures decision calibration conditioned on Z. If we choose Z to be the forecast Z = Q then the no regret guarantee says that acting as if the forecasts were correct is an optimal decision policy among all policies depending only on the forecast.
Note that the feature maps defined above are infinite dimensional when A or L are infinite. However, applying the kernel trick sometimes allows us to estimate the MMD metric efficiently. Still, optimizing the MMD metric may be difficult in practice, especially when the conditioning variable Z is very expressive. We suggest using the metric as a regularizer to incentivize decision calibration during training, and then still apply post-hoe recalibration if needed. We now give two concrete examples, one where the action space is infinite and one where the set of loss functions is infinite.
Example (Point Estimate Decision). Consider a regression problem in which decision-maker chooses a point estimate a E R and incurs the squared loss E(a,y) y) = (a y)) The expected loss of an action a is Ep [(a - y) = (a - Ep [y])² Ep [y - Ep [ Observe that the expected loss only depends on Y through the first two moments, Ep [Y] and Ep [y We use the representation <(v)= (v. =²)), ), so that the kernel is k(y, v') = ((v, v²)) ), (v/, 2/ 2/) From the mean embedding form of MMD, we see that MMD(F,
= ||p-- - PQ 12 = [ - 1 | 1)? Therefore, the metric is zero exactly when the expected loss of each action is the same under the true distribution and the forecasts. The MMD metric incentivizes the forecasts to accurately reflect the marginal loss of each action. However, often want to know the optimal action given particular forecast, not marginally. We can now choose the conditioning variable to refine the guarantee. Let Z be the forecast Z- = Qrix and choose the kernel k((v. 2), (v/, =')) ky y >/) (z, z') where (V.) = ((s. y )) (v/, )) as before and (2, 2²) is universal. For this kernel, the MMD will equal zero if and only if Ep [Y] [rix] = EQ [Y | Qyix] and Ep [X² Qr1x] Eq 2

---

Qyyx] almost surely. The Bayes optimal action suggested by the forecast is optimal among decision policies that depend only on the forecast:
Ep [e (5 (Qrix). Y) Ep ((((Qr1x),
4o E AQQyI))
(5)
In other words, the decision maker is incentivized to behave as if the forecasts are correct.
Example 2 (Threshold Decision). The above example had an infinite action space and single loss function. Here, we consider problem where the action space is small but the set of loss functions is infinite. This setting applies for model vendor who serves model to multiple decision makers, each with potentially different loss function. For example, suppose that Y E [0, T) is blood value for patient for some 7 > O, and the decision task is to determine whether Y exceeds threshold & E [0. 7] that indicates eligibility for clinical trial. Formally, the decision maker chooses a threshold a E {+1} to minimize the loss (a, forecasts for all thresholds E [0. 7J. i.c C = {&; t [0, }. In this case, we choose the feature map <(y) (1 {y 2 () (.) The corresponding kernel is k(y, x²) (o(v), o(v') min v y'}. The mean embeddings are Ep [Ø(Y)] = Py and Eo(o(1))
1 {a &f sign(y t)}. Our goal is to simultaneously calibrate our
Qy, the marginal edfs for Y and Yq- Thus, the MMD metric measures Py Qy 1/ = S: (P) ()))² dt, which is zero exactly when Py Qy- When MMD(FF, P, Q) O, the expected loss of each action is the same under the true distribution and the forecasts, for all E C. Similar to the previous example, we can achieve conditional decision calibration by adding universal kernel over the conditioning variable.
Discussion The connections between decision-making and calibration have been previously studied by Zhao al. [52 and Sahoo al. [39]. Here, we extend results from Zhao et al. [52] include infinite action spaces and further refine the decision guarantees to depend on an arbitrary conditioning variable. In Example we connect our methods to the special case of threshold calibration [39]. Notably, our framework provides a strategy to optimize decision calibration at training time.
6 Experiments
The main goals of our experiments are to: (1) compare the performance our method with other trainable calibration metrics, and with standard post-hoc recalibration methods; (2) study the impact of tailoring the MMD kernel to a specific decision task; and (3) study whether trainable calibration metrics can be used to improve calibration across local neighborhoods of data
6.1 Datasets
We consider standard benchmark datasets and datasets relating to decision-making tasks. For each dataset, we randomly assign 70% of the dataset for training, 10% for validation, and 20% for testing.
Regression Datasets We use four tabular UCI datasets (SUPERCCODDUCTTVVITY [6], CRIME [34] BLOG (6), FB-COMMENT (II). as well as the Medical Expenditure Panel Survey dataset (MEDICAL- EXPENDITURE 77). They are common benchmarks in uncertainty quantification literature 223 39]. The number of features ranges from d = 53 to 280. The total number of examples n, and features d
for each dataset can be found in Table 3
Classification Datasets We use five tabular UCI datasets: BREAST-CANCER 49
HEART-DISEASE
[8], ONLINE-SHOPPERS [40, DRY-BEAN [ and ADULT [2]. The datasets range from m = 2 to 7 label classes, and d = 16 104 features. The total number of examples n, features d, and classes m for each dataset can be found in Table
Crop Yield Dataset We introduce CROP-YIELD dataset, where the task is predict yearly wheat crop yield from weather data for different counties across the US. We use the NOAA database to track summary weather statistics across each year (temperature, precipitation, cooling/heating degree days), and the NASS database to track annual wheat crop yield. The dataset spans from 1990-2007, where data from 1995 was held-out for visualization, and the remaining data was used for training. We use this dataset to evaluate local calibration with respect to the latitude and longitude coordinates.
"oode reproduce experiments be found wttg://shhheeoooeocmmmmclliaaied
7

---

Table 3: Comparison of model performance on five different regression datasets. Models were trained with two objectives: NLL and NLL + MMD (Ours). We display the metrics on the test set for each training procedure, both with and without post-hoc Quantile Calibration [23 fit to the validation set. is the number of examples in the dataset and d is the number of features.
Dataset
Training Objective
NLL 4
Quantile Cal. 4
Decision Cal. +
NLL
00.716 = 0.007
0,220 = 0.004
0.151 0.009
CRIME
Post-hoc
-0.387 = 0.012
0.154 = 0.003
0.089 = 0.005
1992
NLL MMD (Ours)
-0.778 + 0.008
0,164 # 0,004
0.042 0,011
= 102
Post-hoc
-0.660 0.014
0.105 0.004
0,041 0.005
NLL
0.997 0.040
0,402 0.006
4,087 0.004
BLOG
Post-hoc
0.624 0.021
0.053 0.001
4.090 0.001
52397
NLL MMD (Ours)
0.957 = 0.008
0,302 0.002
4.051 0.002
d 280
Post-hoc
0.945 = 0.007
0,042 = 0.001
4.067 = 0.001
NLLL
1,535 0.000
0.078 0.001
0.476 0.002
MEDICAL-EXPENDITURE
Post-hoc
1.888 + 0.002
0.059 * 0.001
0.472 # 0.002
N 33005
NL. MMD (Ours)
1,532 0.000
0.059 0.001
0.438 0.002
d 107
Post-hoc
1.780 0.001
0.045 0.000
0,431 0.001
NLL
3.375 0.008
0.062 0.003
0.211 0.003
SUPERCONDUCTIVITY 21264
Post-hoc
3.707 0.007
0.066 0.001
0.202 0.003
NLL + MMD (Ours)
3.269 # 0.012
0.042 # 0.003
0,182 # 0.003
81
Post-hoc
3.643 = 0.010
0.050 = 0.002
0.186 # 0.003
NLL
0.634 = 0.016
0.334 = 0.004
3.151 # 0.003
FB-COMMENT
Post-hoc
0.734 0.015
0.063 0,001
3,151 0,002
n 40949
NLL MMD (Ours)
0.605 0.004
0.258 0.003
3.131 0.003
d = 53
Post-hoc
0.639 = 0.005
0.054 = 0.001
3.138 0.002
6.2 Experimental Setup and Baselines
Models To represent the model uncertainty, we use a feed-forward neural network with three fully-connected layers. In the classification setting, the model outputs the logits for all m classes. In the regression setting, the model outputs the predicted mean p(Y) and variance o (Y)). which are used to parameterize Gaussian probability distribution. This allows us to tractably compute the inverse edf Qyix and perform differential sampling, enabling us to test our calibration metrics in multiple settings. See Appendix for more details model hyperparameter tuning.
Training Objectives In the regression setting, we consider two training objectives: Negative Log- likelihood (NIL),, and our trainable calibration metric regularizer (NII. MMD). For MMD, we enforce a notion of individual calibration (Table T by choosing the conditioning variable Z = X to match the distributions of (X, Y) and (X, Y). More specifically, we optimize an MMD objective where the kernel &s is decomposed among the pair of random variables as k(((,, y), (a', y')) kx (z, z²) (3. The form of xx and ky varies across experiments. In the classification setting, we consider cross-entropy (XE) loss, and three trainable calibration metrics as regularizers: MMCE 266 KDE 377, and MMD (Ours).
Baselines In the regression setting, we compare our approach against an uncalibrated forecaster trained with NLL, and with post-hoc recalibration using isotonic regression (23). In the classification setting, we compare our approach against an uncalibrated forecaster trained with XE loss. Furthemore, compare our approach to trainable calibration metrics (MMCE (26) and KDE (37)). We compare all approaches against temperature-scalinn (15), a standard post-hoc recalibration method. For both classification and regression, we train the post-hoc recalibration methods validation set, and subsequently evaluate on the held-out test set.
To quantify predictive performance in regression, we evaluate the negative log-likelihood, quantile calibration emor (QCE) 223. and decision calibration error (see Section 6.3 In classifica- tion, we compute accuracy, expected calibration error (calibration), and average Shannon entropy (sharpness) of the forecast. We also report kernel calibration error (KCE), defined as the MMD
Metrics
8

---

Table Experimental results for five tabular classification tasks, comparing MMCE regularization
26
KDE regularization [7]. and MMD regularization (Ours), alongside a standard cross-entrops
(XE) loss without post-hoc recalibration. Here, n is the number of examples the dataset, d the number of features, and m is the number of classes.
Dataset
Training Objective
Accuracy 1
BCE 4
Entrupy 4
KCE 1
BREAST-CANCER
95.372 = 0.160
0.194 0.003
0.453 0.004
6.781 = 0.601
569
XE + MMCE
94.770 = 147
0.060 + 0.001
0.076 # 0.002
0.392 = 0.086
d = 30
XE ECE KDE
94,351 = 0.163
0.062 0.001
0.065 0.001
0.225 = 0.083
XE MMD (Ours)
95.789 0.060
0.052 0.000
0,006 0.000
0.014 0.000
HEART-DISEASE
XE
55,904 196
0.373 0.003
1,512 0.007
0.581 0.016
= 921
MMCE
60.787 0.208
0.267 0.002
0.947 0.003
0.097 0.002
ECE KDB
50.036 = 2.801
0,304 0.012
1.392 0.016
0.450 0.050
MT = 5
XE MMD (Ours)
61.516 + 0.255
0,261 + 0.003
0,945 # 0.006
0.077 = 0.002
ONOINESSHOPPERS
89.816 # 0.037
0.129 0.001
0.221 0.003
0.022 # 0.001
12330
XE + MMCF
89,933 = 0.036
0.128 0.001
0.220 & 0.002
-0.019 # 0.001
d = 28
XE BCE KDE
90.019 0.034
0.127 0.000
0.225 0.002
-0.022 0.001
2
XE MMD (Ours)
89.976 0.031
0.127 0.001
0.218 0.002
-0,026 0.000
DRY-BEAN
XB
92.071 0.025
0.113 0.000
0.264 0.001
0.061 0.002
1 13612
MMCE
92.772 = 0.035
0.099 0.000
0.224 0.002
0.048 0.004
= 16 m 7
XE ECE KDE
92,760 + 0.037
0.097 0,000
0.232 + 0.001
0.041 = 0.003
XE + MMD (Ours)
92.894 = 0.035
0.089 + 0.000
0.174 # 0.002
0.040 = 0.004
ADULT
XE
84,528 = 0.040
0.186 0.000
0.320 # 0.002
-0.014 # 0.002
32561 d = 104
XE MMCE
84,203 0.042
0,191 0.000
0,340 0.002
-0.015 0.002
XE ECE KDE
84.187 0.045
0.178 0.001
0,414 0.002
-0.020 0.003
m
MMD (Ours)
84,565 + 0.035
0.178 0.001
0,315 0.002
-0.018 0.001
objective for an RBF kernel over X' x y. Each metric is computed on a held-out test set. We repeat all experiments across 50 random seeds and report the mean and standard error for each metric.
The results for regression and classification are shown in Table 3 and Table respectively. In regression, our objective (NLL + MMD) achieves better QCE than the NLL objective across all datasets while maintaining comparable (or better) NLL scores on the test set. On most datasets, we find that our method also reduces the sharpness penalty induced by post-hoc calibration. Importantly, we also observe that post-hoe calibration and our method are complementary for calibrating fore- casters; post-hoc methods can guarantee calibration held-out data, and calibration regularization mitigates the sharpness penalty. For classification, we find that our method improves accuracy, ECE, and entropy relative to the baselines tested for calibration regularization. These trends also generally hold with post-hoc calibration, where we observe that our method achieves better calibration and accuracy across most datasets (See Table 5).
Results
Computational Requirements Relative to training with an unregularized NLL objective, our method with MMD regularization requires average wall clock time per epoch of 1.2x for the 5 regression datasets, and 1.3x for the 5 classification datasets.
6.3 Experiment: Decision Calibration
One of the key advantages of our trainable calibration metrics is their flexibility in enforcing multiple notions of calibration. We consider decision-male making task to study the effectiveness of our method in improving decision calibration. The decision task is to choose an action a E {+1). whose loss t(a,y) y) (a & sign(y - <)} depends only on whether the label y exceeds threshold e E TR. Given true distribution P and a forecaster Qy x. the Decision Calibration Error (DCF) is DCE2 (P, = Eas [|Ep[((Y, - ExEy
MMD Objective To optimize for decision calibration, we tailor the kernels kx, by to reflect the decision problem. More concretely, we choose a kernel function that penalizes predicted labels which on the wrong side of c, namely by ky. y') = +1 if sign(y e) = sign(y' c), and -1 otherwise.

---

LCE
0.25
0.2
0.15
0.1
0.05
(a) NLL
(b) Ours (NLL MMD)
Figure I: We visualize Local Calibration Error for forecaster trained to predict annual wheat crop- yield based climate and geospatial data. The standard NLL objective (Left) leads to forecaster that is miscalibrated across local geospatial neighbourhoods, as seen by areas of brighter color. Our kernel-based calibration objective (Right) leads to better calibration across local neighborhoods.
To allow for gradient based optimization, we relax the objective to yield the differentiable kernel ky (y, y') = tanh(y -tanh(y - -e.. Notice that by (V, y²) R +1 when sign(y - c) sign -c), and - otherwise. To encourage decision calibration for all features, we let kx be RBF kernel.
Results The results in Table demonstrate that the MMD training objective tailored to the decision problem achieves the best decision calibration across all datasets. Additional results in Appendix B.3 show that using an MMD kernel tailored to a decision problem (i.e. the tanh kernel) improves decision calibration scores across all datasets.
6.4 Experiment: Local Calibration
Setup To provide intuition on how our metrics can facilitate calibration in local neighbourhoods of features, we study how location affects the reliability of forecasts on the CROP-YIELDD regression dataset. We tailor the kernels kx by to local calibration for geospatial neighborhoods. Namely, we select (xx) which extracts the geospatial features (latitude, longitude) from the full input vector We then define kx (x, x²) = (RBF(((Z)) ((x') and ky (y. 3/) = KRBF y').
Metric Extending prior work local calibration 27 given edf forecaster Qy fea- ture embedding o(-) and kernel k, we define Local Calibration Error (LCE) for regression is The total LCE computed LCE (I) Σe LCE(I; where c 1. Intuitively, LCE measures the QCE across local regions determined by ((-). We visualize the results in Figure | for B = 20.
ICF((;; = E. (v= S Qri,, (c)] k((()), ((=))) / (Σ1 ((;)))
7
Conclusion
We introduced flexible class of trainable calibration metrics based on distribution matching using MMD. These methods can be effectively combined with existing post-hoc calibration methods to achieve strong calibration guarantees while preserving sharpness.
Limitations Optimizing the MMD metric requires that samples from the forecast are differentiable with respect to the model parameters. This restricts the family of applicable forecasters, but differen- tiable relaxations such as Gumbel-soffmax can mitigate this challenge. Furthermore, in practice it is often not possible to achieve zero calibration error via regularization due to computational and data limitations. To mitigate this limitation, suggest pairing our calibration regularizers with post-hoc calibration methods. Lastly, the scale of the MMD metric is sensitive to the chosen kernel (e.g.. the bandwidth of RBF kernel), it can be difficult to understand how calibrated model is in an absolute sense based on the MMD metric.
10

---

8 Acknowledgements
CM is supported by the NSF GREP. This research was supported in part by NSF (#1651565), ARO (W911NF-21-1-0125), ONR (N00014-23-1-2155), and CZ Biohub.
References
[1] Dry Bean Dataset.
UCI
Machine
Learning
Repository, 2020.
DOI:
https//fo..or//1022444////$$S
[2] Barry Becker and Ronny Kohavi. Adult. UCI Machine Learning Repository, 1996. DOI:
https//bb..org/11.2333/3/x3
[3] James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parameter optimization. Advances in neural information processing systems, 24, 2011.
[4] Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Meta-calibration: Learning of model calibration using differentiable expected calibration error. arXiv preprint arXiv:2106.09613 2021.
[5] Jochen Bröcker. Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Society: A journal of the atmospheric sciences, applied meteorology and physical oceanography, 135(643):151221115) 2009.
[6] Krisztian Buza. Feedback prediction for blogs. In Data analysis, machine learning and knowledge discovery, pages 145-152. Springer, 2014.
[7] Joel W Cohen, Steven B Cohen, and Jessica S Banthin. The medical expenditure panel survey: a national information resource support healthcare cost research and inform policy and practice. Medical care, pages S44-S50, 2009.
[8] A Philip Dawid. The well-calibrated bayesian. Journal of the American Statistical Association, 77(379):005-610, 1982.
[9] Morris DeGroot and Stephen Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statistician), 32(1-2): 1983.
[10] Cynthia Dwork, Michael P Kim, Omer Reingold, Guy N Rothblum, and Gal Yona. Outcome indistinguishabiiity. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 1095-1 108, 2021.
[11] Tilmann Gneiting and Matthias Katzfuss. Probabilistic forecasting, Annual Review of Statistics and Its Application, 1:125-151, 2014.
[12] Tilmann Gneiting and Roopesh Ranjan. Combining predictive distributions. Electronic Journal of Statistics, 7:1747-1782, 2013.
[13] Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69 (2):243-266) 268,, 2007.
[14] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
[15] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. International Conference on Machine Learning, pages 1321-1330. PMLR, 2017.
[16] Kam Hamidieh. Superconductivty Data. UCI Machine Learning Repository, 2018. DOI:
http://coi.ng//0.24332/553.
[17] Ursula Hébert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Calibration for the compuuationally--denffaa masses. In International Conference on Machine Learning, pages 1939-1948. PMLR, 2018.

---

[18] Andras Janosi, William Steinbrunn, Matthias Pfisterer, and Robert Detrano. Heart Disease. UCI Machine Learning Repository, 1988. DOI:
[19] Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Michael C Mozer, and Becca Roelofs. Soft calibration objectives for neural networks. Advances in Neural Information Processing Systems, 34:29768-22 -27799,, 2021.
[20] Jon Kleinberg. Sendhil Mullainathan, and Manish Raghavan. Inherent trade-offs in the fair determination of risk scores. arXiv preprint
0000000000 2016.
[21] Volodymyr Kuleshov and Stefano Ermon. Estimating uncertainty online against an adversary. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
[22] Volodymyr Kuleshov and Percy S Liang. Calibrated structured prediction. Advances in Neural Information Processing Systems, 28, 2015.
[23] Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertaintics for deep learning using calibrated regression. In International conference on machine learning, pages 2796-2804. PMLR, 2018.
[24] Meelis Kull and Peter Flach. Novel decompositions of proper scoring rules for classification: Score adjustment precursor calibration. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2015, Porto, Portugal, September 7-11, 2015, Proceedings, Part I 15, pages 68-85. Springer, 2015.
[25] Meelis Kull, Telmo Silva Filho, and Peter Flach. Beta calibration: well-founded and easily implemented improvement on logistic calibration for binary classifiers. Artificial Intelligence and Statistics, pages 623-631. PMLR, 2017.
[26] Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. International Conference on Machine Learning, pages 2805-2814 14. PMLR, 2018.
[27] Rachel Luo, Aadyot Bhatnagar, Yu Bai, Shengjia Zhao, Huan Wang, Caiming Xiong, Silvio Savarese, Stefano Ermon, Edward Schmerling, and Marco Pavone. Local calibration: metrics and recalibration. In Uncertainty in Artificial Intelligence, pages 18851 PMLR, 2022.
[28] Valery Manokhin. Multi-class probabilistic classification using inductive and cross abers predictors. In Conformal and Probabilistic Prediction and Applications, pages 228-240. PMLR, 2017.
[29] Charles Marx, Shengjia Zhao, Willie Neiswanger, and Stefano Ermon. Modular conformal calibration. In International Conference on Machine Learning, pages 15195, PMLR. 2022.
[30] Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in applied probubility, 29(2):429-443, 1997.
[31] Allan H Murphy and Robert I. Winkler. Reliability of subjective probability forecasts of precipitation and temperature Journal of the Royal Statistical Society Series C: Applied Statistics, 26(1):41-47, 1977.
[32] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
[33] Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning, pages 625 632. 2005.
[34] United States. Bureau of the Census et al. Census of Population and Housing, 1990 [United States]: Summary Tape File 3A, Record Sequence Example File. 1992.
[35] John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
12

---

[36] Geoff Pleiss, Manish Raghavan. Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. On faimess and calibration. Advances in neural information processing systems, 30, 2017.
[37] Teodora Popordanoska, Raphael Sayer, and Matthew Blaschko. consistent and differentiable Ip canonical calibration error estimator. Advances in Neural Information Processing Systems, 35:7933-7940 2022.
[38] Mihaela Rosca, Balaji Lakshminarayanan, and Shakir Mohamed. Distribution matching in variational inference. arXiv preprint arXiv: 06847, 2018.
[39] Roshni Sahoo, Shengjia Zhao, Alyssa Chen, and Stefano Ermon. Reliable decisions with threshold calibration. Advances in Neural Information Processing Systems, 34, 2021.
[40] C. Sakar and Yomi Kastro. Online Shoppers Purchasing Intention Dataset. UCI Machine Learning Repository, 2018. DOI: https://doi.org/1... (tt.//oo.o../00022444//////
[41] Kamaljot Singh, Ranjeet Kaur Sandhu, and Dinesh Kumar. Comment volume prediction using neural networks and decision trees. In IEEE UKSim-AMSS 17th International Conference Computer Modelling and Simulation, UKSim2015 (UKSim2015), 2015.
[42] Hao Song. Tom Diethe, Meelis Kull, and Peter Flach. Distribution calibration for regression. In International Conference on Machine Learning, pages 5897-5906. PMLR, 2019.
[43] Zoltán Szabó and Bharath K Sriperumbudur. Characteristic and universal tensor product kernels.
J. Mach. Learn, Res., 18(233):1-29 2017.
[44] Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification. In The 22nd International Conference Artificial Intelligence and Statistics, pages 3459-3467. PMLR, 2019.
[45] Vladimir Vovk, Ivan Petej, Paolo Toccaceli, Alexander Gammerman, Emst Ahlberg, and Lars Carlsson. Conformal calibrators. Conformal and probabilistic prediction and applications, pages 84-99. PMLR, 2020.
[46] David Widmann, Fredrik Lindsten, and Dave Zachariah. Calibration tests in multi-class classification: A unifying framework. Advances in Neural Information Processing Systems, 32, 2019.
[47] David Widmann, Fredrik Lindsten, and Dave Zachariah. Calibration tests beyond classification. arXiv preprint arXiv:2210.13355, 2022.
[48] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8:229-256, 1992.
[49] William Wolberg, Olvi Mangasarian, Nick Street, and W. Street. Breast Cancer Wisconsin (Di- agnostic). UCI Machine Learning Repository, 1995. DOI: https://doo.rr//022233///C/W
[50] Yunfeng Zhang, Vera Liao, and Rachel KE Bellamy. Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 295-305, 2020.
[51] Shengjia Zhao, Tengyu Ma, and Stefano Ermon. Individual calibration with randomized forecasting. In International Conference on Machine Learning, pages 11387-11397, PMLR, 2020.
[52] Shengjia Zhao, Michael Kim, Roshni Sahoo, Tengyu Ma, and Stefano Ermon. Calibrating predictions to decisions: A novel approach to multi-class calibration. Advances in Neural Information Processing Systems, 34, 2021.
[53] Johanna Ziegel and Tilmann Gneiting. Copula calibration. 2014.
13

---

A Expressing Popular Forms of Calibration as Distribution Matching
A.1 Calibration in Regression
Here, we show how the forms of calibration listed in Table can be expressed in terms of conditional distribution matching. Note that many of these forms of calibration are sensible in the classification setting as well (e.g., marginal, group, and decision calibration).
Quantile Calibration
Let Y be continuous random variable. A forecaster quantile calibrated if =t)=
IP (t) =t. Vt [0,
(6)
This is equivalent to stating that Qy x (Y) is uniform on the interval [0, 1J. For continuous random variables, the probability integral transform (Y) is uniform on the interval [o, 1]. Thus, quantile
calibration can be written x (Y) xYY), where denotes equality in distribution.
Threshold Calibration Let be a continuous random variable. forecaster satisfies threshold calibration (39 if ((Qr)x(()) S Qy)x(() I c) for all t [0, 1]. e [0. 1]. and y V. This states that QY) is uniform, conditioned on the event Qv)x(u)= s c. Noting that (Y) is uniform, the constraint is then equivalent to the following:
Qy)x & PV) (vrix(v) sct, Ve [0,1].yE)
(7)
Marginal Calibration A forecaster is marginally calibrated [ if [[(Qrxx())) = Py for all E V. Recall that the distribution of Y is defined by (9 X =) ~ Qyk= Thus, the marginal distribution of Y given by y [[Qyıx]. So marginal calibration states that Ý and Y have the same marginal edf or, equivalently, 5 Y.
Decision Calibration Zhao et al.
52 introduce the concept of L decision calibration for multi- class classification, where y = f0, 1] is the set of onehot vectors over C' classes.
Definition A.1 (c* -Decision Calibration, Zhao et al. (52)). Let _ be the set of all loss functions with K actions L {E: J/ x A R. ||| K}, we say that prediction Qyjx is L deccssio calibrated (with respect to Py x vf e C and Agx
Ex ExBrxavix [e( ((Qy1x))] = E ExxEy-r (
(8)
where Agx the set of all Bayes decision rules over loss functions with actions.
They show that - decision calibration can equivalently be expressed as requiring that E[(?
(ō(Or))) = a}] for all 5 E A gx. We can rewrite this
E[1)8((vix)= al = []]][(Qvix) = =a] v8 AÇKKRE A
(9)
note that ([y])(Qyix) = a) gives the conditional distribution of (Y 1[(Q))x) = a) due to the onehot representation of Y. Thus, it is equivalent to require that
rr) 8(Qrix),
4O E Aex
(10)
where A denotes equality in distribution.
Group Calibration Group calibration 20 says that forecasts are marginally calibrated for each subgroup (e.g., subgroups defined by gender or age). We can write this as Ý A Y Group(X), where Group(X) is a variable indicating which subgroup includes the feature vector X.
Distribution Calibration A forecaster satisfies distribution calibration
42 if for all possible
forecasts Qo and labels y E y. we have P (Y & Qrix = Qo) = Qo(y). This is equivalent to requiring Prix (y) Qrxx = Qyix(y) for all y y. Since the edfs take the same value at all y E y. the random variables Y and Y have the same distribution, giving that Y & Y Qyix.
14

---

Individual Calibration Let Y be a real-valued continuous random variable. Individual calibration
[T states that IP (rr $ t) write this as
for all E [0, 1] and x X. Letting Unif((, 1). we can
Qyix - Pr) 1 (vix(9rix())) - (xxx((rxx())) Y - (U) X -
(11)
(12)
(13)
(14)
Local Calibration A forecaster is locally calibrated [27 if Qyix and PY₁x are equal, on average, any neighborhood of the feature space. Similarity feature space defined by kernel b(x,²') In regression, local calibration for quantiles says that
11 { (Q)xx(() S t) x)) =1, VI [0 X.
(15)
BAA(X, 2)
This can equivalently be stated as
E[1 (Qrxx (X) s xx) E[1 (Prx (t),,x)))) 10,0,11, (16) or, in terms of the canonical feature mapping,
E[1 (Qrxx(V) s t) o(x)) = Ei1 { (Prxx t (())), t (0,1),,, e X.
(17)
Thus, it is sufficient for BiQy xl and [[Prix] to have the same distribution conditioned on the feature mapping (X)). This can be written succinctly as
Y 2 6(X)
(18)
A.2 Calibration in Classification
Here, we express three popular forms of calibration from the classification litraature-namely, canonical, top-label, and marginal calibration [44] Eq (1-3)]-in terms of distribution matching constraints, For the entirety of this section, is a discrete random variable taking values in the m). and 1 is the probability mass function of the forecast given the features
set y X Z.
Canonical Calibration A forecaster satisfies canonical calibration P (Y = qy = gyix (v) for all y E y. This says that the label follows the same distribution under the forecast and the true distribution, conditioned on x. other words, Y Y 4/IX-
Top-Label Calibration A forecaster satisfies top-label calibration if (Y Y* (yxxx())) = 1rlx (Y"), where y* == arg maxyey X (y) is the mode of the forecast. This equation can be rewritten in terms of the indicators
E ( - (Y*x
| = Y (rlx(r*)
and the indicator variables are equal in expectation if and only if they are equal in distribution. Thus, top-label calibration is equivalent to (Y= -Y) 1 = Y (V*).
Marginal Calibration A forecaster satisfies marginal calibration if P (Y v qrpx(()) 4yix(u) for all V. Similar to top-label calibration, this can be written
E [1 = y} gy x E [ v} )v)
and the indicator variables are equal in expectation if and only if they are equal in distribution. Thus, marginal calibration can be written 1 {Y y) A 1 y) x for all y V.
15

---

B Experimental Setup and Additional Results
B.1 Reproducibility
In order to reproduce the experiments and results obtained this paper, we provide details about the hyperparameter tuning and required computational resources.
Hyperparameter Search: We use TPES 3 to perform hyperparameter optimization. For all experiments, we vary:
Layer sizes between 32 and 512
RBF kernel bandwidths between 0.001 and 200
Batch sizes between 16 and 512, with and without batch normalization
Learning rates between 10-7 and 10-1
The loss mixture weight À (as in NLL - MMD and XE +X MMD) between 0.1 and 1000.
For each dataset, we 100 hyperparameter settings for each training objective for regression FILL,NLL+ + MMD). and classification setting {XE, XE MMCE, XE KDE, XE + MMD). For each run, we enable carly-stopping if the validation loss does not improve for more than 50 epochs. In the regression setting, we pick the best performing run based on the validation NLL. For classification, we select the best performing run based on the validation accuracy, with validation ECE used for break ties. For each model and dataset, the best performing model is then re-run with 50 random seeds to gather information about standard errors and statistical significance. To capture variability across different seeds, we report the standard error of the mean in Tables 3 and 5
Kernel Bandwidth We select the RBF kernel bandwidth for training on each dataset using the aforementioned hyperparameter optimization. For each dataset in Tables 4 and andd the KDE metric we report for the bandwidth selected through held-out validation set, scaled so that the error of comparable order of magnitude across datasets (since multiplying a kernel by positive scalar preserves the kernel properties).
Baselines In the classification setting. we compare our approach to two state-of of-thee-rt trainable calibration methods, MMCE (26) and KDE [77. For MMCE, we follow the same experimental setup as the official author implementation, where the loss mixture weight À and the Laplacian kernel bandwidth are chosen via held-out validation set. For KDF, we use the I, canonical calibration using the ECFKDE estimator, calculated according to Equation (9) from Popordanoska et al. [37] We follow the same experimental setup as the official author implementation, where the kernel bandwidth is chosen using leave-one-out-ikeliiiee (LOO MLE), and the loss mixture weight À is chosen via a held-out validation set.
Computational Requirements: All experiments were conducted on a single CPU machine (Intel(R) Xeo Gold 6342 CPU @ 2.80GHz), utilizing cores per experiment. When run on a CPU, the training time for single model ranges from 5 to 15 minutes, depending the dataset size. It is possible to perform a full hyperparameter sweep using this setup in ~22 hours.
To accelerate training, we have also run some experiments using an I IIB NVIDIA GeForce GTX 1080 Ti. When run on GPU, the training time for single model ranges from minutes, depending on the dataset size, and a hyperparameter sweep takes 10 hours.
We introduce a new CROP-YIELD dataset for predicting yearly wheat crop yield from geospatial weather data for different counties across the US. We use NOAA database which contains weather data from thousands of stations across the United States. For each county, we track the weather sequence of each year into few summary statistics for each month (aver- aag//maximmm/mmiimum temperatures, precipitation, cooling/heating degree days). To study how climate change and location affect crop yield, we match crop yield data from the NASS dataset to weather stations in the NOAA database, and learn a function to predict crop yield from weather data. We specifically considered the crop yield of wheat across different counties in the US between 1990-2007. Since crop yield data for 1995 is the least sparse, we hold out data from 1995 for visualization, and train our forecasters on the remaining data.
US Agriculture Dataset:
16

---

B.2 Additional Experimental Results
Table Performance on classification tasks, comparing MMCE regularization [26 KDE regulariza- tion [37], and MMD regularization (Ours), alongside standard cross-entropy (XE) loss. Temperature scaling used for post-hoc recalibration of all methods. the number of examples in the dataset, d is the number of features, and m is the number of classes.
Training Objective with Temp. Scaling
Dataset
Accurucy 1
ECE
Entropy
KCE +
DREAST-CANCER
X
95.372 # 0.160
0.105 + 0.031
0.056 + 0.007
0.028 0.005
n 569
XF + MMCE
94.770 0.147
0.052 0.001
0.011 0.001
0.019 0.000
XE ECE KDE
94.351 0.163
0.074 0.018
0.010 0.001
0.022 0.003
XE MMD (Ours)
95.789 0.060
0.052 0.000
0.004 0.000
0.019 0.000
HEART-DISEASE
XE
55.904 0.196
0.325
0.003
1116=00000
0.077 0.000
921
XE MMCE
60.787 = 0.208
0.262 0,002
0.927 0.004
0.067 0.000
d= 23
XE + ECE KDE
50.036 = 2.801
0.276 0.011
1.192 = 0.033
0.082 0.002
m = 5
XE + MMD (Ours)
61.516 # 0.255
0.252 0.003
0.860 0.003
0.066 0.000
ONLINE-SHOPPERS
X
89.816 0.037
0.130 0.000
0.227 0.001
0.050 0.000
12330
XE MMCE
89.933 0.036
0.128 0.000
0.220
0.051 0.000
d 28 =
XE ECE KDE
90.019 0.034
0.125 0.000
0.216 0.001
0,050 0.000
XE + MMD (Ours)
89,976 0.031
0.126 0.000
0.214 + 0.001
0,050 0.000
DRY-BEAN
XE
92.071 0.025
0.102 0.000
0.206
0.001
0.011 0.000
n 13612 d
XE MMCE
92.772 =
0.092 0.000
0.188 0.001
0.011 0.000
XE ECE KDE
92.760
0.000
0.190
0.011 0.000
XE MMD (Ours)
92.894 0.035
0.086 0.000
0.187 0.001
0.011 0.000
ADULT
XE
84.528 # 0.040
0.188 + 0.000
0.330 = 0.000
0.022 0.000
n = 32561 d = 104 m 2
XE + MMCE
84,203 = 0.042
0.190 + 0.000
0.000
0.022 0.000
XE + ECE KDE
84. + 0.045
0.177 à: 0.001
0.338 0.001
0.023 0.000
XE + MMD (Ours)
84.565 = 0.035
0.174 0.000
0.304
0.020 0.000
Table 6: Performance versus the number of simulated samples per forecast used for the plug-in MMD estimate in regression tasks. All other hyperparameters are held constant, including the number of training steps. Increasing the number of samples gives a better estimate of the MMD objective. generally leading to better performance at the cost of additional compute.
CRIME
BLOG
MEDICAL-EXPENDITUREE
Samples
NLL
QCE
DCE
NLL
QCE
DCE
NLL
QCE
DCE
-0.703
0.198
0.088
0.952
0.379
4.171
1.546
0.071
0.448
-0.755
0.188
0.133
0.959
0.444
4.228
1.535
0.07
0.427
-0.72
0.154
0.043
0.96
0.395
4.09
1.539
0.071
0.419
10
-0.777
0.154
0.06
0.85
0.416
4.077
1.54
0.072
0.459
-0.772
0.157
0.054
0,835
0.415
4.031
1.538
0.063
0.431
-0.779
0.150
0.847
0,432
4.016
1.531
0.065
0.447
100
-0.781
0.153
0.043
0.829
0.386
3.978
1.536
0.064
0.449
200
-0.782
0.151
0.043
0.829
0.373
3.978
1.535
0.064
0.442
SUPERCONDUCTIVITY
FB-COMMENT
# Samples
NLLI
QCE
DCE
NLLL
QCE
DCE
3.369
0.053
0.182
0.637
0.268
3.15
3.35
0.062
0.197
0.477
0.294
5
3.311
0.038
0.216
0.409
0.276
3.146
10
3.333
0.036
0.208
0.59
0.278
3.102
20
3.298
0.055
0,262
0,569
0.258
3.119
3,345
0.041
0.208
0.479
0,223
100
3.318
0.036
0.181
0.469
0.23
3.149
200
3.291
0.034
0.182
0.458
0.231
3.137
17

---

B.3 Experiment: Decision Calibration
As an illuutrrative example of how trainable calibration metrics can improve calibration for a specific decision-makinn problem, we consider common real-world decision problem, requiring us to make an action whose utility depends only on whether the label y is greater or less than particular threshold Concretely, the decision loss & A x y - IR, for an action {-1,+1] +1) and label y E V, is defined as f(a.y)
{a F sign(y c)).
Setup In order to achieve decision calibration, we tailor our kernels 1x. ky to closely follow the decision problem. More concretely, we choose a kemeel function that penalizes predicted labels which are on the wrong side of namely ky (v, v/)
tanh(y c) Lanh(y/ - c). Notice that ky (v.y') & +1 when sign(y - c) sign(y - c), and -1 otherwise. Thus, the calibration metric gives the model training feedback that is well aligned with the decision problem. To encourage decision calibration across all features, we set kx to be the universal RBF kernel.
As part of this experiment, we investigate whether tailoring an MMD kernel to the decision problem will improve decision calibration over using a universal RBF kernel for both ky.
Metric To evaluate the effectiveness of methods, we compute the standard Decision Calibration Error (DCE) across all regression datasets, defined as
DCE Σ -
ExEp - 10 aD1l
(19)
aEAA
where the true distribution over X' x V, and Qyix is a forecaster.
Additional Results: Choice of kernel We present two sets of results. The first set of results is shown in T comparing models trained using our objective MMLL+MMD), where one uses a kernel ky that is tailored to the decision problem (tanh), while the other uses universal RBF kernel. As described in Section 6.33 we observe that using the tanh kernel clearly achieves better decision calibration across all dataset, with minimal degradation in sharpness, i.e. NLL. This demonstrates that we can improve decision calibration using our trainable calibration metrics by tailoring our kernels to closely follow the decision problem.
The second set of results, shown Figure 2 further demonstrates the relationship between our trainable calibration metrics and decision calibration throughout model training. By tailoring our kernels to closely follow the decision problem, we observe that decision calibration is optimized throughout training. In contrast, using objectives that are agnostic to the decision problem (i.e. NLL, or MMD with RBF kernels) will lead noticeably worse decision calibration, with little to no improvement throughout training. The same trend is observed on a held-out validation set, as shown Figure 3
Datuset
Kernel (ky)
NLL
Decision Cal. J
RBF
-0.778
0.008
0.042 0.011
(n 1992, 102)
tanh
-0.657 0.010
0.027 0.006
81.0
RBF
0.957 0.008
4,051 + 0.002
In 02397, di 280)
tanh
0.934 + 0.007
3.052 + 0.001
MEDICAL-IXPENDITURE
RBF
1,530 + 0,000
0.447 0.002
(n = 33005, 107)
tanh
1.532 a 0.000
0.438 0.002
SUPERCONDUCTIVVITY
RBF
3,269 # 0.012
0.182 0.003
(n 21264, 81)
tanh
3.311 +
0.127 0.003
FB. COMMENT
RBF
0.605 + 0.004
3.131 0.003
40949, d 53)
tanh
0.598 0.003
2.342 0.003
Table 7: Comparison of model performance trained with our training objective (NLL + MMD), using different kernels for by in the MMD kernel. The RBF kernel is commonly-used universal kernll, while the tanh kernel is specifically designed to match the decision task for decision calibration. All metrics are reported on the test set.
18

---

0.30
0.08
NLL
MMD: Tanh kernel
kernel
0.25
0.07
NLL MMD: RBF kernel
NLL + MMD: RBF kernel
Train DCB
0.20
Train DCE
0.06
0.05
0.10
0.04
0.05
0.00
150
Epochs
Epochs
CRIME
MEDICAL-EXXEEDIITUEE
0.55
0.50
0.42
11ain DCE
Trann DCC
0.35
0.45
0.30
0.40
0.40
0.35
0.00
15
20
30
40
60
25
50
75
100
125
130
20
40
60
80
100
120
Epochs
Epachs
Epochs
(c) BLOG
(d) SUPERCONDUCHIVITY
(e) FB-COMMENT
Figure 2: We visualize the Decision Calibration Error (DCE) evaluated throughout training the training data, for a Gaussian forecaster trained using different objectives. Our method, by tailoring the kernels to specific decision problem, achieves the best DCE among baselines across all datasets. Using our method, we observe continuous improvement in DCE throughout training. Error bars denote the standard deviation computed over 50 random trials.
NLL
NLL
NLL + MMD: Tanh kernel
0.07
NLL + MMD: Tanh keme!
NLL MMD: RBF kernel
NLL + MMD: RBF kernel
Val DCE
0.2
Val DCE
0.06
0.1
60
80
100 Epochs
150
Epachs
CRIME
(b) MEDICAL-EXPENIEE EXPENDITURE
0.25
0.50
Val OCE
3.65
0.35
0.40
6.30
0.39
2.48
0.05
0.3
0.00
10
30
40
60
25
50
75
100
125
150
20
40
60
100
120
Epochs
Epochs
Epochs
(c) BLOG
(d) SUPERCONDUCTIVITY
(c) COMMENT
Figure 3: We visualize the Decision Calibration Error (DCE) evaluated throughout training held-out validation data. Our method, by tailoring the kernels to specific decision problem, achieves the best DCE among baselines across all datasets. Error bars denote the standard deviation computed over 50 random trials.
19


Reviews:
[{"review_id": "gLcdp7Pk26", "reviewer": "Reviewer_hWT9", "summary": "Calibration as a training regularizer:\n\nThis paper focuses on investigating how to incorporate calibration metrics as training regularizers. It starts with summarizing different calibration metrics using a unified view from distribution matching. The paper then proposed a kernel-based general metric that maps labels and conditioning variables to an RKHS space and measures the maximum mean discrepancy on the kernel space. The paper then extends the metric to the decision-making space with two concrete examples. In experiments, combined with post-hoc calibration, using the proposed calibration kernel metric as a trainable regularizer can achieve a better evaluation of calibration in regression problems.", "questions": "I may have missed it, but I am a little confused by the correspondence between the trainable calibration metric and the evaluation. Table 2 shows training using NLL and the proposed method (individual calibration) and evaluating using both quantile calibration and decision calibration. Why quantile calibration is used here? Why not individual calibration in the evaluation? \n\nWhat happens if the problem is a classification problem? The evaluation of calibration is classification is usually tricky. How much the trainable regularize can benefit there?\n\nMinor: table caption is usually on top of the table.\n\n=====\nAfter reading the rebuttal, I raised my score to 5 but still concerned about the lack of theoretical guarantees and the requirement of additional calibration data and step.", "limitations": "The paper discussed technical limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Pros:\nThe paper provides a unified view of many different calibration metrics.\n\nThe kernel-based calibration regularizer is novel.\n\nThe paper is relatively clearly written.", "weaknesses": "Cons:\n\n1. The decision calibration seems to be a natural extension of the section 4, as the decision calibration framework has been proposed before. So the main contribution is based on the kernel-MMD-based calibration metric, which is relatively straightforward.\n\n2. Also, the paper only suggests using the metric as a regularizer to incentivize decision calibration during training but does not really guarantee anything or analyze formally the impact on the decision calibration scenario.\n\n3. The trainable regularizer needs to be combined with post-hoc calibration method, which may raise the question how effective the method is. Table 2 shows that the MMD alone does not always provide enough improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and the helpful feedback! Below, we clarify our contributions surrounding decision calibration, the role of post-hoc calibration, and respond to additional questions. Given our response to the main issues you raised in the review, we kindly request a re-evaluation of your score, taking our rebuttal into consideration.\n\n> **The decision calibration framework has been proposed before. So the main contribution is based on the kernel-MMD-based calibration metric, which is relatively straightforward.**\n\nWe would like to clarify some important differences between our work and the existing literature on decision calibration. Our framework allows us to introduce calibration objectives for decision problems that cannot be calibrated by the methods proposed by [Zhao et al., 2021] and [Sahoo et al., 2021], the two most similar works of which we are aware. Namely, we can enforce decision calibration for large, and sometimes infinite action spaces (see Example 1, line 200 for a simple concrete example). Additionally, [Zhao et al., 2021] and [Sahoo et al., 2021] each calibrate for a single notion of decision calibration ($L^k$ and threshold calibration respectively), while we show how one can regularize for a wide variety of decision problems (see, e.g., Examples 1 & 2 in our paper). Lastly, our work introduces training objectives for decision calibration, while [Zhao et al., 2021] and [Sahoo et al., 2021] provide post-hoc recalibration procedures. We will clarify the relationship to existing work on decision calibration in the revised paper. \n\n*[Zhao et al., 2021] Zhao, S., Kim, M., Sahoo, R., Ma, T., & Ermon, S. Calibrating predictions to decisions: A novel approach to multi-class calibration. Neurips, 2021.*\n\n*[Sahoo et al., 2021] Sahoo, R., Zhao, S., Chen, A., & Ermon, S. Reliable decisions with threshold calibration. Neurips, 2021.*\n\n> **The paper only suggests using the metric as a regularizer to incentivize decision calibration during training but does not really guarantee anything or analyze formally the impact on the decision calibration scenario.**\n\nOur metrics are optimized in expectation exactly when a forecaster is decision calibrated. When we estimate these metrics from data, we can draw on convergence results for MMD sample estimates. Assuming the MMD kernel is bounded by a value $K \\geq 0$, the probability that the sample estimate for our metric on a dataset of size $n$ deviates its expectation by more than $t$ is bounded above by $\\exp\\left( \\frac{-t^2}{16 K^2} \\right)$ [Gretton et al., 2012]. Consequently, one can in principle link a sample estimate of our metric to guarantees on decision-making (e.g., accurate loss estimation and no regret decisions). However, the primary goal of our regularization approach is to mitigate the tradeoff between calibration and sharpness, not to provide calibration guarantees at training time. If strong calibration guarantees are needed, post-hoc calibration approaches are probably most appropriate (we can use both!). We find empirically that our methods have a positive impact on decision-making performance, and that this benefit is consistent whether or not post-hoc recalibration is performed (see Table 2 in our paper).\n\n> **The trainable regularizer needs to be combined with post-hoc calibration method, which may raise the question how effective the method is. Table 2 shows that the MMD alone does not always provide enough improvement.**\n\nPost-hoc recalibration is an effective strategy for achieving calibration, but usually comes at the cost of reduced sharpness. Regularization approaches to calibration aim to give a better trade-off between calibration and sharpness by jointly optimizing the two metrics. This intuition is supported by our experimental results, where the sharpness penalty for adding post-hoc recalibration is significantly reduced for models trained with our calibration regularization. These trends are supported by other works that regularize for calibration, such as [Kumar et al., 2018; Popordanoska et al., 2022]. Thus, we see post-hoc recalibration and calibration regularization as complementary, with regularization providing improved sharpness and post-hoc methods providing consistent calibration.\n\n> **Table 2 shows training using NLL and the proposed method (individual calibration) and evaluating using both quantile calibration and decision calibration. Why quantile calibration is used here? Why not individual calibration in the evaluation?**\n\nOur intent in reporting quantile calibration and decision calibration was to show that our method can simultaneously achieve strong performance in multiple forms of calibration. We used the individual calibration objective to train these models since this is a stringent constraint that we expect to improve a variety of forms of calibration (we do observe this in practice). We will add the individual calibration metric to the result tables for completeness. \n\n> **What happens if the problem is a classification problem? The evaluation of calibration is classification is usually tricky. How much the trainable regularize can benefit there?**\n\nWe agree that this is interesting! We have added experiments applying our methods to five classification datasets. We find that regularizing with our training objectives improves sharpness (entropy), calibration (ECE), and accuracy relative to MMCE [Kumar et al., 2018] and temperature scaling (see Tables 1 & 2 in the attached PDF, and our general author response).\n\n> **Minor: table caption is usually on top of the table.**\n\nThank you for catching this formatting issue, we have corrected it in the revised paper.\n\n*[Kumar et al., 2018] Kumar, A., Sunita S., and Ujjwal J. Trainable calibration measures for neural networks from kernel mean embeddings. ICML, 2018.*\n\n*[Popordanoska et al., 2022] Popordanoska, T., Sayer, R., Blaschko M., A Consistent and Differentiable Lp Canonical Calibration Error Estimator. NeurIPS, 2022.*\n"}, {"review_id": "N3LFWNGWyb", "reviewer": "Reviewer_cmLZ", "summary": "This paper introduces differentiable kernel-based calibration metrics that unify and generalize popular forms of calibration.\nThe notions of calibration are presented as distribution matching constraints by using various kernels in the Maximum Mean Discrepancy (MMD) metric.\nThrough empirical evaluation, it is demonstrated that incorporating these metrics as regularizers enhances calibration, sharpness and facilitates better decision-making in regression scenarios. \n", "questions": "- The abstract claims that the proposed metric generalizes popular forms of calibration for classification and regression. Could you clarify the relationship of the metric with notions of calibration used in classification, i.e. top-label, marginal and canonical (see e.g. Eq. 1 2 and 3 in [Vaicenavicius et al., 2019, Kull et al., 2019]? \n- What is the computational overhead of the proposed method?\n- How is the bandwidth of the kernels chosen?\n\n*[Vaicenavicius et al. 2019] Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, Thomas B. Schön Evaluating model calibration in classification. In AISTATS 2019.*\n\n*[Kull et al. 2019] Meelis Kull, Miquel Perello-Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, Peter Flach Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration. In NeurIPS 2019*", "limitations": "Limitations are discussed in section 7. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed metrics can be integrated in gradient based optimization alongside proper scoring rules.\n- The paper extends on the discussion regarding the connection between calibration and decision making, which offers valuable insights for practical problems.  \n- The empirical results are sufficiently extensive and verify the effectiveness of integrating the proposed metrics into training objectives, as a means to mitigate calibration error.", "weaknesses": "The related work section does not sufficiently discuss prior works, and fails to highlight the specific advancements made by this paper in comparison. As the current paper proposes a kernel-based differentiable metric to be used as a regularizer alongside a proper scoring rule, there has to be a discussion of existing trainable calibration methods, especially kernel-based methods like [Kumar et al., 2018, Zhang et al., 2020, Popordanoska et al., 2022]. \n\n\n*[Zhang et al., 2020] Jize Zhang, Bhavya Kailkhura, and T Han. Mix-n-Match: Ensemble and compositional methods for uncertainty calibration in deep learning. In ICML 2020*\n\n*[Popordanoska et al., 2022] Teodora Popordanoska, Raphael Sayer, Matthew B. Blaschko A Consistent and Differentiable Lp Canonical Calibration Error Estimator. In NeurIPS 2022*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments! We are glad you appreciated the trainable nature of our metrics, the decision making connections, and the experimental results. We agreed with your suggestions to expand on the applications to classification and to add more detailed descriptions of related work and we have incorporated them into the paper. Responses to your questions and comments can be found below.\n\n> **As the current paper proposes a kernel-based differentiable metric to be used as a regularizer alongside a proper scoring rule, there has to be a discussion of existing trainable calibration methods, especially kernel-based methods like [Kumar et al., 2018, Zhang et al., 2020, Popordanoska et al., 2022].**\n\nWe have updated the related works section to clarify the specific advancements of our work, including with respect to [Kumar et al., 2018, Zhang et al., 2020, Popordanoska et al., 2022]. Please refer to our general author response for details about how our work relates to those works. The biggest differences surround the generality of our approach (the aforementioned methods all focus on a single form of calibration in classification) and the connections we show to decision making. In new experiments, we also find that our method performs favorably to MMCE [Kumar et al., 2018] in the multiclass classification setting, as measured by sharpness (entropy), calibration (ECE), and accuracy.\n\n> **Could you clarify the relationship of the metric with notions of calibration used in classification, i.e. top-label, marginal and canonical?**\n\nWe have added details about how our framework enforces notions of calibration used in classification. See Table 3 in the PDF attached to this response for a description of how our framework incorporates top-label, marginal and canonical calibration in classification (labeled according to [Vaicenavicius et al., 2019] for ease of comparison). See Tables 1 and 2 in the PDF attached to this response for new experimental results in which our method performs favorably with respect to MMCE [Kumar et al., 2018] and temperature scaling without train-time regularization. \n\n> **What is the computational overhead of the proposed method?**\n\nRelative to training purely with the NLL objective, our proposed method requires an average wall clock time per epoch of 1.2x for the 5 regression datasets, and 1.3x for the 5 classification datasets. We will add these results to the updated paper.\n\n> **How is the bandwidth of the kernels chosen?**\n\nIn our experiments, the bandwidth of the kernel is chosen to optimize quantile calibration on held-out data using hyperparameter sweeps. In general, a smaller kernel bandwidth corresponds to less smoothing and requires more data to reliably estimate the training objective. \n\n*[Kumar et al., 2018] Kumar, A., Sunita S., and Ujjwal J. Trainable calibration measures for neural networks from kernel mean embeddings. ICML, 2018.*\n\n*[Popordanoska et al., 2022] Popordanoska, T., Sayer, R., Blaschko M., A Consistent and Differentiable Lp Canonical Calibration Error Estimator. NeurIPS, 2022.*\n\n*[Zhang et al., 2020] Jize Zhang, Bhavya Kailkhura, and T Han. Mix-n-Match: Ensemble and compositional methods for uncertainty calibration in deep learning. ICML, 2020.*\n\n*[Vaicenavicius et al., 2019] Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, Thomas B. Schön Evaluating model calibration in classification. AISTATS 2019.*"}, {"review_id": "NuHPiTA4Ww", "reviewer": "Reviewer_1Qie", "summary": "This paper presents a general framework for train-time calibration based on kernel MMD. The authors demonstrate empirically that choosing good kernels (based on the task at hand) in their framework can lead to improvements in calibration while maintaining sharpness of predictions.\n\nUpdate: I have increased my score to a 6 in light of the authors' responses. I think the empirical portions of the paper could still be improved (ideally comparing to more training-time calibration methods), but I believe the sharpness improvements over MMCE are interesting and useful.", "questions": "- How much longer does it take to train (wall clock time) using the MMD regularization? \n- How important is the choice of output distribution for the model considered? In other words, did you consider predicting the parameters of a distribution other than Gaussian and comparing? \n- As asked in the weaknesses, can we relax the need to differentiably sample from the predictive distribution?\n- In the supplementary material, I believe under distribution calibration it should be $P_{Y \\mid X}(y) \\mid Q_{Y \\mid X} = Q_{Y \\mid X}(y)$.", "limitations": "While the authors do discuss limitations, I feel the limitations mentioned should have received further exposition in the main body of the work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Originality:** While kernel-based calibration methods have been studied in the literature, the perspective provided in this work is original in that the authors show how various different types of calibration can be captured by their MMD framework.\n- **Quality:** The quality of the work is overall good, with appropriate support and examples provided for the claims in the paper.\n- **Clarity:** The paper is mostly well-written. However, there are implementation details regarding the MMD approach that I feel are important and missing (detailed further in weaknesses). \n- **Significance:** This paper adds to the growing body of work showing the benefits of kernel-based methods for improving calibration. However, it is difficult to assess the relative significance of this work (compared to, for example [1]) due to the limited experiments.\n\n[1] https://proceedings.mlr.press/v80/kumar18a.html", "weaknesses": "- **Motivation for Implementation Details:** It would be very helpful to accompany the discussion in Section 4 with an idea of how the conditional kernel MMD is implemented in practice and why a practitioner would make certain decisions. For example, is using a model parameterization like the one in the paper actually that necessary (i.e. predicting parameters of a Gaussian)? Can one not just take a similar approach to [1]?\n- **Experiments:** The experiments in the paper compare to a baseline of training using NLL (and post-hoc calibration), with no comparisons to other train-time calibration approaches. As a result, it is difficult to assess the relative worth of the proposed methodology compared to existing approaches. \n- **Contextualization:** The MMD framework proposed in this paper is very similar to the ideas in [1], and I think the paper would benefit from more direct comparisons to [1] and related prior work that highlight the contributions of this work.\n\nTo summarize, I feel that the lack of implementation details + lack of comparison to other train-time calibration approaches limit the contribution of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback on our work! We are grateful that you appreciated the quality and originality of our work. We believe we have thoroughly responded to each of your comments, and that these changes have improved our work. If you find our responses and the corresponding updates to our paper satisfactory, we would kindly request that you consider adjusting your score.\n\n> **It would be very helpful to accompany the discussion in Section 4 with an idea of how the conditional kernel MMD is implemented in practice and why a practitioner would make certain decisions. For example, is using a model parameterization like the one in the paper actually that necessary (i.e. predicting parameters of a Gaussian)?**\n\nUsing a model that outputs the parameters of a Gaussian distribution is indeed not required. We chose to use Gaussian distributions in our experiments to show how our methods perform in a common setting for distribution prediction in regression. In new experiments and in our general author response, we have described how our methods apply to categorical distributions. What is required is that we can estimate and optimize the MMD expectations with respect to the forecasted distribution. In classification, we completely remove the need for sampling by computing the expectation with respect to the categorical distribution exactly. In regression, any reparameterizable distribution should work nicely. We will add a discussion of the model parameterization and design principles to our revised paper.\n\n>  **It is difficult to assess the relative worth of the proposed methodology compared to existing approaches.**\n\nWe have added experiments comparing our methods to MMCE [Kumar et al, 2018] in the classification setting. See Tables 1 and 2 in the PDF attached to this response and the general response for additional details. We find that our objectives tend to produce better calibration, sharpness, and accuracy scores than MMCE both with and without post-hoc temperature scaling. \n\n> **The paper would benefit from more direct comparisons to [Kumar et al., 2018] and related prior work that highlight the contributions of this work.**\n\nIn addition to the new empirical comparisons, we have expanded the related works section to include a more thorough comparison to [Kumar et al., 2018] and other existing work. See our general author response for a more direct comparison that highlights the contributions of our work.\n\n> **How much longer does it take to train (wall clock time) using the MMD regularization?**\n\nCompared to training with the NLL, our proposed method requires an average of 1.3x wall clock time on the five classification datasets, and 1.2x wall clock time for the five regression datasets. We will add these results to the updated paper.\n\n> **How important is the choice of output distribution for the model considered?**\n\nAs mentioned above, it is not important that the model outputs a Gaussian distribution. In the new experiments with classification, we calibrate categorical distributions. We have found similar results when applying our methods to models that output Gaussian mixture distributions, using Gumbel Softmax relaxation to differentiate through the mixture distribution.\n\n> **Can we relax the need to differentiably sample from the predictive distribution?**\n\nWe can relax the need for differentiable sampling in some cases. In the classification setting, there is no need to differentiably sample from the predictive distribution, since the MMD metric can be analytically computed (we implement this in the new classification experiments). In the regression setting, when a distribution does not admit differentiable sampling (e.g., a Mixture of Gaussians), we can apply a differentiable relaxation (e.g., using Gumbel-Softmax to differentiate with respect to the mixture weights). In practice, several highly expressive predictive distributions exist in the regression setting which can be trained using our method (e.g. Neural Autoregressive Flows [Huang et al, 2018]). In principle, it is also possible to remove the need for differentiable sampling in regression by optimizing our MMD objective using Monte-Carlo RL techniques (e.g. REINFORCE). This is an interesting direction for future research, but beyond the scope of this paper.\n\n> **In the supplementary material, I believe under distribution calibration it should be $P_{Y \\mid X}(y) \\mid Q_{Y \\mid X} = Q_{Y \\mid X}(y)$.** \n \nThank you for catching the error, we have corrected it in the revised paper.\n\n*[Kumar et al., 2018] Kumar, Aviral, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. International Conference on Machine Learning. PMLR, 2018.*\n\n*[Huang et al, 2018] Huang, Chin-Wei, et al. Neural autoregressive flows. International Conference on Machine Learning. PMLR, 2018.*\n\n"}, {"review_id": "8e2sDJzuME", "reviewer": "Reviewer_UW9V", "summary": "Accurate probability prediction is a crucial aspect of a trustworthy model, especially for deep neural networks. This paper focuses on the model calibration problem and proposes a unified framework via distribution matching. To leverage the distribution matching, they used \nMaximum Mean Discrepancy to estimate the miscalibration, where MMD is a popular distribution measure and used in many distribution alignment problems in recent years. The main usage of MMD in this paper is to treat it as a trainable regularizer. In the experiment section, \nthey demonstrate their effectiveness in performing proper distribution alignments.", "questions": "Please refer to the weaknesses.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. They introduced a unified framework for calibration metrics as distribution-matching problems, which is a very interesting view for uncertainty calibration.\n2. They proposed a training-time model calibration method by introducing kernel-based calibration metrics.", "weaknesses": "The paper does not have main weakness, if some, they are:\n\n1. How to apply this method to multi-class classification? I did not see the related experiments. It seems that all the used datasets are regression datasets, but Line 6 said it is also suitable for classification tasks.\n\n2. Deep comparisons with similar kernel methods, like [1]. I did not see the description of their difference. They also used the kernel method to perform model calibration.\n\n[1] Trainable Calibration Measures For Neural Networks From Kernel Mean Embeddings. ICML 2018\n\n3. What are the n and d in Table 2? Why do you use the different n and d regarding different datasets?\n\n4. How to demonstrate the prediction performance improvement using the proposed methods? I did not find the clue in Table 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and feedback! We have focused on improving the related works section to include more direct comparisons to [Kumar et al., 2018] and other kernel-based calibration metrics. We have included updated comparisons in our general author response. We respond to your questions and comments below:\n\n> **How to apply this method to multi-class classification?**\n\nSee Table 3 in the PDF attached to this response for a description of how three popular forms of calibration in classification fit into our framework. In new experiments, we implement the most stringent of the three forms (labeled “canonical” in Table 3 of the PDF). We find that regularizing with our proposed approach achieves better ECE, entropy, and accuracy than when training with the MMCE regularization of [Kumar et al, 2018], or without regularization. We find that these results hold both with and without post-hoc temperature scaling (see Tables 1 and 2 of the PDF attached to this response).\n\n> **Deep comparisons with similar kernel methods, like [Kumar et al., 2018].**\n\nWe have updated the literature review to more clearly indicate how our work is differentiated from existing work, such as [Kumar et al., 2018]. To summarize the main differences to [Kumar et al., 2018] beyond the empirical improvements in Tables 1 and 2 of the attached PDF, MMCE is a special case of our framework applied to one-vs-all confidence calibration in classification. Our method enables enforcing many more forms of calibration (e.g., the 8 regression examples in Appendix A of the paper and the 3 classification examples in Table 3 of the attached PDF). One advantage of this added generality is that we can draw connections to decision making by designing calibration metrics specifically for a decision problem at hand, as in Section 5 of our paper. We will incorporate the comparisons to related work described in our general author response to the revised paper. \n\n\n\n> **What are the n and d in Table 2?**\n\n$n$ is the number of examples in the dataset and $d$ is the number of features. We will update the table caption to clarify this point.\n\n\n\n> **How to demonstrate the prediction performance improvement using the proposed methods?**\n\nWe quantify prediction performance as the combination of calibration and sharpness. We define sharpness with the Negative Log Likelihood (NLL) and calibration with the Quantile Calibration Error (QCE). Note that the NLL is a strictly proper scoring rule so it can also be used as a measure of prediction performance on its own. In Table 2, we find that the best QCE is observed with our method in all 5 datasets, and the best NLL is observed with our method in 4 out of 5 datasets. Additionally, in the new classification experiments we report ECE, entropy, and accuracy (see Tables 1 and 2 in the PDF attached to this response). We find that our method outperforms the baseline NLL method and MMCE [Kumar et al., 2018] in the classification setting as measured by ECE, entropy, and accuracy.\n\n\n\n*[Kumar et al., 2018] Kumar, Aviral, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. International Conference on Machine Learning. PMLR, 2018.*"}]

Author rebuttal:
We thank the reviewers for engaging thoughtfully and providing constructive feedback. We are glad that all reviewers appreciate that this work is “novel” [hWT9], “original” [1Qie], “a very interesting view for uncertainty calibration” [UW9V], and “offers valuable insights for practical problems” [cmLZ].

Our main takeaways from the reviews were that we should (1) clarify the relationship to previous work, (2) add details on how our framework applies in the classification setting, and (3) add more experiments and implementation details for practical guidance. Based on these helpful comments, we have made the following updates to our paper:
1. We significantly expanded the literature review to clarify the relationship between our work and other kernel-based calibration methods. 
2. We derived calibration metrics for classification in our framework (see attached PDF, Table 3) and provide estimators below. These estimators do not require differentiable sampling from the predicted distribution, and demonstrate that our methods apply beyond Gaussian prediction.
3. We performed new experiments on five classification datasets to compare our method to MMCE [Kumar et al., 2018] (see attached PDF, Tables 1 and 2). We found that our method performs favorably in terms of accuracy, calibration (ECE), and sharpness (entropy), both with and without post-hoc temperature scaling. 

We expand on each of these updates in order below. We will carefully address all reviewer comments in the updated paper and we clarify some important questions in individualized responses to the reviewers.

1. **Contextualization with Existing Work**

Multiple reviewers requested clarity on how our work relates to existing kernel-based calibration methods. Below, we briefly discuss each paper raised by the reviewers, and clarify how our work is similar and distinct. In particular, we would like to emphasize generality (we show 11 concrete forms of calibration our framework can enforce) and applications to decision-making as areas that distinguish our work from existing works. We will add these discussions to the revised paper. 

- *[Kumar et al., 2018] Kumar, Aviral, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. International Conference on Machine Learning. PMLR, 2018.*

[Kumar et al., 2018] introduce MMCE, a kernel-based training objective to enforce a one-vs-all confidence calibration measure for multiclass classification. MMCE is a special case of our framework applied to top-label calibration (see attached PDF, Table 3). One key difference between our work and theirs is that we have identified 11 forms of calibration (8 regression, 3 classification) that fit into our framework, as opposed to the one form of classification calibration studied by [Kumar et al., 2018]. Empirically, we find that our methods outperform MMCE in terms of calibration, sharpness, and accuracy (see below).

- *[Popordanoska et al., 2022] Teodora Popordanoska, Raphael Sayer, Matthew B. Blaschko A Consistent and Differentiable Lp Canonical Calibration Error Estimator. In NeurIPS, 2022.*

[Popordanoska et al., 2022] propose a differentiable estimator for distribution calibration exclusively in the classification setting, via kernel density estimation (KDE). Key differences between our method and theirs include: our methods apply to regression problems, we enforce local and group calibration by incorporating conditioning variables (see our Figure 1), and we can tailor our metrics for particular decision problems. In the revised version of our paper, we will include an empirical comparison to [Popordanoska et al., 2022] in the classification setting.


- *[Zhang et al., 2020] Jize Zhang, Bhavya Kailkhura, and T Han. Mix-n-Match: Ensemble and compositional methods for uncertainty calibration in deep learning. In ICML, 2020.*

[Zhang et al., 2020] also consider kernel-based measures of miscalibration. They use kernel density estimates to give a smoothed estimate of the conditional label distribution given a particular prediction in multiclass classification. Our work differs from [Zhang et al., 2020] in that we consider regression problems, we introduce trainable metrics, and we introduce calibration metrics tailored to downstream decision problems. 

2. **Enforcing Calibration in Classification**

See Table 3 of the attached PDF for a description of how our framework expresses calibration in classification tasks. Here, we show how to estimate the MMD metric in a classification setting by marginalizing out the randomness originating from the forecast. This allows us to compute our MMD metric without sampling from the predicted distribution.

Let the label space be $\mathcal{Y}=\\{1,2,\dots,m\\}$. The sample estimate for MMD is

$\widehat{\textrm{MMD}}(\mathcal{F},D,Q) =  \frac{1}{n(n-1)} \sum_{i=1}^{n} \sum_{j \ne i}^{n} h_{ij}$,

where

$$h_{ij} = \sum_{y\in\mathcal{Y}} \sum_{y' \in \mathcal{Y}} q_i(y)q_j(y')k((y,z_i), (y',z_j))-2 \sum_{y \in \mathcal{Y}} q_i(y)k((y,z_i), (y_j,z_j)) + k((y_i,z_i), (y_j,z_j))$$

Here, $q_i(y)$ is the predicted probability of the label $Y=y$ for example $i$. This estimate is differentiable with respect to the predicted label probabilities and can be computed in $O(n^2m^2)$ time, where $m$ is the number of classes and $n$ is the number of datapoints.

3. **New Classification Experiments**

We evaluate the effectiveness of our method on five commonly used UCI classification datasets. In Tables 1 and 2 in the attached PDF, we compare our method’s performance to NLL and MMCE [Kumar et al., 2018]. For model selection and training, we follow the same experimental setup detailed in Section 6.2 of the paper. 

Empirically, we find that our method performs favorably in terms of accuracy, calibration (ECE), and sharpness (entropy). These advantages relative to MMCE are present both with and without post-hoc temperature scaling.


Meta-review: This paper explores the family of training techniques which incorporate calibration metrics as training regularizers. They state a unified view of different calibration metrics, and then propose a kernel-MMD based metric. The proposal performs well experimentally in several regression settings.
Reviewers agree that the method is novel, although fairly similar to Kumar et al. 2018. Further, the paper is well-written, and the experiments are thorough. The authors have responded to many of the reviewer's concerns in the rebuttal. Thus, I recommend acceptance.
I encourage the authors to address the reviewers' comments in the camera-ready. In particular, the paper would be stronger with a clearer comparison to MMCE.
Decision: Accept (poster)