Calibration tests beyond classification
Authors: David Widmann
Fredrik Lindsten
Dave Zachariah
Venue: ICLR 2021 Poster (2021)
Keywords: calibration
uncertainty quantification
framework
integral probability metric
maximum mean discrepancy

Abstract: Most supervised machine learning tasks are subject to irreducible prediction errors. Probabilistic predictive models address this limitation by providing probability distributions that represent a belief over plausible targets, rather than point estimates. Such models can be a valuable tool in decision-making under uncertainty, provided that the model output is meaningful and interpretable. Calibrated models guarantee that the probabilistic predictions are neither over- nor under-confident. In the machine learning literature, different measures and statistical tests have been proposed and studied for evaluating the calibration of classification models. For regression problems, however, research has been focused on a weaker condition of calibration based on predicted quantiles for real-valued targets. In this paper, we propose the first framework that unifies calibration evaluation and tests for general probabilistic predictive models. It applies to any such model, including classification and regression models of arbitrary dimension. Furthermore, the framework generalizes existing measures and provides a more intuitive reformulation of a recently proposed framework for calibration in multi-class classification. In particular, we reformulate and generalize the kernel calibration error, its estimators, and hypothesis tests using scalar-valued kernels, and evaluate the calibration of real-valued regression
problems.

CALIBRATION TESTS BEYOND CLASSIFICATION
David Widmann
Fredrik Lindsten
Department of Information Technology Uppsala University, Sweden david. widmann@ic uu.se
Division of Statistics and Machine Learning Linköping University, Sweden fredrik. Lindsteneeie 1ii..
Dave Zachariah
Department of Information Technology Uppsala University, Sweden dave.
.
ABSTRACT
Most supervised machine learning tasks are subject to irreducible prediction errors. Probabilistic predictive models address this limitation by providing probability distributions that represent a belief over plausible targets, rather than point esti- mates. Such models can be a valuable tool in decision-makinn under uncertainty, provided that the model output is meaningful and interpretable. Calibrated models guarantee that the probabilistic predictions are neither over- nor under-confident. In the machine learning literature, different measures and statistical tests have been proposed and studied for evaluating the calibration of classification models. For regression problems, however, research has been focused weaker condition of calibration based on predicted quantiles for real-valued targets. this paper, we propose the first framework that unifies calibration evaluation and tests for general probabilistic predictive models. It applies to any such model, including classifica- tion and regression models of arbitrary dimension. Furthermore, the framework generalizes existing measures and provides a more intuitive reformulation of a recently proposed framework for calibration in multi-class classification. In par- ticular, we reformulate and generalize the kernel calibration error, its estimators, and hypothesis tests using scalar-valued kernels, and evaluate the calibration real-valued regression problems]
INTRODUCTION
We consider the general problem of modelling the relationship between a feature X and target Y in probabilistic setting, i.e., we focus on models that approximate the conditional probability distribution P(Y]X) of target Y for given feature X. The use of probabilistic models that output probability distribution instead of point estimate demands guarantees on the predictions beyond accuracy, enabling meaningful and interpretable predicted uncertainties. One such statistical guarantee is calibration, which has been studied extensively in metercological and statistical literature (DeGroot Fienberg 1983 Murphy & Winkler 1977).
A calibrated model ensures that almost every prediction matches the conditional distribution of targets given this prediction. Loosely speaking,
classification setting predicted distribution of the
model is called calibrated (or reliable). if the empirically observed frequencies of the different classes match the predictions the long run, the same class probabilities would be predicted repeatedly. classical example is weather forecaster who predicts each day if it is going to rain on the next day. she predicts rain with probability 60% for a long series of days, her forecasting model is calibrated for predictions 60% if it actually rains on 60% of these days.
this property holds for almost every probability distribution that the model outputs, then the model is considered to be calibrated. Calibration is an appealing property of probabilistic model since it
'The source code of the experiments is available at httpa://gi htt.///ttthttcco/g/ http:///ggttuub...co//e/ecrooooo com/
Ca 1 1ibratt inn_ io_nionn2221111

---

provides safety guarantees on the predicted distributions even in the common case when the model does not predict the true distributions X)) Calibration, however, does not guarantee accuracy (or refinement)- model that always predicts the marginal probabilities of each class is calibrated but probably inaccurate and of limited use. On the other hand, accuracy does not imply calibration either since the predictions of an accurate model can be over-confident and hence miscalibrated, as observed, e.g., for deep neural networks (Guo et al. 2017).
In the field of machine learning, calibration has been studied mainly for classification prob- lems (Bröcker 2009; Guo et al., [017;; Kull et al., 2017; 2019; Kumar et al. 2018 Platt 2000
Vaicenavicius et 2019; Widmann al. 2019 Zadrozny 2002) and for quantiles and confidence intervals of models for regression problems with real -valuv targets (Fasiolo et al. [2020] Ho & Lee
2005, Kuleshov al., 2018 Rueda et al., 2006; Taillardat et al. 2016). In our work, however, we do
not restrict ourselves to these problem settings but instead consider calibration for arbitrary predictive models. Thus, we generalize the common notion of calibration as:
Definition 1. Consider a model Px P(Y X) of conditional probability distribution P( Then model P is said to be calibrated and only
P() Px
almost surely.
If P is classification model. Definition coincides with the notion of (multi-class) calibration by Bröcker (2009); Kull et al. (2019); Vaicenavichus et al. (2019). Alternatively, in classification some
authors (uuo et al. 2017] Kumar et al. | et al. (013) study the strictly weaker property
of confidence calibration Kull et al. 2019), which only requires
P( arg max Px| max Px) = max Px
almost surely.
This notion of calibration corresponds to calibration according to Definition ! for reduced problem with binary targets y == L(Y arg max Px) and Bernoulli distributions Px = Ber(max Px) ) as probabilistic models.
For real-valued targets, Definition coincides with the so-called distribution-leva calibration by Song et al. (019)). Distribution-level calibration implies that the predicted quantiles are calibrated, i.e., the outcomes for all real-valued predictions of the, e.g.. 75% quantile are actually below the predicted quantile with 75% probability (Song et al. 2019 Theorem 1). Conversely, although quantile- based calibration is a common approach for real- vellud regression problems (Fasiolo et al., 2020 Ho & Lee
2005) Kuleshov et al.) 2018 |ueda et al., 20066] Taillardat et al., 2016), it provides weaker guarantees on the predictions. For instance, the linear regression model in Fig. T empirically shows quantiles that appear close to being calibrated albeit being uncalibrated according to Definition]
a
b
1.00
1.00
ideal
cumubetive probebility
datt
-075
donsky
0.50-
0.25
-2
0.00
0.00
-1.0
.0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
0.00
0.25
0.50
0.75
1.00
quantie leval
Figure 1: Illustration of conditional distribution P(Y]X) with scalar feature and target. We consider Gaussian predictive model P. obtained by ordinary least squares regression with 100 training data points (orange dots). Empirically the predicted quantiles on 50 validation data points appear close to being calibrated, although model P is uncalibrated according to Definition [ Using the framework in this paper, on the same validation data a statistical test allows us to reject the null hypothesis that model P is calibrated at a significance level of a 0.05 (p < 0.05). See Appendix A.1 for details.
Figure also raises the question of how to assess calibration for general target spaces in the sense of Definition [ without having to rely on visual inspection. In classification, measures of calibration such the commonly used expected calibration error (ECE) (Guo et al. 2017 Kull et al. 2019
2

---

Nacini et al. 2015 Vaicenavicius al. 2019) and the maximum calibration error (MCE) NNainn et al. 2015) try to capture the average and maximal discrepancy between the distributions on the left hand side and the right hand side of Eq. ① or Eq. (2), respectively. These measures can be generalized to other target spaces (see Definition B. but unfortunately estimating these calibration errors from observations of features and corresponding targets is problematic. Typically, the predictions are different for (almost) all observations, and hence estimation of the conditional probability IP (X)x)) which is needed in the estimation of ECE and MCE, is challenging even for low-dimensional target spaces and usually leads to biased and inconsistent estimators Vaicenavicius et al. 2019).
Kernel-based calibration errors such as the maximum mean calibration error (MMCE) (Kumar et al. 2018) and the kernel calibration error (KCE) (Widmann et al., (019) for confidence and multi-class calibration, respectively, can be estimated without first estimating the conditional probability and hence avoid this issue. They are defined as the expected value of a weighted sum of the differences of the left and right hand side of Eq. (T) for each class, where the weights are given as a function of the predictions (of all classes) and chosen such that the calibration error is maximized. A reformulation with matrix-valued kernels (Widmann et al. 2019) yields unbiased and differentiable estimators without explicit dependence on P(Y[Px), which simplifies the estimation and allows to explicitly account for calibration in the training objective (Kumar et al. (2018). Additionally, the kernel- framework allows the derivation of reliable statistical hypothesis tests for calibration in multi-class classification (Widmann et al. 2019).
However, both the construction as a weighted difference of the class-wise distributions in Eq. (1 and the reformulation with matrix-valued kernels require finite target spaces and hence cannot be applied to regression problems. To be able to deal with general target spaces, we present a new and more general framework of calibration errors without these limitations.
Our framework can be used to reason about and test for calibration of any probabilistic predictive model. As explained above, this is in stark contrast with existing methods that are restricted to simple output distributions, such classification and scalar-valued regression problems. A key contribution of this paper is new framework that is applicable to multivariate regression, as well as situations when the output of different (e.g., discrete ordinal) or more complex (e.g., with clear practical implications.
type,
Within this framework a KCE for general target spaces is obtained. We want to highlight that for multi-class classification problems its formulation is more intuitive and simpler to use than the measure proposed by Widmann et al. (219)) based on matrix-valuee kernels. To ease the application of the KCE we derive several estimators of the KCE with subquadratic sample complexity and their asymptotic properties in tests for calibrated models, which improve on existing estimators and tests in the two-sample test literature by exploiting the special structure of the calibration framework. Using the proposed framework, we numerically evaluate the calibration of neural network models and ensembles of such models.
2 CALIBRATION ERROR: A GENERAL FRAMEWORK
In classification, the distributions on the left and right hand side of Eq. ① can be interpreted as vectors in the probability simplex. Hence ultimately the distance measure for ECE and MCE (see Definition B.I can be chosen as a distance measure of real-valued vectors. The total variation, Euclidean, and squared Euclidean distances are common choices (Guo et al. 2017: Kull et al. 2019: Vaicenavicius al. 2019). However, in general setting measuring the discrepancy between P[Y Px) and Px cannot necessarily be reduced to measuring distances between vectors. The conditional distribution P(Y Px) can be arbitrarily complex, even if the predicted distributions are restricted to simple class of distributions that can be represented as real-valued vectors. Hence in general we have to resort to dedicated distance measures of probability distributions.
Additionally, the estimation of conditional distributions [Px is challenging, even more so than in the restricted case of classification, since in general these distributions can be arbitrarily complex. To circumvent this problem, we propose use the following construction: We define random variable Z ~ Px obtained from the predictive model and study the discrepancy between the joint distributions of the two pairs of random variables (Px, Y) and (Px, Zx respectively, instead of
3

---

the discrepancy between the conditional distributions P(Y and Px. Since
(Px, dh (Px Ex) if and only if P(Y|Px)= = Px almost surely,
model P is calibrated if and only if the distributions of (Px, Y) and (Px, 7x) are equal.
The random variable pairs (Px. Y) and (Px. Zx) take values in the product space P x V. where P is the space of predicted distributions P'x and y is the space of targets Y. For instance, classification, P could be the probability simplex and y the set of all class labels, whereas in the case of Gaussian predictive models for scalar targets could be the space of normal distributions and y be R.
The study of the joint distributions of (Px. Y) and (Px. Zx) motivates the definition of a generally applicable calibration error as an integral probability metric (Müller 1997, /Sriperumbudur et al. [2009] 2012) between these distributions. In contrast to common f-divergences such as the Kullback- Lelblec Leiblr divergence, integral probability metries do not require that one distribution is absolutely continuous with respect to the other, which cannot be guaranteed in general.
Definition 2. Let y denote the space of targets Y, and the space of predicted distributions Px. We define the calibration eπror with respect to space of functions F of the form f: P ) - R as
CE == sup Epx. f(Px Y)- - Epx 2x f(Px, Z JEF
(3)
By construction, if model P is calibrated, then CE₃= = 0 regardless of the choice of F. However, the converse statement is not true for arbitrary function spaces F. From the theory of integral probability metrics (see, e.g.. Müller 1997. Sriperumbudur et al. |209 2012). we know that for certain choices of F the calibration error in Eq. (3) is a well- metric on the product space P x V, which implies that CE2 = O if and only if model P is calibrated. Prominent examples include the maximum mean discrepancy (MMD) Gretton et al. 2007), the total variation distance, the Kantorovich distance,
and the Dudley metric (Dudley 1989 p. 310).
As pointed out above, Definition 2 is generalization of the definition for multi-class classification proposed by Widmann et al.] (2019) which is based on vector-valued functions and only applicable to finite target spaces- to any probabilistic predictive model. In Appendix E we show this explicitly and discuss the special case of classification problems in more detail. Previous results (Widmann et al. 2019) imply that in classification MMCE and, for common distance measures d(:;-) such as the total variation and squared Euclidean distance, ECE, and MCE, are special cases of CEy. In Appendix G we show that our framework also covers natural extensions of ECE2 and MCE, to countably infinite discrete target spaces, which our knowledge have not been studied before and occur, e.g., in Poisson regression.
The literature of integral probability metrics suggests that we can resort to estimating CE,= from i.i.d. samples from the distributions of (Px, Y) and (Px. Zx). For the MMD, the Kantorovich distance. and the Dudley metric tractable strongly consistent empirical estimators exist (Sriperumbudur et al.) 2012). Here the empirical estimator for the MMD is particularly appealing since compared with the other estimators "it is computationally cheaper, the empirical estimate converges at faster rate the population value, and the rate of convergence is independent of the dimension d of the space (for =d) (Sriperumbudur et al. (0112)).
Our specific design of (Px, 7x) can be exploited to improve on these estimators. If ff Zz) can be evaluated analytically for a fixed prediction Pa, then CEF can be es- timated empirically with reduced variance by marginalizing out Zx. Otherwise f(P' Za) has to be estimated, but in contrast to the common estimators of the integral probability metrics discussed above the artificial construction of Zx allows us approximate it by numerical integration methods such as (quasi) Monte Carlo integration or quadrature rules with arbitrarily small error and variance. Monte Carlo integration preserves statistical properties of the estimators such as unbiasedness and consistency.
"s discuss in Section
the MMD is metric if and only if the employed kemnel is characteristic.
4

---

3 KERNEL CALIBRATION ERROR
For the remaining parts of the paper we focus the MMD formulation of CEz due to the appealing properties of the common empirical estimator mentioned above. We derive calibration-specifi analogues of results for the MMD that exploit the special structure of the distribution of (Px. Zx) to improve on existing estimators and tests in the MMD literature. To the best of our knowledge these variance-reduced estimators and tests have not been discussed in the MMD literature.
Let k: (P x V) x (P x )) - R be a measurable kernel with corresponding reproducing kernel Hilbert space (RKHS) H. and assume that
Px 11/2 (IPx
< and Ex ((Px, (Px Ex)) co.
We discuss how such kernels can be constructed a generie way in Section 3.1 below.
Definition 3. Let F& denote the unit ball in H. i.e., F == {ff E Hlllflx s 1}. Then the kernel calibration error (KCE) with respect kernel k is defined as
KCEg == CEE, sup feFx
E - Exx [(xx, 2x)
As known from the MMD literature, more explicit formulation can be given for the squared kernel calibration error SKCE, KCF (see Lemma B.2) similar explicit expression for SKCF, was obtained by Widmann et al. (2019) for the special case of classification problems. However, their expression relies on y being finite and is based on matrix-valued kernels over the finiteddimensional probability simplex P. A key difference to the expression in Lemma B.2 is that we instead propose use real-valued kernels defined on the product space of predictions and targets. This construction is applicable to arbitrary target spaces and does not require ) to be finite.
3.1 CHOICE OF KERNEL
The construction of the product space P x "suggests the use of tensor product kernels k kp ky. where kp: P x P + R and ky: } x y + IR are kernels on the spaces of predicted distributions and targets, respectively
By definition, so-called characteristic kernels guarantee that KCE = 0 if and only if the distributions of (Px. Y) and (Px, Zx) are equal (Fukumizu al., 2004 2008). Many common kernels such as the Gaussian and Laplacian kernel on R are characteristic (Fukumizu et al. 2008 Szabó & Sriperumbudur 20188, Theorem 4) showed that tensor product kernel kip @ ky is characteristic if kp and ky are characteristic, continuous, bounded, and translation-invant the implication does not hold for general characteristic kernels (Szabó & Sriperumbudur 2018 Example 1). For calibration evaluation, however, it is sufficient to be able to distinguish between the conditional distributions Px) and (P Px) Px- Therefore, in contrast to the regular MMD setting, it is sufficient that kernel ky is characteristic and kernel kp is non-zero almost surely, to guarantee that KCE = if and only if model P is calibrated. Thus it is suggestive to construct kernels on general spaces of predicted distributions
kernels on Rd but
kpp (p. p') = exp (- (p.p'))
(4)
where .))(:.) is metric P and v, >>>000 O are kernel hyperparameters. The Wasserstein distance is a widely used metric for distributions from optimal transport theory that allows to lift ground metric on the target space and possesses many important properties (see, e.g., Peyré & Cuturi 2019
Chapter 2.4). In general, however, it does not lead to valid kernels kp, apart from the notable exception of elliptically contoured distributions such as normal and Laplace distributions (Peyré & Cuturi 2019 Chapter 8.3).
As mentioned above, framework rephrases and generalizes the construction used by Widmann et al. 2019). The matrix-valued kernels that they employ can recovered by setting kp a Laplacian kernel the probability simplex and ky y/) 5y.0"
'For a general discussion about characteristic kemelss and their relation to universal kemelss we refer to the
paper by (Sriperumbudur al. 2011)
5

---

In machine learning, common probabilistic predictive models output parameters of distributions such as mean and variance of normal distributions. Naturally these parameterizations give rise to injective mappings o: P > IR that can be used to define Hilbertian metric
dp(p,p') = 11((p) - <(1') 1l1
For such metrics, kp in Eq. (4 is a valid kernel for all X > O and v E (0, 2] (Berg et al. 1984 Corollary 3.3.3, Proposition 3.2.7). In Appendix [D33 we show that for many mixture models, and hence model ensembles, Hilbertian metrics between model components can be lifted to Hilbertian metries between mixture models. This construction a generalization of the Wasserstein-like distance for Gaussian mixture models proposed by [Chen et al. (2019) 2020 Delon & Desolneux (2020).
3.2 ESTIMATION
Let Yi)...., (Xn, Yn) be a data set of features and targets which are i..d. according to the law of Y), Moreover, for notational brevity, for (p. (p' (p',v') y²) E P Y we let
'p (( v), (p', 2')) Zz~ k(p, 2), (p', Z*)) ((p, v), (p',v') v/) ((p, v). (F'.v))
Ez Ez-p h Z), (p',v))
Note that contrast to the regular MMD we marginalize out Z and Z'. Similar to the MMD, there exist consistent estimators of the SKCE, both biased and unbiased.
Lemma 1. The plug-in estimator of SKCE& is non-negatively biased. It is given by
SKCE3 Σ M((Px, Yi). (Px
1 n² 1J=1
Inspired by the block tests for the regular MMD (Zaremba al. 2013), we define the following class of unbiased estimators. Note that in contrast to SKCEg they do not include terms of the form ((Px,, Yi), (Px,..))
Lemma 2. The block estimator of SKCE, with block size B E (2....,nk, given by
Ln/B]
SKCEg
B
Σ
4((Px, X2)).
2
(-I))BisiSBB
an unbiased estimator SKCE,
The extremal estimator with = n is so-called U-statistic of SKCE, (Hoeffding 1948 van der Vaart 1998), and hence is the minimum variance unbiased estimator. All presented estimators consistent, i.e., they converge to SKCE; almost surely as the number n of data points goes to infinity, The sample complexity of SKCE, and SKCE&, B is O(n²) and O(Bn). respectively.
3.3 CALIBRATION TESTS
A fundamental issue with calibration errors in general, including ECE, is that their empirical estimates do not provide an answer to the question if a model actually calibrated. Even if the measure is guaranteed to be zero if and only if the model is calibrated, usually the estimates of calibrated models are non-zero due to randomness in the data and (possibly) the estimation procedure. In classification, statistical hypothesis tests of the null hypothesis
Ho: model calibrated,
so-called calibration tests, have been proposed as tool for checking rigorously P cali- brated (Bröcker & Smith 2007 Vaicenavicius et al. 2019 Widmann al. 2019). For multi-class classification, Widmann et al. (2019) suggested calibration tests based on the asymptotic distributions of estimators of the previously formulated KCE. Although for finite data sets the asymptotic distri- butions are only approximations of the actual distributions of these estimators, in their experiments with 10 classes the resulting p-value approximations seemed reliable whereas p-values obtained by

---

so-called consistency resampling (Bröcker & Smith 2007 Vaicenavicius al. 2019)
underestimated
the p-value and hence rejected the null hypothesis too often (Widmann et al. 2019).
For fixed block sizes v
SKCEa) 4 (0) n 0o, and, under Ho.
(Z, 1) as n - DC, where Z are independent xi distributed random variables. See Appendix B for details and definitions of the involved constants. From these results one can derive calibration tests that extend and generalize the existing tests for classification problems, as explained in Remarks B.1 and B.2) Our formulation illustrates also the close connection of these
nSKCE 4. Ei=1 X;
tests to different two-sample tests (Gretton et al. 2007 Zaremba et al. 2013).
4 ALTERNATIVE APPROACHES
For two-sample tests, Chwialkowski et al. (2013) suggested the use of the so-called unnormalized mean embedding (UME) overcome the quadratic sample complexity of the minimum variance unbiased estimator and its intractable asymptotic distribution. As we show in Appendix C there exists an analogous measure of calibration, termed unnormalized calibration mean embedding (UCME), with corresponding calibration mean embedding (CME) test.
As an alternative our construction based on the joint distributions of (Px, Y) and (Px, Zx), one could try to directly compare the conditional distributions P(Y)Px) and P(Zx|Px) = Px. For instance, [Ren al.) (2016) proposed the conditional MMD based on the so-called conditional kernel mean embedding (Song et al. 2009 2013). However, as noted by Park & Muandet) (2020). its common definition as operator between two RKHS based on very restrictive assumptions, which are violated in many situations (see, e.g.. Fukumizu et al. 2013] Footnote 4) and typically require regularized estimates. Hence, even theoretically, often the conditional MMD is "not an exact measure of discrepancy between conditional distributions" (Park & Muandet 2020)). In contrast, the maximum conditional mean discrepancy (MCMD) proposed in a concurrent work by Park & Muandet (2020) is a random variable derived from much weaker measure-theoretiinn assumptions. The MCMD provides a local discrepancy conditional on random predictions whereas KCE is a globbal real-valued summary of these local discrepancies 5
EXPERIMENTS
In our experiments we evaluate the computational efficiency and empirical properties of the proposed calibration eгrоr estimators and calibration tests on both calibrated and uncalibrated models. By means of a classic regression problem from statisties literature, we demonstrate that the estimators and tests can be used for the evaluation of calibration of neural network models and ensembles of such models. This section contains only an high-level overview of these experiments to conserve space but all experimental details are provided in Appendix A
5.1 EMPIRICAL PROPERTIES AND COMPUTATIONAL EFFICIENCY
We evaluate error, variance, and computation time of calibration error estimators for calibrated and uncalibrated Gaussian predictive models in synthetic regression problems. The results empirically confirm the consistency of the estimators and the computational efficiency of the estimator with block size B- = 2 which, however, comes at the cost of increased error and variance.
Additionally, we evaluate empirical test errors of calibration tests at fixed significance level a 0.05. The evaluations, visualized in Fig. for models with ten-dimensional targets, demonstrate empirically that the percentage of incorrect rejections of Ho converges to the set significance level as the number of samples increases. Moreover, the results highlight the computational burden of the calibration test that estimates quantiles of the intractable asymptotic distribution of nSKCE by bootstrapping.
'In our calibration setting, the MCMD is almost surely equal to sup |EIPx (f(Y))Px) Ezxirx (f(2)) where Fy == {f: + |l Σ 1} for an RKHS Hy with kernel ky: y y + R. If kernel By is characteristic, MCMD = 0 almost surely if and only if model P is calibrated (Park & Muandet 2020 Theorem 3.7). Although the definition of MCMD only requires kernel ky on the target space, kernel kp on the space of predictions has to he specified for the evaluation of its regularized estimates.
7

---

As expected, due to the larger variance of SKCE2,2 the test with fixed block size B 2 shows decreased test power although being computationally much more efficient.
calibrated model
type II error type eerror
emplrical test error
0.0
0.2
SKCE
0.00
SKCE vn)
uncalibrated model
SKCE (B
0.75
0.50
significance level
9-
10
10-
samples
time [s]
Figure 2: Empirical test errors for 500 data sets of n E (4, 16, 64, 256, 1024] samples from models with targets of dimension d 10. The dashed black line indicates the set signficance level 0.05.
5.2 FRIEDMAN 1 REGRESSION PROBLEM
The Friedman 1 regression problem (Friedman) 1979 1991 Friedman et al. 1983 is a classic non-linear regression problem with ten-dimensional features and real-valued targets with Gaussian noise. We train a Gaussian predictive model whose mean is modelled by shallow neural network and single scalar variance parameter (consistent with the tata-generating model) ten times with different initial parameters. Figure 3 shows estimates of the mean squared error (MSE), the average negative log eegliikllhood (NLL), SKCE,, and p-value approximation for these models and their ensemble on the training and a separate test data set. All estimates indicate consistently that the models are overfit after 1500 training iterations. The estimations SKCE, and the p-values allow to focus on calibration specifically, whereas MSE indicates accuracy only and NLL, any proper scoring rule (Bröcker 2009), provides a summary of calibration and accuracy. The estimation of SKCE, in addition to NLL could serve another source of information for early stopping and model selection.
MSE
NLL
2m844
1003
#mn13 0)
training
1043
WMn115 5)
models
estimate
5Kh00 01
ensemble
SKCE (biased)
p-value
10
models
10
ensemble
10
iteration
Figure 3: Mean squared error (MSE), average negative log-likelihood (NLL), SKCE (SKCE (biased)), and p-value approximation (p-value) of ten Gaussian predictive models for the Friedman regression problem versus the number of training iterations. Evaluations the training data set (100 samples) are displayed in green and orange, and on the test data set (50 samples) in blue and purple. The green and blue line and their surrounding bands represent the mean and the range of the evaluations of the ten models. The orange and purple lines visualize the evaluations of their ensemble.
8

---

6 CONCLUSION
We presented a framework of calibration estimators and tests for any probabilistic model that captures both classification and regression problems of arbitrary dimension as well as other predictive models. We successfully applied it for measuring calibration of (ensembles of) neural network models.
Our framework highlights connections of calibration to two-sample tests and optimal transport theory which we expect to be fruitful for future research. For instance, the power of calibration tests could be improved by heuristics and theoretical results about suitable kernel choices or hyperparameters (cf. Jitkrittum et al.) 20T6). It would also be interesting to investigate alternatives to KCE captured by our framework, c.g., by exploiting recent advances in optimal transport theory (cf. Genevay et al. (016).
Since the presented estimators of SKCE, are differentiable, we imagine that our framework could be helpful for improving calibration of predictive models, during training (cf. Kumar et al., 2018) or G.g..Guo post-hoc. Currently, many calibration methods (see, e.g., Guo al. 2017 Kull al. [2019] [Sog
ctt al.) 2019) are based on optimizing the log-likelihood since it is a strictly proper scoring rule and thus encourages both accurate and reliable predictions. However,
for any proper scoring rule,
"Per se, it is impossible to say how the score will rank unreliable forecast schemes [...]. The lack of reliability of one forecast scheme might be outbalanced by the lack of resolution of the other" [Bröcker] (2009)). In other words, if one does not use a calibration method such as temperature scaling (Guo al. 2017) that keeps accuracy invarian it unclear if the resulting model trading off calibration for accuracy when using log- liglliill for re-calibration. Thus hypothetically flexible calibration methods might benefit from using the presented calibration error estimators.
ACKNOWLEDGMENTS
We thank the reviewers for all the constructive feedback on our paper. This research is financially supported by the Swedish Research Council via the projects Learning of Large-Scale Probabilistic Dynamical Models (contract number: 2016-04278), Counterfactual Prediction Methods for Heteroge- neous Populations (contract number: 201 2018-05040), and Handling Uncertainty in Machine Learning Systems (contract number: 2020-04122), by the Swedish Foundation for Strategic Research via the project Probabilistic Modeling and Inference for Machine Learning (contract number: ICA16-0015), by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenherg Foundation, and by ELLIIT.
REFERENCES
M. A. Arcones and E. Giné. On the bootstrap of U and V statistics. The Annals of Statistics, 20(2): 655-674, 1992.
C. Berg. J. P. R. Christensen, and P. Ressel. Harmonic Analysis on Semigroups. Springer New York, 1984.
Bröcker and I., A. Smith. Increasing the reliability reliability diagrams. Weather and Forecasting, 22(3):651 -611, June 2007.
Jochen Bröcker. Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Society, 135(643):5512-1111 July 2009.
Y. Chen, T. T. Georgiou, and A. Tannenbaum. Optimal transport for Gaussian mixture models. IEEE Access, 7:6269-622 -2288, 2019.
Y. Chen, J. Ye, and J. Li. Aggregated Wasserstein distance and state registration for hidden Markov models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 4239):2133-2147, September 2020.
K. Chwialkowski, A. Ramdas, D. Sejdinovic, and A. Gretton. Fast two-sample testing with analytic representations of probability measures. In Proceedings of the 28th International Conference on Neural Information Processing Systems, pp. 1981-1989, Cambridge, MA, USA, 2015. MIT Press.
Temmerature scaling he defined and applied for general probabilistic predictive models, see Appendix

---

M. H. DeGroot and S. E. Fienberg. The comparison and evaluation of forecasters. The Statistician, 32(1/2):12, March 1983.
C. Deledalle, S. Parameswaran, and T. Q. Nguyen. Image denoising with generalized Gaussian mixture model patch priors. SIAM Journal on Imaging Sciences, 11(4):256-220) 2209,, January 2018.
Delon and A. Desolneux. A Wasserstein-type distance the space of Gaussian mixture models. SIAM Journal on Imaging Sciences, 13(2):936977,, January 2020.
R. M. Dudley. Real analysis and probability. Wadsworth Brooks/Cole Pub. Co, Pacific Grove, Calif, 1989.
M. Fasiolo, S. N. Wood, M. Zaffran, R. Nedellec, and Y. Goude. Fast calibrated additive quantile regression. Journal of the American Statistical Association, pp. 1-11, March 2020.
J. H. Friedman. A tree-structured approach to nonparametric multiple regression. In Lecture Notes in Mathematics, pp. 5-22. Springer Berlin Heidelberg, 1979.
H. Friedman. Multivariate adaptive regression splines. The Annals of Statistics, 19(1):1677 1991.
J. H. Friedman, E. Grosse, and W. Stuetzle. Multidimensional additive spline approximation. SIAM Journal on Scientific and Statistical Computing, 4(2):291-301, June 1983.
K. Fukumizu, F. R. Bach, and M. Jordan. Dimensionality reduction for supervised learning with reproducing kernel Hilbert spaces. Journal of Machine Learning Research, 5(Ian):73-99, 2004.
K. Fukumizu, A. Gretton, X. Sun, and B. Schölkopf. Kernel measures of conditional dependence. In Advances in Neural Information Processing Systems 20, pp. 489-49946 2008.
K. Fukumizu, L. Song, and A. Gretton. Kernel Bayes' rule: Bayesian inference with positive definite kernels. Journal of Machine Learning Research, 14(82)3755-3777 -37833 2013.
M. Gelbrich. On a formula for the 1 Wasserstein metric between measures on Euclidean and Hilbert spaces. Mathematische Nachrichten, 147(1):185-200 1990.
A. Genevay, M. Cuturi, G. Peyré, and F. Bach. Stochastic optimization for large-scale optimal transport. In Advances in Neural Information Processing Systems 29, pp. 1440-3448 2016.
X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 249 256..25..77 PMLR, 5 2010.
E. Gómez, M. A. Gómez-Villeass, exponential family of distributions. Communications in Statistics Theory and Methods, 27(3): 589-600, January 1998.
Viilegas, and M. Marín. A multivariate generalization of the power
E. Sattrr--nnnneee--aaaaa M. A. Gómez- VönezzVillegas, and J. M. Marín. Multivariate exponential power distributions mixtures of normal distributions with Bayesian applications. Communica- fions in Statistics - Theory and Methods, 37(6):72-9988 985, February 2008
A. Gretton, K. Borgwardt, M. Rasch, B. Schölkopf, and A. J. Smola. A kernel method for the two-sample-problem. Advances Neural Information Processing Systems 19, pp. 513-520. 2007.
A. Gretton, K. Fukumizu, Z. Harchaoui, and B. K. Sriperumbudur. A fast, consistent kernel two- sample test. In Advances in Neural Information Processing Systems 22, pp. 673-681. 2009.
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1321-1330. PMLR, 8 2017.
F. K. Gustafsson, M. Danelljan, and T. B. Schön. Evaluating scalable Bayesian deep learning methods for robust computer vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020.
10

---

Y. H. S. Ho and S. M. S. Lee. Calibrated interpolated confidence intervals for population quantiles. Biometrika, 92(1):234 241, March 2005.
W. Hoeffding. A class of statistics with asymptotically normal distribution. The Annals of Mathemat- ical Statistics, 19(3):99--325, September 1948.
H. Hotelling. The generalization of student's 's ratio. The Annals of Mathematical Statistics, 2(3): 360-378, August 1931.
M. Innes. Flux: Elegant machine learning with Julia. Journal of Open Source Software, 3(25):602. May 2018.
M. Innes, E. Saba, K. Fischer, D. Gandhi, M. C. Rudilosso, N. M. Joy, T. Karmali, A. Pal, and V. Shah. Fashionable modelling with Flux, 2018.
W. Jitkrittum, Z. Szabó, K. P. Chwialkowski, and A. Gretton. Interpretable distribution features with maximum testing power. In Advances in Neural Information Processing Systems 29, pp. 181-189. 2016.
N. L. Johnson, S. Kotz, and N. Balakrishnan. Continuous univariate distributions: Vol. J. Wiley, New York, 2nd edition, 1994.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
V. Kuleshov, N. Fenner, and S. Ermon. Accurate uncertainties for deep learning using calibrated regression. In Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2796-2804. PMLR, 7 2018.
M. Kull, T. Silva Filho, and P. Flach. Beta calibration: well-founded and easily implemented improvement on logistic calibration for binary classifiers. In Proceedings of the 20th International Conference on A Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pp. 623-631. PMLR, 4 2017.
M. Kull, M. Perello Nieto, M. Kängsepp. T. Silva Filho, H. Song, and P. Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with Dirichlet calibration. In Advances in Neural Information Processing Systems 32, pp. 12316-12328 2019.
A. Kumar, S. Sarawagi, and U. Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2805 2814. PMLR, 2018.
A. M. Mathai and S. B. Provost. Quadratic forms in random variables: Theory and applications, volume 126. M. Dekker, New York, 1992.
C. A. Micchelli and M. Pontil, On learning vector-valued functions. Neural Computation, 17(1): 177-204, January 2005.
A. Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2):429 443, June 1997.
A. H. Murphy and R. L. Winkler. Reliability of subjective probability forecasts of precipitation and temperature. Applied Statistics, 26(1):41, 1977.
M. P. Naeini, Cooper, and M. Hauskrecht. Obtaining well calibrated probabilities using Bayesian binning. In AAAI Conference on Artificial Intelligence, 2015.
Park and K. Muandet. A thesurc-th approach to kernel conditional mean embeddings. In Advances in Neural Information Processing Systems, volume 33, pp. 21247-21259, 2020.
G. Peyré and M. Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning, 1 1(5-6))355-607, 2019.
J. Platt. Probabilities for SV Machines, pp. 61-73. MIT Press, 2000.

---

Y. Ren, J. Zhu, J. Li, and Y. Luo. Conditional generative moment- mathhing networks. In Advances in Neural Information Processing Systems 29, pp. 2928-2936. 2016.
M. Rueda, S. Martinez-Puertas, H. Martinez-PPurrtas and A. Arcos. Calibration methods for estimat- ing quantiles. Metrika, 66(3):355-37,, December 2006.
R. J. Serfinn (ed.). Approximation Theorems of Mathematical Statistics. John Wiley & Sons, Inc., November 1980.
H. Song, T. Diethe, M. Kull, and P. Flach. Distribution calibration for regression. In Proceedings of the 36th International Conference on Machine Learning. volume 97 Proceedings of Machine Learning Research, pp- 5897-5906. PMLR, 6 2019.
L. Song, J. Huang, A. J. Smola, and K. Fukumizu. Hilbert space embeddings of conditional distribu- tions with applications to dynamical systems. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, pp. 961-968. Association for Computing Machinery, 2009.
L. Song, K. Fukumizu, and A. Gretton. Kernel embeddings of conditional distributions: A unified kernel framework for nonparametrie inference in graphical models. IEEE Signal Processing Magazine, 30(4):98-111, July 2013.
B. K. Sriperumbudur, K. Fukumizu, A. Gretton, B. Schölkopf, and G. R. G. Lanckriet. On integral probability metrics, -i-divergences and binary classification, 2009.
B. K. Sriperumbudur, K. Fukumizu, and G. R.G. Lanckriet. Universality, characteristic kernels and RKHS embedding of measures. Journal of Machine Learning Research, 12(70):2389-244 2410, 2011.
B. K. Sriperumbudur, K. Fukumizu, A. Gretton, B. Schölkopf, and G. R. G. Lanckriet. On the empirical estimation of integral probability metrics. Electronic Journal of Statistics, 6(0):1550- 1599, 2012.
Z. Szabó and B. K. Sriperumbudur. Characteristic and universal tensor product kernels. Journal of Machine Learning Research, 18(233):1-29, 2018. 2018.
M. Taillardat, O. Mestre, M. Zamo, and P. Naveau. Calibrated ensemble forecasts using quantile regression forests and ensemble model output statistics. Monthly Weather Review, 144(6):2375- 2393, June 2016.
Vaicenavicius, D. Widmann, C. Andersson, Lindsten, Roll, and T. B. Schön. Evaluating model calibration in classification. In Proceedings of Machine Learning Research, volume 89 of Proceedings of Machine Learning Research, pp. 3459-3467. PMLR, 4 2019.
A. W. van der Vaart. Asymptotic Statistics. Cambridge University Press, October 1998.
C. Villani. Optimal Transport. Springer Berlin Heidelberg, 2009.
D. Widmann, F. Lindsten, and D. Zachariah. Calibration tests in multi-class classification: A unifying framework. Proceedings of the 32th International Conference on Neural Information Processing Systems, pp. 12236-12246. 2019.
S. Yakowitz and J. D. Spragins. On the identifiability of finite mixtures. The Annals of Mathematical Statistics, 39(1 2:20--114, February 1968.
Bianca Zadrozny. Reducing multiclass to binary by coupling probability estimates. In Advances in Neural Information Processing Systems 14, pp. 1041-1048. MIT Press, 2002.
W. Zaremba, A. Gretton, and M. Blaschko. B-test: A non-parametric, low variance kernel two-sample test. In Advances in Neural Information Processing Systems 26, pp. 755-763. 2013.
12

---

A EXPERIMENTS
The source code of the experiments and instructions for reproducing the results are available htttps://ittuub.co. wtwpo//ddwettnno///eeeeeoee hww.oo/ooonaiibaataoo.oCCCCCCCC
Additional material
such as automatically generated HTML output and Jupyter notebooks is available at https
7/devmotion, wwt..b..tt.......... wtt.://aaaaa...oo....... ICLR2021/
A.1 ORDINARY LEAST SQUARES
We consider a regression problem with scalar feature and scalar target Y with input-dependent Gaussian noise that is inspired by a problem by Gustafsson et al. (2020). Feature X is distributed uniformly at random in [-1, 1], and target is distributed according to
(x) 111++ X
where (0, 0.15)). We train linear regression model P with homoscedastic variance using ordinary least squares and data set of 100 i..d. pairs of feature X and target Y (see Fig,
P(Y)))
P(У]X)
1.00
1-
0.75
density
0.50
-1-
0.25
22
0.00
-1.0
-0.5
0.5
1.0
-0.5
0.5
Figure 4: Data generating distribution P(Y[X) and predicted distribution P(YIX) of the linear regression model. Training data is indicated by orange dots.
A validation data set of 50 i.i.d. pairs of X and Y is used evaluate the empirical cumulative probability
[0. (P(Y & Y| x,))
of model P for quantile levels T E [0, 1]. Model P would be quantile calibrated (Song et al. 2019
T=Px'y (P(Y & Y X') <)
for all E [0, 1]. where (X and (X', Y") are independent identically distributed pairs of random variables (see Fig. 5).
Additionally, we compute a p-value estimate of the null hypothesis Ho that model P is calibrated using an estimation of the quantile the asymptotic distribution of nSKCE,,, with 100000 bootstrap samples on the validation data set (see Remark B.2). Kernel & is chosen as the tensor product kernel
le (p. v). (p'.)) exp
Wz(p.p') exp ( (y -
(m,, - mp (ap a )2 exp (- (v //)
exp
where W2 is the 2- Wasserstein distance and mp. my and op. oy denote the mean and the standard deviation of the normal distributions p and p (see Appendix D.I). We obtain < 0.05 our experiment, and hence the calibration test rejects Ho at the signíficance level 0.05.
13

---

1.00
Ideal
cumulative probability
data
0.75
0.50
0.25
0.00
0.00
0.25
0.50
0.75
1.00
quantile level
Figure Cumulative probability versus quantile level for the linear regression model on the validation data (orange curve). The green curve indicates the theoretical ideal for quantile-calibrated model.
A.2 EMPIRICAL PROPERTIES AND COMPUTATIONAL EFFICIENCY
We study two setups with d-dimensional targets Y and normal distributions Px of the form (cll,, 0.1) as predictions, where c U(0, 1). Since calibration analysis is only based on the targets and predicted distributions, neglect features X in these experiments and specify only the distributions Y and Px-
In the first setup simulate calibrated model. We achieve this by sampling targets from the predicted distributions, i.c., by defining the conditional distribution of Y given Px as
Y| = Ni,, Σ) (i,, E).
In the second setup simulate uncalibrated model of the form
P|Px = WW4, E) ~ 1e ual", Σ).
We perform evaluation of the convergence and computation time of the biased estimator SKCF& and the unbiased estimator SKCE2, a with blocks size E (2, vn, n). We use the tensor product kernel
((v,v)) (F/,/)) = exp W2(p. exp (v--1////2)
( m (op - op.)² exp (- //y-3////2)
= exp
where W2 the Wasserstein distance and mp: my and an or denote the mean and the standard deviation of the normal distributions p and p'.
Figures 6 to visualize the mean absolute error and the variance of the resulting estimates for the calibrated and the uncalibrated model with dimensions d = 1 and d = 10 for 500 independently drawn data sets of n E (4, 16, 64, 256, 1024] samples (Px, Y). Computation time indicates the minimum time in the 500 evaluations
computer with a 3.6 GHz processor. The ground truth values the uncalibrated models were estimated by averaging the estimates SKCE&, for 1000 independently drawn data sets of 1000 samples of (Px Y) (independent from the data sets used for the evaluation of the estimates). Figures and illustrate that the computational efficiency of SKCF in comparison with the other estimators comes at the cost of increased error and variance for the calibrated models for fixed numbers of samples.
We compare calibration tests based on the (tractable) asymptotic distribution of In/ SKCE,, with fixed block size B E (2, vn) (see Remark B.1 1. the (intractable) asymptotic distribution of nSKCE which is approximated with 1000 bootstrap samples (see Remark B.2). and Hotelling's
14

---

10-2.4
10-3.1
mean error
10
10 4.5
-0
variance
10
10-10
2
10
10-4.
10-3.0
10-1s
samples
time [s]
SKCE + SKCE (B 2) + SKCE (B = vn) + SKCE (B = n)
Figure Mean absolute error and variance of 500 calibration error estimates for data sets of {4, 16, 64, 256, 1024) samples from the calibrated model of dimension d = 1.
10
mean error
10-1
10
10
variance
10-6.0
10-7.5
10-9.0
10-5.0
10-4,5
10-2,0
10-1.
samples
time [s]
SKCE * SKCE (B 2) + SKCE (B = vn) + SKCE (B = n)
Figure Mean absolute error variance of 500 calibration error estimates for data sets of {4, 16, 64, 256, 1024) samples from the calibrated model of dimension d = 10.
15

---

10-1.2
mean error
10
10-2.-
10 2.4
10-2.-
variance
10-3,-
10.4.
2
10-45
10-2.0
10-1.
samples
time [s]
SKCE + SKCE (B 2) + SKCE (B = vn) + SKCE (B = n)
Figure Mean absolute error and variance of 500 calibration error estimates for data sets of {4, 16, 64, 256, 1024) samples from the uncalibrated model of dimension d = 1.
10-1.2
10 1.6
mean error
10-2.0
10-24
10
variance
10.3.2-
10.
10-4.
10-5.0
10-4,s
10-2,a
10-1.5
samples
time [s]
SKCE * SKCE (B 2) + SKCE (B = vn) + SKCE (B = n)
Figure Mean absolute error and variance of 500 calibration error estimates for data sets of {4, 16, 64, 256, 1024) samples from the uncalibrated model of dimension 10.
16

---

-tsstatistic for UCME,, with 10 test locations (see Appendix C We compute the empirical test errors (percentage of false rejections of the null hypothesis Ho that model P is calibrated if P is calibrated, and percentage of false non-rejections of Hn if P is not calibrated) at fixed significance level a 0.05 and the minimal computation time for the calibrated and the uncalibrated model with dimensions d = 1 and d 10 for 500 independently drawn data sets of n. E (4, 16, 64, 256, 1024) samples of (Px, Y). The 10 test predictions of the CME test are of the form N (m, 0.1²1,) where m is distributed uniformly at random in the d-dimensional unit hypercube [o, 1] the corresponding 10 test targets are i.i.d. according to (0, 0.1 I.)
Figures 10 and IT show that all tests adhere the set significance level asymptotically as the number of samples increases. The convergence of the CME test with 10 test locations is found to be much slower than the convergence of all other tests. The tests based on the tractable asymptotic distribution of / In/ mnBBja BJSCCEs for fixed block size B are orders of magnitudes faster than the test based on the intractable asymptotic distribution of nSKCE approximated with 1000 bootstrap samples. We that the efficiency gain comes at the cost of decreased test power for smaller number of samples, explained by the increasing variance of SKCF,, for decreasing block sizes B. However, in our examples the test based on SKCE,, k vi still achieves good test power for reasonably large number of samples (> 30).
calibrated model
type error
empirical test error
0.50
SKCE (B 2)
SKCE (B vn)
uncalibrated model
SKCE (B n)
type error
CME
06
significance level
04-
02-
20
10
10
10-1
10
# samples
time [s]
Figure 10: Empirical test emrors for 500 data sets of n E {4. 16, 64, 256, 1024] samples from models with targets dimension 1. The dashed black line indicates the set signficance level = 0.05.
callbrated model
type error type errrr
0.25
empirical test error
0.50
0.75
SKCE (B 2)
0.00
SKCE (B vn)
uncalibrated model
SKCE (B n)
0.75
CME
significance level
0.0
2-
10
10
10
20
samples
time [s]
Figure 11: Empirical test emrors for 500 data sets of n E (4. 16, 64, 256 1024} samples from models with targets of dimension d 10. The dashed black line indicates the set signficance level 0.05.
17

---

A.3 FRIEDMAN REGRESSION PROBLEM
We study the so-called Friedman I regression problem, which was initially described for 200 inputs in the six-dimensional unit hypercube (Friedman 1979 Friedman et al. 1983) and later modified to 100 inputs in the 10-dimensional unit hypercube (Friedman 1991). In this regression problem real-valued target Y depends on input X via
10 (xt X2) + 20(X3 0.5)² 2 10X, + 5X, +
where noise is typically chosen to be independently standard normally distributed. We generate training data set of 100 inputs distributed uniformly at random in the 10-dimensional unit hypercube and corresponding targets with identically and independently distributed noise following a standard normal distribution.
We consider models p(0,o²) of normal distributions with fixed variance o² x
p(0.a") = (ff
where (z),, the model of the mean of the distribution P(Y = z), is given by fully connected neural network with two hidden layers with 200 and 50 hidden units and ReLU activation functions. The parameters of the neural network denoted by 0.
We use a maximum likelihood approach and train the parameters A of the model for 5000 iterations by minimizing the mean squared error the training data set using ADAM (Kingma & Ba 2015) (default settings in the machine learning framework Flux..l (Innes] 2018 Innes et al.) 2018)) In each iteration, the variance o² is set to the maximizer of the likelihood of the training data set.
We train 10 models with different initializations of parameters 0. The initial values of the weight matrices of the neural networks are sampled from the uniform Glorot initialization (Glorot & Bengio 2010) and the offset vectors are initialized with zeros. In Fig. [2] we visualize estimates of accuracy and calibration measures on the training and test data set with 100 and 50 samples, respectively, for 5000 training iterations. The pinball loss is a common measure and training objective for calibration of quantiles (Song et al. 2019). It defined as
Ex.y I., (Y, (inantile(Px,11))
where L, (v. 5) = (1 - (J y) Tγy 7 and quantile( Pz, r) = (P S و 2 Tm
for quantile level 7 E [0, 1]. In Fig. 12 we plot the average pinball loss (pinball) for quantile levels T E {0.05, O.t,...,,............... )..95). We evaluate SKCE*., (SKCE (unbiased)) and SKCE2 (SKCE (biased))
for the tensor product kernel
((p, y). (p'. v/) exp W2(p.p')) exp ( (y 19--- -/2
exp (m mmp )2 (op o exp (- (y //)2/2),
where W2 is the 2- Wasserstein distance and my and o., σ. denote the mean and the standard deviation of the normal distributions p and p' (see Appendix D.I). The p-value estimate (p-value) is computed by estimating the quantile of the asymptotic distribution of nSKCE4.. with 1000 bootstrap samples (see Remark B.2 The estimates of the mean squared error and the average negative log- likelihood are denoted by MSE and NLL. All estimators indicate consistently that the trained models suffer from overfitting after around 1000 training iterations.
Additionally, we form ensembles of the ten individual models at every training iteration. The evaluations for the ensembles are visualized in Fig. 12 as well. Apart from the unbiased estimates of SKCE,, the estimates of the ensembles are consistently better than the average estimates of the ensemble members. For the mean squared error and the negative log-likelihood this behaviour guaranteed theoretically by the generalized mean inequality.
B THEORY
B.1 GENERAL SETTING
Let (2, be probability space. Define the random variables X: (2, + (X, Ex) and Y: (2 + (V,E, Ey) such that Σx contains all singletons, and denote a version of the regular conditional distribution of Y given X x by P(Y = x) for all x E X.
18

---

MSE
SKCE (unbiased)
10
10--
sinito. 0033)
training
NLL
SKCE (blased)
estimate
smM1166)
(nn124)
10
10"
models
ensemble
10
test
models
pinball
p-value
ensemble
12P
10"
0.3
1500
1500
4500
iteration
Figure 12: Estimates of different accuracy and calibration measures of ten Gaussian predictive models for the Friedman I regression problem versus the number of training iterations. Evaluations on the training data set (100 samples) are displayed in green and orange, and on the test data set (50 samples) in blue and purple. The green and blue line and their surrounding bands represent the mean and the range of the evaluations the ten models. The orange and purple lines visualize the evaluations of their ensemble.
Let P: (x,Ex) (P, B(P)) be a measurable function that maps features X' to probability measures in P on the target space y. We call P a probabilistic model, and denote by P3 == P(x) its output for feature
X. This gives rise the random variable Px (2,, - (PP) as Px P(X). We denote version of the regular conditional distribution Y given Px Pg by PX = P2) for all Pa E P.
B.2 EXPECTED AND MAXIMUM CALIBRATION ERROR
The common definition the expected and maximum calibration error (Guo al. 2017: Kull et al.. 2019 Nacini et al.) 2015) Vaicenavicius et al. 2019) for classification models can be generalized to arbitrary predictive models.
Definition B.1. Let d(-,.) be distance measure of probability distributions of target Y, and let be the law of Px. Then we call
and MCE, μ ess sup d the expected calibration error (ECE) and the maximum calibration error (MCE) of model P with respect to measure d, respectively.
ECE, E ((
B.3 KERNEL CALIBRATION ERROR
Recall the general notation: Let (Px)) + RR be a kernel, amd denote its corresponding RKHS by H.
If not stated otherwise, assume that
(KI) k(..)) is Borel-measurable.
(K2) & is integrable with respect to the distributions of (Px, Y) and (Px, Zx), ic..
Ex k1/2 ((Px, (P, <
and
Exx.2x 1/2 (1Px, (Px, (Px, <0.
19

---

Lemma B.1. There exist kernel mean embeddings UPxY:MPx2x H. such that for all f E H
(f.ppyy' = Epx,y f(Px,Y) and (f.MPx2xhH = Epx.2x f(xx, Zx).
This implies that
PPxY - Exx, k(,, (Px, and PPxZx = Exx, k(s (Px, Zx)).
Proof. The linear operators Trx yf Epx,Y Y f(Px, Y) and Txx = Epx, 2x [(xx, Zx) for
all E H are bounded since
[TxxY = EPx,Y f(Px, EEpr If(Px, = Epx,Y (K((Px, Y,
I x [|{((xx,)) M)-)1lll/] = 1151 Exx,y (Pxx))))
and similarly
[Txx 2x fl $ Il Epx.2x
(PPx Zx), (Px. Zx)).
Thus Riesz representation theorem implies that there exist JuPXY, MPx Zx E H such that T (f.PPx)) H and Tx (f. MPxxxx The reproducing property of H implies
MPxY (p. v) (k((p. v). MPx)) Exx Y k(P. y), (Px,
for all (p. v) E P x V, and similarly pAx Zx (p. = 1E 2x ((p: 4), Zx)).
Lemma B.2. The squared kernel calibration error (SKCE) with respect to kernel k, defined as SKCE, == KCE is given by
SKCE; = EXx.YYPxx k
- 22 + Exxx,Zx Px ((( 7x), (Px:, Zx:)),
Ex:))
where (Pxi, Y,, Zx.) is independently distributed according to the law of (Px.Y,Zy)
Proof. From Lemma B.I we know that there exist kernel mean embeddings μPXY MPx E that satisfy
(f.PPYY 1Px2x)* = (f.ppxyln (f.PPxZxhM =E f(Px. - Exx, s(Px, (x)
for all / E H. Hence by the definition of the dual norm
CE9, sup
EPx.Y f(Px, Epx.2x S(Px, 7x)
fEFx
sup feJn
(JIMPx,Y MPX = ||1xxY 1xxx.7xl1 11H,
which implies
SKCE2 = - - 1Px7x1
From Lemma B.I we obtain
SKCE* EExxYYPyy. Y), Y")) - PP ((x.)pp....xxxxx),)))) ((Pxx)), (Px,,
+ Px 16 (((,,
which yields the desired result.
Recall that (Px, Yi).... (Px., Y.) is validation data set that is sampled i.i.d. according to the law of (Px, Y) and that for (p. 9), (p', ,') P x y
h((p, v), (v'. w/)) = k((p.)) v), (w' ,'') Ez-p k((p, 2), (v',"'))) B2 k((p, y), (o'2")) + Ez-p, k((p. 2), (p',z'))
Lemma B.3. For all i,j 1, I,
(n((x,,))) (Px, Xi))| <0
almost surely.
20

---

Proof. Let E (I....,m).. By assumption (K2) we know that
(P((, Yi), (Px,, Y3)) = k²/2 ((Px. (Px., (..P)...)))) (P)) (Px,, Xj)) < 00 almost surely. Moreover.
Ezx, ((Px,) 2x,); (Px,, Ki)) S Ezx | Zx,), Vi))
(12 Zx,), (Px: Zx. x ((Px, Yj), (Px,, (Px,, xi) o
I Bzx,
almost surely, and similarly
Exx, (((Px,, xx.), (Px, [x,,) < 00 almost surely, Thus 1n( Yi), (Px, K2)) & (k((P,, Yi),
((), (Px, 1,)1 + |EEx, ,((Px, Zx.): (Px, Yj)] + - ((Px,; Yi), (Px,: xx,))| [x Ex, k((Px,: 7x,); (Px,, 7x,))) < 00
almost surely.
Lemma 1. The plug-in in estimator of SKCE& is non-negatively biased. It is given by
1
SKCE = n² 1J=1
M(Px,,))) (Px,
Proof. From Lemma B.2 we know that KCE₃ < oc, and Lemma B.3 implies that SKCE& < co almost surely.
For I,...,n, n, the linear operators Tif = Ezx, f(Px.; 7x.) for f E H are bounded almost surely since
Tifl = Bzx, Zx.) S Ezx, [f(Px, Zx.)| = =22x ((x((P,,
< Ezx,
[|x(((x,,
|f lf = 11flll Ezx, k2 (PPx, 2x,), (Px,; -x.))
Hence Riesz representation theorem implies that there exist P H. such that Tif (fip) almost surely. From the reproducing property of H we deduce that p= v) (1: (1r,)).) ,PPiH Ezx, (((p v). (Px,, Zx,)) for all (p. y) P y almost surely.
Thus by the definition of the dual norm the plug-in in estimator KCE, satisfies
KCE₃ = sup fEF
(f(Px, Yi) - Ezx, f(Px., Ex,))
()..)).))))) ps. 17M
= sup
feF,
Σ (k(((x,,) Y.). - -i),
= sup JeF
H1
((G V.).) x)).) -p:)
1/2
(PPx,))))) -pi Σ
k( V)).)) -Pi)
h:
1=1
1/2
1/2
(((x,, Yi), Vi), (Pxj. Y31)
=SKCE,
< 00
1/2 almost surely, and hence indeed SKCE is the plug-in estimator of KCF,.
21

---

Since (Px, Y), (Px,, Y"), (P); (.),,,,,Px Yn) are identically distributed and pairwise inde- pendent, we obtain
n E ESKCEE
Σ
Bpx X.P,,
X,))
W= #f
+
E h((P,,, (Px,,
-=1
n 1) E ((Px) ((Px) Y), (Px, Y")) + n E (PP,,)))
nn 1)SKCE& + E.xy (((Px,)), (Px,
(B.1)
With the same reasoning as above, there exist p. p' H such that for all f E H Ezx f(Px, Ex) (f. pin and Ez f(xx, Zx) (f,p') almost surely. Thus obtain
=
h(Px, Y). (Px, (x((Px. p, (()((()))))))))))))))))))))))))) p0)
almost surely, and therefore by Lemma B.2 and the Cauchy-Schwarz inequality
SKCE₃
P (P.
EXX Y
M,,, (((P,,);;))) - V"),, - p')x
S EPX.Y. Y
v'), p') m1/2
S EXx.Y.xx. ({(((())))))) -plxlll) ((xx,X()))) -
,'),,)
11/2 S Px,x (Pxx, -1ll P (aklll((( (Pxx, r'),)) - 2
7
Since (Px, Y) and (Pxi, Y') are identically distributed, we obtain
SKCF& S E Ex,,y [Ix( Y),) - pll Exx, (Px,
Thus together with Eq. B.I we get
n E SKCE& 2 n(n 1)SKCE&& + nSKCE, = n SKCEE
and hence SKCE& has a non-negative bias.
Lemma 2. The block estimator of SKCE& with block size B E (2....,n), given by
BJ
SKCE2 =
n
B
Σ
1(Px,, (I), ()),
2
b=1
is an unbiased estimator of SKCE
Proof.
From Lemma B.2 we know that SKCE, < oo, and Lemma B.3 implies that SKCF& < 00
almost surely.
For E {I,..., ......,L/DD.. [J.B]]. let
:o
B
-1
Σ
M(Px, YY. Yj))
(B.2)
(b-I)Bei<jsbB
be the estimator of the 6th block. From Lemma B.3 it follows that n < 0o almost surely for all Moreover, for all b. ?ip is so-called U-statistic of SKCF& and hence satisfies E Bi SKCF,, (see, e.g., van der Vaart 1998). Since (Px,, Yi)...., (Px., Yn) are pairwise independent, this implies that SKCF, is unbiased estimator of SKCF..
22

---

B.4 CALIBRATION TESTS
Lemma B.4. Let B E n.. If M < oo, then for all EE (I..... [ [n/B[}
B
VT. o =
((B-2)54 + W
2
where To is defined according to Eq.
B.2 and
<1 == Bxx Ep
((XP), -SKCE,
(B.3)
If model P is calibrated, it simplifies to
B
oB
EX 12 ((Px, (Px,
Proof. Let {1..... B) Since y ((Pxx))))))) (x, < oc, the Cauchy- Schwarz inequality implies V 77 < oo as well.
As mentioned in the proof of Lemma Samnmaaabovee. 7o U-statistic of SKCEg, From the general formula of the variance of U-statistic (see. c.g., HHoefddin 1948 p. 298-299) we obtain
V 1/6
B
(1)
B - 2 3--1----
(2)
2
B 2 2
VI (I,
2
2 2-2)
(2 2)(1 + VPX.X.x. n((Px)((P)))))) ((PP))))
2
where
C1 = E3 y () (), Y1) - SKCE
If model is calibrated, then (Px, Y) d (Px, 2), and hence for all (p. v) P x
Exx.y h((p,((()))))) v). (Px, Y)) Exx.Y k( ((p, v). (Px, - Ez Epx,y y k(p, 2'), (Px, (Px,Y)) Ep z k6 ((p, y), (Px, 2) + 1p E px 2 k 71), (Px, Y))
O.
This implies Ç1 Exx,Y E² yx Y), (Px,, Y") 0 and SKCE due to Lemma B.2 Thus
03
E EPx.Y.Py 12 (Pxx (Px,))),
as stated above.
Corollary B.1. Let B (2.....n.) y ()( Y") < oc, then
VKCEEL In/BI-1
where o3 is defined according to Lemmal B.4
Proof. Since the estimators .......
M//B in each block are pairwise independent, this is an imme-
diate consequence of Lemma B.4
Corollary B.2. Let B E (2....,n) nn.. If W.V...PPP.....
(Px, < oc, then
Tn/BJ (SKC SKCE.) 4 (0,
as xx,
where block size B is fixed and o is defined according to Lemma B.4
Proof. The result follows from Lemma 2 Lemma B.4 and the central limit theorem (see, e.g.,
Serfling 1980 Theorem A in Section 1.9).
23

---

asymptotically tight Remark B.1. Corollary [.2 shows that SKCE&, is consistent estimator of SKCE& in the large sample limit as + oo with fixed number of samples per block. In particular, for the linear estimator with B we obtain
[m/2] (SKCE - SKCE,) 4 (0.
-
as 71 oc.
Moreover, Lemma B.4 and Corollary calibrated can be estimated by
B.2 show that the p-value of the null hypothesis that model P
Mn/BjSKCEE ōB
where @ is the cumulative distribution function of the standard normal distribution and og is the empirical standard deviation of the block estimates m. ......... Mn/BJ, and
+ MT/BBBCL I)SKCE
V2G
where 82 is an estimate of P.Ps Y h22 ((Px, Y), (Pxi, Y"). Similar p-value approximations for the two-sample test with blocks of fixed size were used by Chwialkowski et al. (2015).
Corollary B.3. Assume V PX ha..m//( (((( Y), (Px,,))) Y') <c. Let E {1, [n/2]]. Then for all b {1.....s} $}
VB. SKCEs) 3 NC0,4C)) as B + oc,
(B.4)
where ii defined according to Eq.
B.2
with Bs, the number of equally-sized blocks fixed,
and Si is defined according to Eq.
B.3
If model P is calibrated, then v B 76
SKCE2) = VBn, is asymptotically right since Ç1 = 0, and
Bîg 4 Σ Ar(2,
as B - oc,
(B.5)
where Zi are independent X² distributed random variables and X; E IR are eigenvalues of the Hilbert-Schmidt integral operator
Kf y) = Exx, (H((p,)), (Px, ())x
for Borel-measurable funciioo f: x R with Y f2xx, <..
Proof. Let s E (1....,,nee [n/...,,n/[]] and b [1...., s).
As mentioned above in the proof of Lemma 2 the estimator Th. defined according to Eq. (B.2). a so-called U-statistic of SKCE, (see, c.g., van
Vaart 1998). Thus Eq. (B.4) follows from the asymptotic behaviour of U-statistics (see, e.g., van der Theorem 12.3).
Vaart
1998
If P calibrated, then we know from the proof of Lemma B.4 that Ç = 0, and hence n is so-called degenerate- U-statistic (see, e.g.. van der Vaart) 1998 Section 12.3). From the theory of degenerate U-statistics it follows that the sequence Bij,, converges in distribution to the limit distribution in Eq. B.5), which is known as Gaussian chaos.
Corollary B.4. Assume V Px.XXPx (((xx Y), (Pxx, Y") co. Let (1...., [n/2]}. Then
VB(SKCF - SKCF.) 4, N(0044 (1() as - 00,
where the number sized blocks is fixed, Bs, and Ç1 is defined according Eq. (B.3) If model calibrated, then /B BS(KCEE -SKCEE, /BSKCE since Ç1 0, and
BSKCE& SS-1 Σ 1Z s) B 00,
where Z, are independent x2 distributed random variables and A E IR are eigenvalues of the Hilbert-Schmidi integral operator
KS y) = Exxx (H((,,)),, ())))))))))))))
for Borel-measurable functions functttton
p:P x with Ex (Px, Y) <..
24

---

Proof. Since the estimators m ... Te in each block are pairwise independent, this is immediate consequence of Corollary B.3
Remark B.2. Corollary B.4 shows that SKCE&, is consistent estimator of SKCE, in the large sample limit as B - oo with fixed number n B] of blocks. Moreover, for the minimum variance unbiased estimator with B n, Corollary B.4 shows that under the null hypothesis that model P is calibrated
RSKCER,
(Z - 1)
+ 00;
i=1
where Z, are independent xi distributed random variables. Unfortunately quantiles of the limit distribution of Σi=1 \₁ 1) (and hence the p-value of the null hypothesis that model P is calibrated) can not be computed analytically but have be estimated by, e.g., bootstrapping (Arcones) & Giné 1992), using a Gram matrix spectrum (Gretton et al., 2009), fitting Pearson curves (Gretton al. 2007). or using Gamma approximation (Johnson et al. 1994 p. 343, p. 359).
Corollary B.5. Assume VPx P. (((xx,)) (Px,, Y") < co. Then
Ln/B (SCCEE, - SKCEE) 4 ((004G)
as B - oo and [n/B] + co,
(B.6)
where B is the block size and is the number of equally-sized blocks, Bs, and Si is defined according to Eq. (B.3).
model P is calibrated, then / BBBBB((SKCE..C SKCEa) = / Ln/B] BSKCE&, is asymp- totically tight since Ç1 = 0, and
V n/BB BSKCE& S N 1 E as + 0o and |n/ B] - oc,
where x E R are eigenvalues of the Hilbert-Schmidt integral operator
Ks(p. y) = Exx,Y (h((p. v), (Px, (P), Y)) with Ex y (Px,
for Borel-measure
functions f: functiooo
80.
Proof. The result follows from Corollary B.3 and the central limit theorem (see, e.g., Serfling 1980
Theorem A in Section 1.9).
Remark B.3. Corollary B.5 shows that SKCE,,, is a consistent estimator of SKCE, in the large sample limit as B ao and [n/B] oo, i.e., both the number of samples per block and the number of blocks go to infinity. Moreover, Corollaries B.3 and B.5 B.5 show that the p-value of the null hypothesis that P is calibrated can be estimated by
(n/B B//BSSA BBJSKCEKB
OB
where nn is the empirical standard deviation of the block estimates ./, M///B]- Similar p-value approximations for the two-sample problem with blocks of increasing size were proposed and applied by Zaremba et al. (2013).
C CALIBRATION MEAN EMBEDDING
C. DEFINITION
Similar to the unnormalized mean embedding (UME) proposed Chwialkowski et al. (2015) in the standard MMD setting, instead of the calibration error CE,, 1PxY 4Px Zx H we can consider the unnormalized calibration mean embedding (UCME).
25

---

Definition C.1. J E N. The unnormalized calibration embedding (UCME) for kernel with / test locations defined as the random variable
UCME 2, J (
- (xxxxxxxx)) (T)))²
jml
= =J
Σ ==1
(Exx.y k(T,, (Px, Y)) Exx.xx k(T,, (Px, Zx)))
where Ti...., Ty are i.i.d. random variables (so-called test locations) whose distribution is absolutely continuous with respect to the Lebesgue measure on P x
As mentioned above, in many machine learning applications we actually have P x ) C R (up to some isomorphism). In such case, if is analytic, integrable, characteristic kernel, then for each E N UCME& is a random metric between the distributions of (Px, Y) and (Px, Zx ), as shown by Chwialkowski al.] [2015] Theorem 2). In particular, this implies that UCME,, 0 almost surely If and only if the two distributions are equal.
C.2 ESTIMATION
Again we assume (Px1: ......,(PP,, Ya) is validation data set of predictions and targets, which are i.i.d according to the law (Px, Y). The consistent, but biased, plug-in estimator UCME given by
n
(Px, Ezx, (Px, 2x.1))
UCME = J 1-1
Σ
n
C.3 CALIBRATION MEAN EMBEDDING TEST
As Chwialkowski et (2015) note, model is calibrated, every fixed sequence of unique
test locations /NUCME converges in distribution to sum of correlated x² random variables, 00. The estimation of this asymptotic distribution, and its quantiles required for hypothesis testing, requires a bootstrap or permutation procedure, which is computationally expensive. Hence Chwialkowski et al. (2015) proposed the following test based on Hotelling's -tsstiiistic (Hotelling 1931).
For i 1, ........ n, let
(Tx. - Ezx, Zx.))
24 =
ER',
K(T, (Px. Yi) - Ezx, k(T,, (Px,, 2x.))
and denote the empirical mean and covariance matrix of Zı....; Zn by Z and S, respectively. If UCME,,, is random metric between the distributions of (Px. Y) and (Px, Zx), then the test statistic
Qn = s-
is almost surely asymptotically x2 distributed with J degrees of freedom if model P calibrated, as n + Do with J fixed; morcover, model P uncalibrated, then for any fixed r E almost surely P(Q,. r) calibration test calibration mean embedding (CME) test.
(Chwialkowski et al. 2015 Proposition 2). We call the resulting
D KERNEL CHOICE
natural choice for the kernel le (P x y) x (P x )) + IR on the product space of predicted distributions P and targets y is tensor product kernel of the form kp ky, i.e., a kemel of the form
x((p.v) v). (w'v'') v')) ('y..''
26

---

where kp: P x P - R and ky: y x y - IR are kernels the spaces of predicted distributions and targets, respectively.
As discussed in Section 3.1 if kernel k is characteristic, then the kernel calibration error KCEg of model P is zero if and only if P is calibrated. Unfortunately, as shown by [Zzabó & Sriperumbudur (2018) Example 1), even if kp and ky characteristic, the tensor product kernel k kpp & Sky might not be characteristic. However, when analyzing calibration, it is sufficient to be able to distinguish distributions for which the conditional distributions P(Y)Px) and P(Zx|Pxx = Px are equal almost surely. Thus it is sufficient if ky is characteristic and kp is non-zero almost surely.
Many common kernels such the Gaussian and Laplacian kernel Rd characteristic and can therefore be chosen as kernel ky for real-valued target spaces. The choice of kp might be less obvious since P is a space of probability distributions. Intuitively one might want use kernels of the form
kp (p. = exp (- (p.(p.)))
(D.1)
where dp: P x P - a metric on P and v, x > O are kernel hyperparameters. Kernels this form would be generalization of the Gaussian and Laplacian kernel, and would clearly be non-zero almost surely.
Unfortunately, this construction does not necessarily yield valid kernels. Most prominently, the Wasserstein distance does not lead to valid kernels kp in general (Peyré & Cuturi 2019 Chapter 8.3). However, ((((.) is Hilbertian metric, i.e., a metric of the form
dp(p, p') = |p(pp) - <(p')l
for some Hilbert space H and mapping Φ: P + H, then kp in Eq. D.I) a valid kernel for all A and (0, 2] (Berg et al. 1984 Corollary 3.3.3, Proposition 3.2.7).
D.1 NORMAL DISTRIBUTIONS
Assume that y' = Rd and P (W(,, E): E E Rdxd psd]. i.e., the model outputs normal distributions Px = Mux, Σx). The distribution of these outputs is defined by the distribution of their mean fax and covariance matrix Σx.
Let Pa N(2x Ex,) E E E Rd, and y > 0. We obtain
Eza~Ps exp
->|1zz (xpp-11lzz -v13)
112 La + 2y | exp 1app--766 1-7((p-)) T) + 2yE..)) -1
( (pz y)
from Mathai & Provost
1992
Theorem 3.2.a.3.. In particular, if = diag(2 Err,,. Exed), then
Eza~P exp (-77lZz - -m1²)
- 1/2 exp (-7(1+275n0) ( ((pni-yi) u (1 (1+2+
II
Let P.. =N(uu', Egr) be another normal distribution. Then we have
Ez,mPa. exp ( 21 - Zz. 1x.
= 14 1/2
E2,imP, cxp
( - Zeo ((a 24 (Nz Zz))
= (En + +zx))
1/2 exp
- 7- pwe) (Ia (T + Ex:)) -1 (uo - Mar))
Thus if E., diag(2 Et, bb,lb..., Exzed) and Eg diag( Eee, Exe a). then
exp ( 1->||Zz Zz. 12)
(1 + (Eris + Ex),)) 112 exp 7 22 + Ex.,)) (Ma,s -Ma m²
27

---

Hence see that Gaussian kernel
(y) = exp 1-yly flyly-1 3/ 2'112)
with inverse length scale 7 > on the space of targets y = R allows us to compute Zz)) analytically. Moreover, the Gaussian kernel is
Ez,~P, ky(Zz, y) and P.. ky characteristic Rd (Fukumizu et al., 2008). Hence, discussed above, by choosing a kernel kp that is non-zero almost surely we can guarantee that KCF₂ = if and only if model P is calibrated.
On the space of normal distributions, the 2- Wasserstein distance with respect the Euclidean distance between P₂ = (M2,, Ex) and Pa- NNu'' Exr) is given by
1/2 E 1/2) 1/2)
(Pa, Pa) 11 pr
Tr Σx + Ex
Ex
which can be simplified to
(2( ||me
Morllz
Me'
E//
Frob
= This shows that the 2- Wasserstein distance is a Hilbertian metric on the space of normal distributions. Hence discussed above, the choice kp Pa. P..) = exp XW (P,, Pz.))
yields valid kernel for all >> and 20.
Thus for all X, > 0 and E (0, 2]
(((p, v). (p' v = exp ( - WW (p,p') exp (- - 711y - v/1))
a valid kernel on the product space P x y of normal distributions on [R and g that allows to evaluate ((P, v), x/), analytically and guarantees that KCE, 0 if and only if model P is calibrated.
D.2 LAPLACE DISTRIBUTIONS
Assume that y R and P {L(H, 3): E R. > 0}, i.c., the model outputs Laplace distributions Px = f(xx,3x)) with probability density function
(x = 28x
exp ( 1x3xx pxl)
for y y' R. The distribution of these outputs is defined by the distribution of their mean μ and scale parameter Bx.
Let P. L(M,, 32) P. y R,, and 0. If 1f Ez.~P, exp (-7/4x- -v)
x have
= -18 Bzr exp 8- |tx vl) exp (-71kz ( 21))
Additionally, if Br =7-1, the dominated convergence theorem implies
Ez,-P, exp Z_ vl)
= lim
(B2,2 - -1) -1 (xyrepp ( (-B-t1me v1) exp ( 7144 81) 0+0
(1 + 7/u - -l) exp (opp--lles (-y|mp - -y)).
.nd Let P.. L(p 32r) be another Laplace distribution. 3r # -1, Gr # -1 and Gx # Bx,, we
obtain
7:33
Ez, Pea.PeeZuinP. exp (- 2/0 22rl)) (82-22
exp 3-'11a p,x,
1)(3)
3 2)
733,
exp 3.r -1 piz
(82,72
1)
(2
exp ( 7/pur - psxl)
(g
() 2,12
28

---

As above, all other possible cases can be deduced by applying the dominated convergence theorem. More concretely,
if fr = Br = 7 then
E P exp ( ( --| - Zz/1)
1x 1x-13) exp ( 714 1orl)) (3 3 + 37//4 Marl 1 + +
8
if B3 Ba' and Br # y-1 then
1
E E.. exp ( (- -->|Zx - Zw/l)
exp --/lu Mew
(3272 1)2
1 (3g + |12 - Pxxl)
B
exp (- (- 14-11z Harl)
+
2(B2n2 -1
(3272 - 1)2
if B.g + Bxr and B3 = =- then
3r.
Ez, P.. exp ( -110 Zv.1)
exp
(- -Barr -1 1p 1
Mad
(3x 272 1)
7114 Uor
Br.2
+
exp
(->1 Merl),
2(B 272 - 1)
23
and B2 # B2 and Bor = y-- then
E E2g~Pe, exp
7/7 -7xrl)
exp (- 335 Frx pex
(3222 1)
1 + 7/1= - Mx/
82n2
exp ( - 7|Mu Marl))
2( 2(3222 - 1) (32 1)²
The calculations above show that by choosing a Laplacian kernel
(v exp ( >ly - -
with inverse length scale 7 > O on the space of targets y = IR, we can compute Ez,~r, ky y) and P2,,.P..2,.00 ky Zz, Z analytically. Additionally, the Laplacian kernel is characteristic
R (Fukumizu et al. 2008).
Since the Laplace distribution is elliptically contoured distribution, we know from Gelbrich (1990 Corollary 2) that the 2-Wasserstein distance with respect to the Fuclidean distance between P C(pe, 9x) and Ps C(p B2r) can be computed in closed form and is given by
W P2,, Pa.) (#z )z)) + 2(B -3..)²
Thus we see that the 2- Wasserstein distance is also Hilbertian metric on the space of Laplace distributions, and hence
(Pa, exp ( (Pa, is a valid kemnel for 0 < < 2 and all À > O.
Therefore, as discussed above, for all X, 7 > and v E (0,
k((p.)) (p/ v')) exp ( XW (p. ) exp -ly v'l)
a valid kernel on the product space P x y Laplace distributions and that allows to evaluate h ((p. v), (p', v')) analytically and guarantees that KCF₃ = 0 if and only if model P is calibrated.
29

---

D.3 PREDICTING MIXTURES OF DISTRIBUTIONS
Assume that the model predicts mixture distributions, possibly with different numbers of components. special case of this setting ensembles of models, in which each ensemble member predicts component of the mixture model.
Let p.p' P with p E. TiPi and p' = -; Typ/ where n. n' are histograms and Pt. P/J are the mixture components. For kernel by and y y obtain
Zz Roy (2,1) Σ Te Ew (Z
and
Bz-p.Z'mp ky 2, 7
Σ no Ez 2'mp, ky(2,77))
Of course, for these derivations to be meaningful, we require that they do not depend on the choice of histograms N, x' and mixture components pt. P/J-
Definition D.1 (see Yakowitz & Spragins (1968). ). A family P of finite mixture models is called K'
identifiable if two mixtures p Ei= TiP; P and p' = Ej=ı n/pp E EP, written such that all p and all p/ are pairwise distinct, are equal and only K = K' and the indices can be reordered K such that for all l E {K there exists some k' [1...., K} with nT = Tt. and pk = pk.
Clearly, if P is identifiable, then the derivations above do not depend the choice of histograms and mixture components. Prominent examples of identifiable mixture models are Gaussian mixture models and mixture models of families of products of exponential distributions (Yakowitz & Spragins 1968).
Moreover, similar to optimal transport for Gaussian mixture models by Chen al. 2019
2020
Delon & Desolneux 2020), we can consider metrics of the form
1/8
Σ we (pisk/))
inf WE
where
[[ x(T,,x)== = { w: Wiis= = T Wis mis= Vi.j: was 2 20}
wi.
are the couplings of π and n', and .(.,.) ) a cost function between the components of the mixture model.
Theorem D.1. Let P be a family of finite mixture models that is identifiable in the sense of Defini- tion D.I and let E (1).
If d(...) a (Hilbertian) metric on the space of mixture components, then the Mixture Wasserstein distance of order s defined by
1/s
MW. (p.p')
inf well(saa")
Σ
Wi (pr. (p.. i,j
(D.2)
is a (Hilbertian) metric on P.
Proof. First of all, note that for all p.p' E P an optimal coupling á exists (Villani 2009 Theo- rem 4.1). Moreover, Ei, wi di.,d'(P..PP) (p:.P') 2 0, and hence MW, (p. p') exists. Moreover, since P identifiable, we see that MW. (p, p') does depend on the choice of histograms and mixture components. Thus MW, , is well-defined.
Clearly, for all p.p' P we have MW MW.(p.p') 2 0 and MW.(p.7) MW. (W'pp..p.. Moreover,
MW.(p.P) min
Σ wEll(s,,)))
why d' S
Σ
No (p.,))
Σ n, (pr.p:)
= Σ x,02=
30

---

and hence MW, (p. p) = 0. On the other hand, let p.p' p' P with optimal coupling á with respect to т and n', and assume that MW (p.p') p') = O. We have
p Σ TiPi =
Σ fj
Wi
Σ
Wi,Pi.
id: Bey 20
Since MW, (p.p') = 0, we have d (ps: 1/y for all i, and hence d' (pt. if dei, > MW,(p.p) Since is metric, this implies p: p/ Uhg 0. Thus we get
p =
Σ igj: 210 20
WisPi
Σ
Σ Tispps = Tntpt=1 = 2
Function MW, also satisfies the triangle inequality, following a similar argument as Chen al. (2119). Let p(1) , E and denote the optimal coupling with respect 7 (1) and T²2) by 20(12) and the optimal coupling with respect to m!2) and (3) by (((3) Define w(13) by
(12) (23)
(13) w
j.k
1,k
3: a
40
(18) Clearly wi,k
2 for all i, ki, and we see that
(12) (23)
W W
(12) (23)
Σ
(13)
Σ
J.A
j.A
(2)
(2)
j: m
(2)
40
j: m/ 40
(2)
(23)
j
J.k
(223) n
(23)
ui
2
n (2) 40
J: m 40
12) =0 5:
for all k. Since for all j, k. nT. for all k
(23)
we know that n.
(2)
implies 20 (23) 0 for all k. Thus
Σ
w (13) i.k
(3)
Similarly we obtain for all i
Σ
(13) w i.6
= (1)
31

---

Thus ((1)) II(()), x ) and therefore by exploiting the triangle inequality for metric d and the Minkowski inequality we get
1/8
(23) (12)
1/a
Σ 1.k
(13) i
d'
(1)
.Pk
(3))
Σ
Σ
(2)
j.k
d
(ps
(1) (3)
Pkk
1,k j: m. 2 p
(12) (23)
.2 ))))))))) 1/8
Σ
ij j,b
(d(v, (1) (2) d (2)
(2)
1,k d: m 1
(12) (23)
1/3
Pi 1 (2))
(1)
2)
1.k j: /
12) (23)
1/8
+ Σ Σ
L
1.k
d'
Pj Pk
(2)
x1²))
i, j: x
(2) #0
1/8
- Σ
12) d'
(1) (2)
Pi :23
j: n 40
+ Σ Σ
( d
(1fP) (8)
k J: n #0
(12) (1) (2)) )) 1/8
1/s
1
Σ
w ij
pi .p
+
Σ w
(23) i,k
(2) (p/.) (pj (p)
Pk
(3)
= MWs(p" 22) + MW s (p .))
Thus MW, is a metric, and it is just left to show that it is Hilbertian if is Hilbertian. Since is a Hilbertian metric, there exists a Hilbert space H and mapping @ such that
d(z,y) ||x(x) dx))
Let Tn E Th with E.r, rg 0 and respect to (0) and T() by ((.)) Then we have
(m)) E P. Denote the optimal coupling with
Σ rrrs Σ
w "1 ( |1
Σ rillob11 (o) )13 Σ r
(2.5) k.l
4 k,
i.k
j
Σ rallo(pk
(8) )12 Σ
(6)
TT
k/
(D.3)
Σ i
(3)
110(p )1 Σ ry =,
rerk
and similarly
E rerr 1
Σ
(1
|o(o0)))))) O.
(D.4)
k.1
Moreover, for all k., l we get
Σ (ary (4.5) [J
( (0(p))) (2) (p(())) )²m
) .0
Σ rry Whll 0 (p()) Σ rgv kd (p()))) H
(14)
=
Σ rry x o (p()") || 20 0.
(1)
and hence
Σ Errs Σ
(1-5) (4) ((0p""),0((())
20,
(D.5)
7
kil
32

---

and similarly
Ena rT Σ T ".0l""))(p"))) 20.
(D.6)
Hence from Eqs.
(D.3) to D.6) we get
Σ
rir Mwr p()))) = =-
rits
Σ
d
Pk p(
(1)
1
2.j
Σ r
(ij)
Pk xk)) - -o(p)") 1x
(tJ)
12
H
4,
Σ
(3,5)
2 (p)))) ( (p(())
(8)
R.
k.l
Σ
Σ
1 (p())) . (6(2))
r
k.l
1
0
W
kil
1
50,
which shows that MW is a negative definite kernel (Berg et al.
1984
Definition 3.1.1). Since
a 0 1/8 co, MW, a is negative definite kernel well (Berg et al. 1984 implies that metric MW, is Hilbertian (Berg et al. 1984 Proposition 3.3.2).
Corollary 3.2.10), which
Hence we can lift Hilbertian metric for the mixture components to a Hilbertian metric for the mixture models. For instance, if the mixture components are normal distributions, then the 2- Wasserstein distance with respect to the Euclidean distance is Hilbertian metric for the mixture components. When we lift it to the space P of Gaussian mixture models we obtain the MW2 metric proposed by Chen et al. (2019 2020); |) Delon & Desolneux (2020). As shown by Delon & Desolneux (2020), the discrete formulation MW2 obtained by our construction is equivalent to the definition
/
inf MW2 7EUl(p. JGGM (oc) xB"
(D.7)
for two Gaussian mixtures p,p' on R", where H(p, p²) the couplings of p and p' (not of the histograms!) and GMM2n (x)) = Uezo GM2. is the set of all finite Gaussian mixture distribu- tions on 2n The construction of the discrete formulation as a solution to constrained optimization problem similar to Eq. (D.7) can be generalized to mixtures of f-distributions. However, it is not possible for arbitrary mixture models such as mixtures of generalized Gaussian distributions, even though they are elliptically contoured distributions (Deledalle et al. 2018 Delon & Desolneux] 2020).
The optimal coupling of the discrete histograms can be computed efficiently using techniques from linear programming and optimal transport theory such as the network simplex algorithm and the Sinkhorn algorithm. As discussed above, if metric dp is of the form in Eq. (D.2), functions of the form
kp p²) exp ( (p.
are valid kernels on P for all À > 0 and E (0,
Thus taken together, if by is a characteristic kernel on the target space y and d(..)) is Hilbertian metric on the space of mixture components, then for all E (1, X > O, and v E (0,2)
1: ((p. v). (f/.//) exp ( AMW" (p. (p.))) (w.
is a valid kernel on the product space x of mixture distributions and targets that allows to evaluate ((P..) v), (p',y") analytically and guarantees that KCFg = 0 if and only if model P is calibrated.
33

---

E CLASSIFICATION AS A SPECIAL CASE
We show that the calibration error introduced in Definition is a generalization of the calibration error for classification proposed by Widmann et al. (019). Their formulation of the calibration error is based on a weighted sum of class-wise discrepancies between the left hand side and right hand side of Definition[] [ where the weights are output by a vector-valued function of the predictions. Hence their framework can only be applied to finite target spaces, i.e., if [3]] < oC.
Without loss of generality, we assume that = {l....,d, d} for some d E N \ (1). our notation, the previously defined calibration enror, denoted by CCE (classification calibration error), with respect to function space C [f: P > RRd} } given by
(P(Y = ylpx) (x(()))))))) )
CCEg
sup gEG
Epx
Σ VEV
For the function class
F: =
we get
CCEg sup Exx,Y f(Px, Y) - EBxx f(xx 2x) = CEF. fEF
Similarly, for every function class F {f: P x y - R}, we can define the space
g = g:: P + Rd p + 1), sl.
for which
CE = sup gEg
EPx
= CCEg.
VEY
Thus both definitions are equivalent for classification models but the structure of the employed function classes differs. The definition of CCE is based vector-valued functions the probability simplex whereas the formulation presented in this paper uses real-valued function on the product space of the probability simplex and the targets.
An interesting theoretical aspect of this difference is that in the case of KCE we consider real-valued kernels on P x y instead of matrix-valued kernels P, as shown by the following comparison. By E Rd we denote the ith unit vector, and for prediction p E P its representation up E R in the probability simplex defined as
(1p) p(()))
for all targets y E V.
Let k: (P x )) x (P x) - R. We define the matrix-valuee function K: P x + Rdxd by [K(p.p')] v. k(pp. y). v))
for all y. E y and p.p' E P. From the positive definiteness of kernel is it follows that K is matrix-valued kernel (Micchelli Pontil 2005 Definition 2). We obtain
SKCE+ E [K(Px, Px )r)))r - 2 P.. [x:xx,
Px Px. Ex [K(xx, (x.xx,Px)) Px.)] 2x.2x
-EPr.YPW. of K(xx, Px )x.)er 2 2F M.P. (xx)pxxxxx
+EP Y up K(Px, Px.)
= y (ey - vpx) K(Px, (xy) -
-vpr)) :
which is exactly the result by Widmann et 20199 for matrix-valued kernels.
As a concrete example, Widmann et al. (011 used a matrix-valued kernel of the form (p. + exp (->llp- P'Il) in their experiments. In our formulation this corresponds the real-valued tensor product kernel ((p, y), v')) exp (-yllp p' (1)"
34

---

F TEMPERATURE SCALING
Since many modern neural network models for classification have been demonstrated to uncali- brated (Guo al. 2017). it is of high practical interest being able to improve calibration of predictive models. Generally, y, one distinguishes between calibration techniques that are applied during training and post-hoc calibration methods that try calibrate existing model after training.
Temperature scaling (Guo et al. 2017) is a simple calibration method for classification models with only one scalar parameter. Due to its simplicity it can trade off calibration of different classes (Kull et al. 2019), but conveniently it does not change the most-confident prediction and hence does not affect the accuracy of classification models with respect the 0-1 loss.
In regression, common post-hoc calibration methods are based on quantile binning and hence insufficient for our framework. [Song al. (2019) proposed calibration method for regression models with real-valued targets, based on special case of Definition This calibration method was shown to perform well empirically but is computationally expensive and requires users to choose hyperparameters for Gaussian process model and its variational inference. As a simpler alternative, we generalize temperature scaling to arbitrary predictive models in the following way.
Definition F.1. Let Pr be the output of a probabilistic predictive model P for feature æ. If Pa has probability density function Ps with respect reference measure μs, then temperature scaling with respect to with temperature T > 0 yields new output Q₂ whose probability density function Ox with respect to j satisfies
tr x Ps 1/T.
The notion for classification models given by Guo et al. (0017) can be recovered by choosing the counting measure the classes as reference measure.
For some exponential families on Fd we obtain particularly simple transformations with respect to the Lebesgue measure xd that keep the type of predicted distribution and its mean invariant. Hence in contrast to other calibration methods, for these models temperature scaling yields analytically tractable distributions and does not negatively impact the accuracy of the models with respect the mean squared error and the mean absolute error.
For instance, temperature scaling of multivariate power exponential distributions (Gómez al. 1998) in Rdd of which multivariate normal distributions are special case, with respect to X corresponds to multiplication of their scale parameter with T1/3 where 3 the so-called kurtosis parameter Gómez Sanchez-Manzano al. 2008). For normal distributions, this corresponds multiplication of the covariance matrix with T.
Similarly, temperature scaling of Beta and Dirichlet distributions with respect reference measure u(dx) = r 11-1(1111 - - (o,1)( (dx)
and
u(dr)= = " 1
respectively, corresponds to division of the canonical parameters of these distributions by T without affecting the predicted mean value.
All in all, we see that temperature scaling for general predictive models preserves some of the nice properties for classification models. For some exponential families such as normal distributions reference measure y can be chosen such that temperature scaling is simple transformation of the parameters of the predicted distributions (and hence leaves the considered model class invariant) that does not affect accuracy of these models with respect to the mean squared error and the mean absolute error.
G EXPECTED CALIBRATION ERROR FOR COUNTABLY INFINITE DISCRETE TARGET SPACES
In literature, ECE, and MCE, are defined for binary and multi-class classification problems (Guo) al. 2017 Naeini al. 2015 Vaicenavicius al. 2019). For common distance measures the
35

---

probability simplex such as the total variation distance and the squared Euclidean distance, ECEa and MCEd can be formulated as a calibration error in the framework of Widmann et al. (2019). which is special case of the framework proposed in this paper for binary and multi-class classification problems.
In contrast previous approaches, framework handles countably infinite discrete target spaces as well. For every problem with countably infinitely many targets, such as, e.g., Poisson regression, there exists an equivalent regression problem on the of natural numbers. Hence without loss of generality we assume y = N. Denote the space of probability distributions on N, the infinite dimensional probability simplex, with A x Clearly, A 0e can be viewed subspace of the sequence space that consists of all sequences I = (In) with 2n 2 0 for all E N and ||0|| = 1.
Theorem G.1. Let 1 < p < with Hölder conjugate q. If R Epx (f((Px
F= [= x
MEN 1 S 11,
then
CE = Epx P(Y Px) Px ll
Let μ be the law of Px. Ff (f: 09
R Epx (( n)) S 1}, then
CEF = μi- ess sup sup & yEN
PY = y Px = {) -((1)))
Moreover, === (f: A 00 x N R μ ess sup supyet [f v) S 1}, then
CE = Bpx |P Px 1-
Proof. Let 1 00, and let μ be the law of Px and v be the counting measure on N. Since both and are o-ninie measures, the product measure μ v uniquely determined and a-finite well. Using these definitions, we can reformulate F as
F= (f x N: IIfl 1}.
Define the function 5: A N + R (μ 3 -almmo surely by
8(E = P(Y y Px {) (((v)).
Note that 8 well-defined since we assume that all singletons on x0 are p-measurable. Moreover, 5 E - x N; p @ v), which follows from (5. + P(Y = y Px {) and (E, y) + E(ty)) being functions ( N: @ v).
Since @ // is a finite measure, the extremal equality of Hölder's inequality implies that
CE,
sup Ex, S(Px,Y) - EExx 7x s(Px. Zx)
feF
sup /EF
Epx.Y f(Px, Y) Exx f(Px, Zx)
sup JEF
JA f(E.8)8(E,g) (a vd(()))))
||6' 1oa.....
Note that the second equality follows from the symmetry of the function spaces F: for every F. also -f E F. Hence for 1 < p S.p. 00, we obtain
CF / A x 18(E. 1) (I 6 (d(E))))
Erx [1 v)) Erx |P(YPxx) Pxl1g
For = 1, we get
CE = μl- ess sup sup 185. v)1 = p sup
CSS SE VEN
P(Y = u/ = 5) E((()))
tE vEN
which concludes the proof.
36

---

We that our framework deals with countably infinite discrete target spaces seamlessly whereas the previously proposed framework by Widdann et al. (2019) is not applicable to such spaces. is mathematically pleasing to that for countably infinite discrete targets the calibration errors obtained in Theorem G.1 within our framework coincide with the natural generalization of ECF, and MCF. given in Appendix B8.2
37


Reviews:
[{"review_id": "yredTQOClAY", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors define (Definition 2) a generalized form of calibration error for any model with probabilistic output and any output space (binary classification, multiclass classification, real-valued regression, ordinal regression, structured prediction, etc.).\n\nThe main novelty of Definition 2 seems to be the introduction of a dummy variable $Z_X$ which is sampled from the predicted distribution $P_X$ over the target space, in order to compute an integral probability metric that compares the joint distribution of $(P_X, Y)$ to the joint distribution of $(P_X, Z_X)$. The authors note that in some cases, $Z_X$ can be integrated out of the definition either analytically or using numerical integration. It is not clear, from my limited background knowledge in the related work, why it should be helpful to introduce a dummy variable and then integrate it out.\n\nAt first reading, Definition 2 seems too general to be useful in practice, as it requires the choice of a space of functions $\\mathcal{F}$. The authors argue (with details in the appendices, which are not provided to me at this time) that it generalizes several previous definitions of calibration error including the maximum mean discrepancy, the total variation distance, the Kantorovich distance, and the Dudley metric.\n\nSection 3 went beyond my area of expertise and beyond my comprehension; I feel unqualified to provide an informed review of this section. I think some reference to kernels or RKHSs should be made in the title or the abstract.\n\nThe experiment in Section 5.2 demonstrates the utility of the proposed SKCE calibration metric, alongside more common metrics like negative log-likelihood (NLL) and mean squared error (MSE). It appears that the SKCE curves (for both training and test data) have a very similar shape to the NLL curves, so it's not clear what benefit SKCE provides above and beyond the more common and easily-computed NLL. I would have liked to see more convincing experimental evidence of the marginal benefit of this approach beyond common calibration metrics.\n\nRegarding the significance of the work, I can add that in practice, I find that relatively few ML users are concerned with the calibration of their models, and these are entirely restricted to problems of classification (almost always binary classification) or quantile regression. The novelty of this work seems to lie mostly in its applicability beyond these types problems: the authors write, \"A key contribution of this paper is a new framework that is applicable to multivariate regression, as well as... discrete ordinal or more complex (e.g., graph-structured) [output],\" so I venture a guess that the intended audience for this work is relatively small.\n\n**Minor comments**\n\nIn equation (2), I believe the RHS should be $\\max P_X$ instead of $\\arg \\max P_X$.\n\nSection 2, paragraph 2, you wrote \"instead of the discrepancy between the conditional distributions $\\mathbb{P}(Y | X)$ and $P_X$.\" Did you mean to write $\\mathbb{P}(Y | P_X)$ instead of $\\mathbb{P}(Y | X)$? That would make more sense to me, since the sentence would refer to comparing the LHS and RHS of Equation (1).\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "GSZuabszvv", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses probabilistic data driven model calibration, i.e. aligning predicted target probabilities with actual ones. \n\nThis problem has been extensively studied for classification tasks but solutions for regression tasks have limitations as illustrated by Fig.1. The authors intend to fill this gap and introduce a general kernel-based calibration framework that subsumes other ones previously defined for classification. The contribution of the authors is thus clearly stated and positioned w.r.t. prior arts. \n\nThe authors start by proposing an alternative definition of calibration (Def.2) in order to cast the problem into integral probability metrics. It is this re-definition of the problem that allow them to encompass prior arts as special cases. \nFor a number of practical and theoretical reasons, the authors focus on a special case of this framework which involves the computation of the MMD as a metric. \n\nBased on the MMD literature but also relying on the structure of their problem (where the auxiliary variable can be marginalized out), the authors provide several consistent estimator with known rates in dataset size.\n\nThe validity of the proposed estimates is assessed through convincing numerical experiments that involve a calibration test.\n\nI honestly do not have much critic to address to this work which seems to have reached a level of maturity perfectly adapted for publication in ICLR. The only damper is that the proposed methodology allows to detect miscalibration not yet to cure it. However, the authors seem to have some ideas on that too as mentioned in their conclusion.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "FHjZ-Fwvi4n", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors present an approach for testing calibration in conditional probability estimation models. They build on a line of work in the kernel estimation literature assessing whether the conditional distributions are well calibrated (i.e. P(Y | f(X)) = f(X), where f is some predictive model). They develop an MMD kernel estimator and expand on practical choices of kernels that are computationally tractable. They then derive an asymptotic null distribution for calibrated models, enabling control over the error rate when labeling a model uncalibrated. A few simulation studies are done with neural networks to show the applicability of the method.\n\nReview:\nThis is an excellently written paper. The intro and first few chapters are a joy to read and really explain the problem well. There is a lot of nuance to calibration, so I really appreciated the precision and clarity in the exposition.\n\nThe idea itself also seems quite elegant. Generalizing a previously published kernel approach from only discrete distributions to handle a more general class of problems may seem like a small conceptual step. However, I think the authors did a good job explaining the challenges of this extension. The resulting estimators are now applicable to many more problems than the existing work.\n\nNote I am not an expert in kernel learning, so I have not evaluated the proofs for correctness.\n\nMy main issue comes with the lack of empirical studies. The toy problem is not terribly interesting and does not reveal any particular insight. It leads me to believe that maybe this is not that useful of a method, since the authors did not have anywhere that they could apply it to and derive meaningful insights or uses. The comment in the conclusion about the differentiability of their kernels is interesting and I think incorporating this into the training procedure could potentially show some very clear pragmatic use of this method.\n\nOverall, I like the paper. It is clearly written and presents what I think is an interesting and novel idea. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a well written paper addressing a challenging problem with an original approach.  While one reviewer claims there is not a strong call for calibration of regression tasks, this may well be because methods don't exist.  Certainly, calibration is a critical tool for classification.

The major failing of the paper, however, is the empirical evaluation.  Given that no prior work exists, it is arguably OK to not do this, but one could easily reject the paper on this issue alone, as AnonReviewer4 was inclined to do.  One reviewer, however, thought highly of the paper, which bumped up its average score, more than I think it should have got (due to the poor experimental work).

The abstract could be improved by mentioning the use of kernels, the nature of this solution is a substantial part of the paper.
Decision: Accept (Poster)