Title: Calibration by Distribution Matching: Trainable Kernel Calibration Metrics
Venue: NeurIPS 2023
Accepted: yes
Keywords: Uncertainty Quantification

Calibration

Decision Making

Probabilistic Forecasting
Abstract: Calibration ensures that probabilistic forecasts meaningfully capture uncertainty by requiring that predicted probabilities align with empirical frequencies. However, many existing calibration methods are specialized for post-hoc recalibration, which can worsen the sharpness of forecasts. Drawing on the insight that calibration can be viewed as a distribution matching task, we introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression. These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization. Furthermore, we provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions. Our empirical evaluation demonstrates that employing these metrics as regularizers enhances calibration, sharpness, and decision-making across a range of regression and classification tasks, outperforming methods relying solely on post-hoc recalibration.

Calibration by Distribution Matching: Trainable Kernel Calibration Metrics

Abstract

Calibration ensures that probabilistic forecasts meaningfully capture uncertainty by requiring that predicted probabilities align with empirical frequencies. However, many existing calibration methods are specialized for post-hoc recalibration, which can worsen the sharpness of forecasts. Drawing on the insight that calibration can be viewed as a distribution matching task, we introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression. These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization. Furthermore, we provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions. Our empirical evaluation demonstrates that employing these metrics as regularizers enhances calibration, sharpness, and decision-making across a range of regression and classification tasks, outperforming methods relying solely on post-hoc recalibration.

1 Introduction

Probabilistic forecasts are valuable tools for capturing uncertainty about an outcome of interest. In practice, the exact outcome distribution is often impossible to recover [51], leading to overconfident forecasts [15]. Calibration provides statistical guarantees on the forecasts, ensuring that the predicted probabilities align with the true likelihood of events. For example, consider a weather forecast that assigns an 80% probability of rain tomorrow. A well-calibrated forecast would imply that, on average, it rains on 80% of the days with such predictions. By appropriately quantifying uncertainty, calibration empowers decision-makers to efficiently allocate resources and mitigate risks, even when the outcome distribution cannot be recovered exactly.

Many forms of calibration have been proposed in both classification [8, 21, 35] and regression [42, 23, 53] to ensure that uncertainty estimates match true frequencies. From a methodological perspective, the literature on enforcing calibration is fractured; many algorithms to enforce calibration are specialized for a particular form of calibration [39, 29, 52, 10, 17] or are designed exclusively for post-hoc calibration [27, 23]. Although post-hoc methods are effective at enforcing calibration, they often lead to a degradation in the predictive power, i.e. sharpness, of the forecaster [26]. This motivates the need for versatile metrics that can enforce various forms of calibration during training.

In this work, we introduce a unified framework wherein existing notions of calibration are presented as distribution matching constraints. To enforce distribution matching, we use the Maximum Mean Discrepancy ( MMD ), giving us unbiased estimates of miscalibration that are amenable to gradientbased optimization. We frame these metrics as regularizers and optimize them alongside proper scoring rules. This allows us to enforce calibration while preserving sharpness.

Framing calibration as distribution matching [38] allows us to use the Maximum Mean Discrepancy ( MMD ) [14] to give unbiased estimates of miscalibration that are amenable to gradient-based optimization. An advantage of this approach is that we express existing calibration measures by varying the kernel in the MMD metric. Moreover, we can express new notions of calibration specific to a downstream decision task. This allows us to prioritize errors that impact decision-making [52]. Our contributions can be summarized as follows:

We propose a framework based on distribution matching to unify many notions of calibration.

We provide differentiable training objectives to incentivize a wide variety of popular forms of calibration during empirical risk minimization in supervised learning. We frame these metrics as regularizers that can be optimized alongside proper scoring rules. This allows us to enforce calibration without degrading sharpness.

We show how to define and enforce new calibration metrics tailored to specific downstream decision problems. When these metrics are successfully optimized, decision makers can accurately estimate the decision loss on unlabeled data.

We perform an empirical study across 10 tabular prediction tasks (5 regression and 5 classification) and find that using our metrics as regularizers improves calibration, sharpness, and decision-making relative to existing recalibration methods.

2 Related Work

Calibration Our work builds on the literature on calibrated forecasting [31, 8, 9, 13]. Recently, many forms of calibration have been proposed in classification [5, 25, 24, 32, 35, 33, 10, 17, 28, 36, 27], regression [23, 42, 53, 39, 51], and beyond [47, 22]. Algorithms to enforce calibration are often designed specifically for one form of calibration. For example, significant attention has been devoted to effectively estimating the Expected Calibration Error (ECE) to calibrate confidence levels in binary classification [19, 44, 4, 15, 35]. Our work aims to supplement this work by providing general calibration metrics that can be used as training objectives. These methods can be used in concert with post-hoc calibration, to encourage calibration during model training and correct any remaining miscalibration after training is complete.

Decision Calibration An interesting line of recent work focuses on calibrating predictions for downstream decision problems [39, 52]. Zhao et al. [52] shows that decision calibration can be achieved when the set of possible actions is finite. We show cases in which this result can be extended to infinite action spaces by leveraging a low-dimensional sufficient statistic of the decision problem. Whereas Zhao et al. [52] compute calibration by approximating a worst-case decision problem with a linear classifier, we use the kernel mean embedding form of an MMD objective, making it simple to calibrate with respect to a restricted set of loss functions by adjusting the kernel.

Kernel-Based Calibration Metrics Motivated by issues arising from the histogram binning of the Expected Calibration Error, a recent stream of research uses kernels to define smooth calibration metrics. Zhang et al. [50] and Popordanoska et al. [37] use kernel density estimates to estimate calibration in classification settings. In contrast, our work applies to regression and takes a distribution matching perspective. Widmann et al. [47] and Kumar et al. [26] are the most similar existing works to our own, and also introduce MMD-based calibration metrics. Our work differs from Widmann et al. [46] due to our focus on trainability and optimization, connections to existing forms of calibration, and applications to decision-making. Kumar et al. [26] can be seen as a special case of our method applied to top-label calibration in multiclass classification (see Table 1). Our work extends the literature on MMD-based calibration metrics, showing how to express and optimize 11 existing forms of calibration by varying the kernel in the MMD objective (see Tables 1 and 2).

3 Calibration is Distribution Matching

We consider the regression setting, where, given a feature vector x ∈ X = R d , we predict a label y ∈ Y ⊂ R . We are given access to n i.i.d. examples ( x 1 , y 1 ) , . . . , ( x n , y n ) distributed according to some true but unknown cumulative distribution function (cdf) P over X ×Y . A lower case p denotes the probability density function (pdf) or probability mass function (pmf), when it exists. We use

subscripts for marginal and conditional distributions, so P Y | x ( · ) is the conditional cdf for Y given X = x and p Y ( · ) is the marginal pdf for Y . When the input X is random, we write P Y | X as the cdf-valued random variable that takes value P Y | x upon the event that X = x .

Our goal is to learn a forecaster Q that, given a feature vector x , forecasts a cdf Q Y | x ( · ) over Y . A forecaster implicitly defines a joint distribution over X × Y by combining the forecasted conditional pdf with the marginal pdf for the features, i.e. q ( x, y ) = p X ( x ) q Y | x ( y ) . The forecaster's joint cdf Q ( x, y ) is defined by integrating the pdf q ( x, y ) over the appropriate region of X × Y . In the same way as ( X,Y ) ∼ P is a sample from the true distribution, we write a sample from Q as ( X, ̂ Y ) ∼ Q , where X ∼ P X and ̂ Y is a label drawn from the forecast ( ̂ Y | X = x ) ∼ Q Y | x .

An ideal forecaster perfectly recovers the true probabilities so that Q = P . However, in practice we often only observe each feature vector x once, making perfect forecasts unattainable [51]. Instead, calibration enforces that forecasts are accurate on average , leaving us with a tractable task that is strictly weaker than perfect forecasting. Each form of calibration requires the forecasts to respect a particular law of probability. For example, if Y is binary, distribution calibration states that on examples for which the forecast is q Y | x ( Y = 1) = c , we should observe that Y = 1 with frequency c . In regression, quantile calibration [23] states that y should exceed each quantile c ∈ [0 , 1] of the forecast with frequency c . Calibration can also be enforced within subgroups of the data (i.e. group calibration [36]), in smooth neighborhoods of the input space (i.e. local calibration [27]), and even for a single feature vector (i.e. individual calibration [51]) by using randomization.

Calibration can be expressed as a conditional distribution matching problem, requiring that certain variables associated with the true distribution match their counterparts from the forecasts.

Lemma3.1 (informal) . Quantile calibration is equivalent to distribution matching between Q Y | X ( Y ) and P Y | X ( Y ) . Distribution calibration is equivalent to distribution matching between ̂ Y and Y given Q Y | X . Individual calibration is equivalent to distribution matching between ̂ Y and Y given X .

The choice of random variables that we match in distribution determines the form of calibration recovered. We show how to express many popular forms of calibration in terms of distribution matching in Table 1, and include additional details on these correspondences in Appendix A. In Section 4, we describe how to express and estimate these forms of calibration using kernel-based calibration metrics. In Section 5, we show how to tailor these metrics to downstream decision problems. Finally, in Section 6, we apply these metrics as regularizers and explore the empirical effects on calibration, sharpness, and decision-making.

4 Calibration Metrics as Trainable Regularizers

In this section, we introduce a framework to enforce calibration using distribution matching. We quantify calibration using integral probability metrics which admit unbiased estimates from data

and are amenable to gradient-based optimization. This allows us to optimize calibration alongside a standard training objective, achieving calibration without degrading sharpness. Jointly optimizing calibration and sharpness has been shown to outperform only enforcing calibration post-hoc [26].

Integral probability metrics (IPMs) are a flexible set of metrics to quantify the difference between two distributions [30] in terms of a family of witness functions. Since we need to perform distribution matching conditionally , we consider witness functions over both the label Y , as well as a conditioning variable Z that assumes the same distribution under P and Q .

Different choices for the witness functions F and the conditioning variable Z measure different notions of calibration. Similar to Widmann et al. [47], we focus on the case where F is the unit ball of a Reproducing Kernel Hilbert Space (RKHS) H , so that the IPM corresponds to the Maximum Mean Discrepancy [14], denoted MMD( F , P, Q ) . MMDcan either be expressed in terms of the canonical feature map ϕ ( y, z ) ∈ H or the reproducing kernel k (( y, z ) , ( ̂ y, z ′ )) = ⟨ ϕ ( y, z ) , ϕ ( ̂ y, z ′ ) ⟩ H for H :

Given a dataset D consisting of n i.i.d. pairs ( x i , y i ) ∼ P , we can generate forecasted labels ̂ y i ∼ Q Y | x i . The conditioning variables z i are computed from ( x i , y i ) according to the expressions given in Table 1. The plug-in MMD estimator is unbiased and takes the form [14]:

̸

where h ij is a one-sample U statistic:

The variance of this estimator can be reduced by marginalizing out the simulated randomness used to sample ̂ y i ∼ Q Y | x i . This can be done either analytically, or empirically by resampling the forecasted labels multiple times per example. We find that empirically marginalizing out the simulated randomness improves training stability in practice. The full training objective combines a proper scoring rule (we use negative log-likelihood) with an MMD estimate to incentivize calibration. To train, we divide the dataset D into batches D b ⊂ { 1 , . . . , n } large enough so that the MMD estimate on a batch has low enough variance to provide useful supervision (we find that D b of size 64 suffices in practice). Given a family of predictive models { Q θ : θ ∈ Θ } (e.g., a neural network), we optimize the following training objective on each batch:

In order to differentiate ̂ MMD 2 ( F , D b , Q θ ) with respect to θ , we require that the kernel k is differentiable and the samples can be expressed as a differentiable function of the model parameters (e.g., using a reparameterization trick). For parameterizations that do not admit exact differentiable sampling, we can apply a differentiable relaxation (e.g., if Q θ is a mixture distribution, we can use a Gumbel-Softmax approximation). In principle, it is possible to remove the need for differentiable sampling altogether by optimizing our MMD objective using Monte-Carlo RL techniques (e.g., REINFORCE [48]). However, this is beyond the scope of our work. Note that regularizing for calibration at training time does not give calibration guarantees, unless one can guarantee the objective will be optimized out-of-sample. In this sense, regularization and post-hoc calibration are complementary: regularization mitigates the trade-off between calibration and sharpness while post-hoc methods can provide finite-sample calibration guarantees (e.g., [45, 29]). For this reason, we suggest using regularization and post-hoc calibration together (see Table 3).

On the Choice of the Kernel When the kernel k is universal, the MMD metric is zero if and only if Y and ̂ Y are equal in distribution, given Z [14]. Universality is preserved when we decompose the kernel over Y ×Z into a pointwise product of two universal kernels, one over Y and one over Z [43].

Lemma 4.1. Let k be the function defined by the pointwise product k (( y, z ) , ( y ′ , z ′ )) = k y ( y, y ′ ) k z ( z, z ′ ) where k y and k z are universal kernels over spaces Y and Z respectively. Then, under mild conditions, k is a universal kernel over Y × Z .

See [43, Theorem 4] for a proof. In our experiments, we explore using a universal RBF kernel over Y × Z , since it satisfies the mild conditions needed for Lemma 4.1. To enforce decision calibration, we also consider a kernel k Y that is intentionally not universal, and instead only measures differences between P and Q that are relevant for decision-making (see Section 5). Furthermore, to measure Quantile Calibration and Threshold Calibration, we must match the distributions of the probability integral transforms P Y | X ( Y ) and Q Y | X ( Y ) instead of Y and ̂ Y . Thus, we replace k Y with a universal kernel over [0 , 1] , the codomain of the probability integral transform.

4.1 Calibration Metrics for Classification

Thus far, we have focused on estimating calibration metrics in the regression setting. We now describe how our framework applies to classification.

In the classification setting, given a feature vector x ∈ X = R d we predict a label y ∈ Y = { 1 , . . . , m } . As before, we are given access to n i.i.d. examples ( x 1 , y 1 ) , . . . , ( x n , y n ) distributed according to some unknown distribution P over X × Y . Our goal is to learn a forecaster Q that, given a feature vector x , forecasts a probability mass function (pmf) q Y | x ( · ) over labels Y .

We express popular notions of calibration used in classification (i.e., canonical, top-label, marginal) in terms of distribution matching in Table 2. See Appendix A.2 for additional details on calibration and distribution-matching in classification. We use the same unbiased plug-in estimator for MMD shown in Equation 3. In the classification setting, we can analytically marginalize out the randomness originating from the forecast when computing the MMD estimate. After performing this marginalization, the one-sample U-statistic h ij is is given by

where the conditioning variables z i are computed from ( x i , y i ) , as detailed in Table 2. Here, q i ( y ) = q Y | x i ( y ) is the predicted probability of the label Y = y given features x i . When the forecast and target variables depend on variables other than Y (e.g., Y ∗ depends on q Y | X in top-label calibration), we add those variables as inputs to the kernel. Note that the plug-in MMD estimator with this U-statistic is differentiable in terms of the predicted label probabilities, and that it can be computed in O ( n 2 m 2 ) time.

5 Calibration and Decision-Making

Some of the most practically useful results of calibration are the guarantees provided for downstream decision-making [52]. Universal kernels guarantee distribution matching in principle, but in some cases require many samples. However, if we have information about the decision problem in advance, we can design calibration metrics that are only sensitive to errors relevant to decision-making. In this section, we show how to design such metrics and show that they enable us to (1) accurately evaluate the loss associated with downstream actions and (2) choose actions that are in a sense optimal.

We begin by introducing a general decision problem. For each example, a decision-maker chooses an action a ∈ A and incurs a loss ℓ ( a, y ) that depends on the outcome. Specifically, given information z (such as the forecast), a decision policy δ : Z → A assigns an action δ ( z ) . The set of all such policies is denoted ∆( Z ) . The Bayes optimal policy under a label distribution P Y is denoted δ ∗ ℓ ( P Y ) := arg min a ∈A E P [ ℓ ( a, Y )] . Decision calibration [52] provides two important guarantees:

Loss estimation says that the expected loss of each action is equal under the true distribution and the forecast. This means we can estimate the loss on unlabeled data. No regret says that the Bayes optimal action according to the forecast δ ∗ ℓ ( Q Y | x ) achieves expected loss less than or equal to any decision policy δ ∈ ∆( Z ) that depends only on z . If we condition on the forecast by choosing z = Q Y | x , then a decision-maker who only has access to the forecast is incentivized to act as if the forecast is correct. Additionally, we consider cases where the loss function is not known in advance. For example, when a vendor releases a foundation model to serve a large population of users, it is likely that different users have distinct loss functions. Thus, we also show settings in which we can achieve accurate loss estimation and no regret for all loss functions ℓ included in a family of losses L .

(

∥ µ P -µ Q ∥ 2 = ∑ a ∈A ∑ ℓ ∈L E P [( ℓ ( a, Y )) -E Q [ ℓ ( a, ̂ Y )]] . When either A or L is infinite, the corresponding sum is replaced by an integral. This metric measures whether each action has the same expected loss marginally under the true distribution and the forecasts. Thus, accurate loss estimation is only guaranteed marginally and the no regret result is relative to constant decision functions that choose the same action for all examples.

A General Recipe for Decision Calibration We now describe an approach to measure decision calibration using our kernel-based metrics. For each action a ∈ A and loss ℓ ∈ L , we want to measure the discrepancy between the expectation of the loss ℓ ( a, Y ) under P and Q . To achieve this, we define the feature map that computes the losses of all actions ϕ ( y ) = ℓ ( a, y )) a ∈A ,ℓ ∈L . This assigns to P and Q the mean embeddings µ P = E P [( ℓ ( a, Y )) a ∈A ,ℓ ∈L ] and µ Q = E Q [( ℓ ( a, ̂ Y )) a ∈A ,ℓ ∈L ] , respectively. Letting F be the unit ball of the Hilbert space associated with this feature map, the mean embedding form of MMD gives us that MMD( F , P, Q ) 2 = 2 ( ) 2

Next, we refine the decision calibration results to hold conditionally on a variable Z . Recall that the kernel over y is given by k y ( y, y ′ ) = ⟨ ϕ ( y ) , ϕ ( y ′ ) ⟩ . We lift this kernel into Y × Z by defining k (( y, z ) , ( y ′ , z ′ )) = k y ( y, y ′ ) k z ( z, z ′ ) where k z is a universal kernel. The MMD metric corresponding to this kernel measures decision calibration conditioned on Z . If we choose Z to be the forecast Z = Q Y | X , then the no regret guarantee says that acting as if the forecasts were correct is an optimal decision policy among all policies depending only on the forecast.

Note that the feature maps defined above are infinite dimensional when A or L are infinite. However, applying the kernel trick sometimes allows us to estimate the MMD metric efficiently. Still, optimizing the MMD metric may be difficult in practice, especially when the conditioning variable Z is very expressive. We suggest using the metric as a regularizer to incentivize decision calibration during training, and then still apply post-hoc recalibration if needed. We now give two concrete examples, one where the action space is infinite and one where the set of loss functions is infinite.

Example 1 (Point Estimate Decision) . Consider a regression problem in which a decision-maker chooses a point estimate a ∈ R and incurs the squared loss ℓ ( a, y ) = ( a -y ) 2 . The expected loss of an action a is E P [ ( a -Y ) 2 ] = ( a -E P [ Y ]) 2 + E P [ Y 2 ] -E P [ Y ] 2 . Observe that the expected loss only depends on Y through the first two moments, E P [ Y ] and E P [ Y 2 ] . We use the representation ϕ ( y ) = ( y, y 2 ) , so that the kernel is k ( y, y ′ ) = 〈 ( y, y 2 ) , ( y ′ , y ′ 2 ) 〉 . From the mean embedding form of MMD, we see that MMD( F , P, Q ) = ∥ µ P -µ Q ∥ 2 = ( E [ Y ] -E Q [ ̂ Y ]) 2 +( E P [ Y 2 ] -E Q [ ̂ Y 2 ]) 2 . Therefore, the metric is zero exactly when the expected loss of each action a is the same under the true distribution and the forecasts. The MMD metric incentivizes the forecasts to accurately reflect the marginal loss of each action. However, we often want to know the optimal action given a particular forecast, not marginally. We can now choose the conditioning variable to refine the guarantee. Let Z be the forecast Z = Q Y | X and choose the kernel k (( y, z ) , ( y ′ , z ′ )) = k y ( y, y ′ ) k z ( z, z ′ ) where k y ( y, y ′ ) = 〈 ( y, y 2 ) , ( y ′ , y ′ 2 ) 〉 as before and k z ( z, z ′ ) is universal. For this kernel, the MMD will equal zero if and only if E P [ Y | Q Y | X ] = E Q [ ̂ Y | Q Y | X ] and E P [ Y 2 | Q Y | X ] = E Q [ ̂ Y 2 |

Q Y | X ] almost surely. The Bayes optimal action suggested by the forecast is optimal among decision policies that depend only on the forecast:

In other words, the decision maker is incentivized to behave as if the forecasts are correct.

̸

Example 2 (Threshold Decision) . The above example had an infinite action space and a single loss function. Here, we consider a problem where the action space is small but the set of loss functions is infinite. This setting applies for a model vendor who serves a model to multiple decision makers, each with a potentially different loss function. For example, suppose that Y ∈ [0 , T ] is a blood value for a patient for some T > 0 , and the decision task is to determine whether Y exceeds a threshold t ∈ [0 , T ] that indicates eligibility for a clinical trial. Formally, the decision maker chooses a threshold a ∈ {± 1 } to minimize the loss ℓ t ( a, y ) = 1 { a = sign( y -t ) } . Our goal is to simultaneously calibrate our forecasts for all thresholds t ∈ [0 , T ] , i.e. L = { ℓ t : t ∈ [0 , T ] } . In this case, we choose the feature map ϕ ( y ) = ( 1 { y ≥ t } ) t ∈ [0 ,T ] . The corresponding kernel is k ( y, y ′ ) = ⟨ ϕ ( y ) , ϕ ( y ′ ) ⟩ = min { y, y ′ } . The mean embeddings are E P [ ϕ ( Y )] = P Y and E Q [ ϕ ( ̂ Y )] = Q Y , the marginal cdfs for Y and Y Q . Thus, the MMD metric measures ∥ P Y -Q Y ∥ 2 2 = ∫ T t =0 ( P Y ( t ) -Q Y ( t )) 2 dt , which is zero exactly when P Y = Q Y . When MMD( F , P, Q ) = 0 , the expected loss of each action a is the same under the true distribution and the forecasts, for all ℓ ∈ L . Similar to the previous example, we can achieve conditional decision calibration by adding a universal kernel over the conditioning variable.

Discussion The connections between decision-making and calibration have been previously studied by Zhao et al. [52] and Sahoo et al. [39]. Here, we extend results from Zhao et al. [52] to include infinite action spaces and further refine the decision guarantees to depend on an arbitrary conditioning variable. In Example 2, we connect our methods to the special case of threshold calibration [39]. Notably, our framework provides a strategy to optimize decision calibration at training time.

6 Experiments

The main goals of our experiments are to: (1) compare the performance our method with other trainable calibration metrics, and with standard post-hoc recalibration methods; (2) study the impact of tailoring the MMD kernel to a specific decision task; and (3) study whether trainable calibration metrics can be used to improve calibration across local neighborhoods of data. 2

6.1 Datasets

We consider standard benchmark datasets and datasets relating to decision-making tasks. For each dataset, we randomly assign 70% of the dataset for training, 10% for validation, and 20% for testing.

Regression Datasets We use four tabular UCI datasets (SUPERCONDUCTIVITY [16], CRIME [34], BLOG [6], FB-COMMENT [41]), as well as the Medical Expenditure Panel Survey dataset (MEDICALEXPENDITURE [7]). They are common benchmarks in uncertainty quantification literature [23, 39]. The number of features ranges from d = 53 to 280 . The total number of examples n , and features d for each dataset can be found in Table 3.

Classification Datasets We use five tabular UCI datasets: BREAST-CANCER [49], HEART-DISEASE [18], ONLINE-SHOPPERS [40], DRY-BEAN [1], and ADULT [2]. The datasets range from m = 2 to 7 label classes, and d = 16 to 104 features. The total number of examples n , features d , and classes m for each dataset can be found in Table 4.

Crop Yield Dataset We introduce a CROP-YIELD dataset, where the task is to predict yearly wheat crop yield from weather data for different counties across the US. We use the NOAA database to track summary weather statistics across each year (temperature, precipitation, cooling/heating degree days), and the NASS database to track annual wheat crop yield. The dataset spans from 1990-2007, where data from 1995 was held-out for visualization, and the remaining data was used for training. We use this dataset to evaluate local calibration with respect to the latitude and longitude coordinates.

6.2 Experimental Setup and Baselines

Models To represent the model uncertainty, we use a feed-forward neural network with three fully-connected layers. In the classification setting, the model outputs the logits for all m classes. In the regression setting, the model outputs the predicted mean µ ( Y ) and variance σ 2 ( Y ) , which are used to parameterize a Gaussian probability distribution. This allows us to tractably compute the inverse cdf Q -1 Y | X and perform differential sampling, enabling us to test our calibration metrics in multiple settings. See Appendix B for more details on model hyperparameter tuning.

Training Objectives In the regression setting, we consider two training objectives: Negative Loglikelihood ( NLL ), and our trainable calibration metric as a regularizer ( NLL+MMD ). For MMD , we enforce a notion of individual calibration (Table 1) by choosing the conditioning variable Z = X to match the distributions of ( X, ̂ Y ) and ( X,Y ) . More specifically, we optimize an MMD objective where the kernel k is decomposed among the pair of random variables as k (( x, y ) , ( x ′ , y ′ )) = k X ( x, x ′ ) · k Y ( y, y ′ ) . The form of k X and k Y varies across experiments. In the classification setting, we consider cross-entropy (XE) loss, and three trainable calibration metrics as regularizers: MMCE [26], KDE [37], and MMD ( Ours ).

Baselines In the regression setting, we compare our approach against an uncalibrated forecaster trained with NLL , and with post-hoc recalibration using isotonic regression [23]. In the classification setting, we compare our approach against an uncalibrated forecaster trained with XE loss. Furthemore, we compare our approach to two trainable calibration metrics ( MMCE [26] and KDE [37]). We compare all approaches against temperature-scaling [15], a standard post-hoc recalibration method. For both classification and regression, we train the post-hoc recalibration methods on a validation set, and subsequently evaluate on the held-out test set.

Metrics To quantify predictive performance in regression, we evaluate the negative log-likelihood, quantile calibration error (QCE) [23], and decision calibration error (see Section 6.3). In classification, we compute accuracy, expected calibration error (calibration), and average Shannon entropy (sharpness) of the forecast. We also report kernel calibration error (KCE), defined as the MMD

objective for an RBF kernel over X × Y . Each metric is computed on a held-out test set. We repeat all experiments across 50 random seeds and report the mean and standard error for each metric.

Results The results for regression and classification are shown in Table 3 and Table 4, respectively. In regression, our objective ( NLL+MMD ) achieves better QCE than the NLL objective across all datasets while maintaining comparable (or better) NLL scores on the test set. On most datasets, we find that our method also reduces the sharpness penalty induced by post-hoc calibration. Importantly, we also observe that post-hoc calibration and our method are complementary for calibrating forecasters; post-hoc methods can guarantee calibration on held-out data, and calibration regularization mitigates the sharpness penalty. For classification, we find that our method improves accuracy, ECE, and entropy relative to the baselines we tested for calibration regularization. These trends also generally hold with post-hoc calibration, where we observe that our method achieves better calibration and accuracy across most datasets (See Table 5).

Computational Requirements Relative to training with an unregularized NLL objective, our method with MMD regularization requires an average wall clock time per epoch of 1.2x for the 5 regression datasets, and 1.3x for the 5 classification datasets.

6.3 Experiment: Decision Calibration

̸

One of the key advantages of our trainable calibration metrics is their flexibility in enforcing multiple notions of calibration. We consider a decision-making task to study the effectiveness of our method in improving decision calibration. The decision task is to choose an action a ∈ {± 1 } , whose loss ℓ ( a, y ) = 1 { a = sign( y -c ) } depends only on whether the label y exceeds a threshold c ∈ R . Given a true distribution P and a forecaster Q Y | X , the Decision Calibration Error ( DCE ) is DCE 2 ( P, Q ) = ∑ a ∈A ∥ E P [ ℓ ( Y, a )] -E X E ̂ Y ∼ Q Y | X [ ℓ ( ̂ Y , a )] ∥ 2 2 .

MMDObjective To optimize for decision calibration, we tailor the kernels k X , k Y to reflect the decision problem. More concretely, we choose a kernel function that penalizes predicted labels which are on the wrong side of c , namely k Y ( y, y ′ ) = +1 if sign( y -c ) = sign( y ′ -c ) , and -1 otherwise.

(a)

NLL

0

0.05

0.1

0.15

0.2

0.25

0.3

LCE

(b) Ours ( NLL + MMD )

To allow for gradient based optimization, we relax the objective to yield the differentiable kernel k Y ( y, y ′ ) = tanh( y -c ) · tanh( y ′ -c ) . Notice that k Y ( y, y ′ ) ≈ +1 when sign( y -c ) = sign( y -c ) , and -1 otherwise. To encourage decision calibration for all features, we let k X be an RBF kernel.

Results The results in Table 3 demonstrate that the MMD training objective tailored to the decision problem achieves the best decision calibration across all datasets. Additional results in Appendix B.3 show that using an MMD kernel tailored to a decision problem (i.e. the tanh kernel) improves decision calibration scores across all datasets.

6.4 Experiment: Local Calibration

Setup To provide intuition on how our metrics can facilitate calibration in local neighbourhoods of features, we study how location affects the reliability of forecasts on the CROP-YIELD regression dataset. We tailor the kernels k X , k Y to local calibration for geospatial neighborhoods. Namely, we select ϕ ( x ) which extracts the geospatial features (latitude, longitude) from the full input vector x . We then define k X ( x, x ′ ) = k RBF ( ϕ ( x ) , ϕ ( x ′ )) and k Y ( y, y ′ ) = k RBF ( y, y ′ ) .

Metric Extending prior work on local calibration [27], given a cdf forecaster Q Y | X , a feature embedding ϕ ( · ) and kernel k , we define Local Calibration Error ( LCE ) for regression is LCE( x ; c ) = ( ∑ i 1 { y i ≤ Q -1 Y | x i ( c ) } · k ( ϕ ( x ) , ϕ ( x i )) ) / ( ∑ i k ( ϕ ( x ) , ϕ ( x i ))) -c . The total LCE is computed as LCE total ( x ) = 1 B ∑ B i =1 LCE( x ; c i ) 2 , where c i = i B . Intuitively, LCE measures the QCE across local regions determined by ϕ ( · ) . We visualize the results in Figure 1 for B = 20 .

7 Conclusion

We introduced a flexible class of trainable calibration metrics based on distribution matching using MMD . These methods can be effectively combined with existing post-hoc calibration methods to achieve strong calibration guarantees while preserving sharpness.

Limitations Optimizing the MMD metric requires that samples from the forecast are differentiable with respect to the model parameters. This restricts the family of applicable forecasters, but differentiable relaxations such as Gumbel-softmax can mitigate this challenge. Furthermore, in practice it is often not possible to achieve zero calibration error via regularization due to computational and data limitations. To mitigate this limitation, we suggest pairing our calibration regularizers with post-hoc calibration methods. Lastly, the scale of the MMD metric is sensitive to the chosen kernel (e.g., the bandwidth of a RBF kernel), so it can be difficult to understand how calibrated a model is in an absolute sense based on the MMD metric.

References

Dry Bean Dataset. UCI Machine Learning Repository, 2020. DOI: https://doi.org/10.24432/C50S4B.

Barry Becker and Ronny Kohavi. Adult. UCI Machine Learning Repository, 1996. DOI: https://doi.org/10.24432/C5XW20.

James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyperparameter optimization. Advances in neural information processing systems , 24, 2011.

Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Meta-calibration: Learning of model calibration using differentiable expected calibration error. arXiv preprint arXiv:2106.09613 , 2021.

Jochen Bröcker. Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Society: A journal of the atmospheric sciences, applied meteorology and physical oceanography , 135(643):1512-1519, 2009.

Krisztian Buza. Feedback prediction for blogs. In Data analysis, machine learning and knowledge discovery , pages 145-152. Springer, 2014.

Joel W Cohen, Steven B Cohen, and Jessica S Banthin. The medical expenditure panel survey: a national information resource to support healthcare cost research and inform policy and practice. Medical care , pages S44-S50, 2009.

A Philip Dawid. The well-calibrated bayesian. Journal of the American Statistical Association , 77(379):605-610, 1982.

Morris H DeGroot and Stephen E Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statistician) , 32(1-2):12-22, 1983.

Cynthia Dwork, Michael P Kim, Omer Reingold, Guy N Rothblum, and Gal Yona. Outcome indistinguishability. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 1095-1108, 2021.

Tilmann Gneiting and Matthias Katzfuss. Probabilistic forecasting. Annual Review of Statistics and Its Application , 1:125-151, 2014.

Tilmann Gneiting and Roopesh Ranjan. Combining predictive distributions. Electronic Journal of Statistics , 7:1747-1782, 2013.

Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 69 (2):243-268, 2007.

Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research , 13(1):723-773, 2012.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning , pages 1321-1330. PMLR, 2017.

Kam Hamidieh. Superconductivty Data. UCI Machine Learning Repository, 2018. DOI: https://doi.org/10.24432/C53P47.

Ursula Hébert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Calibration for the (computationally-identifiable) masses. In International Conference on Machine Learning , pages 1939-1948. PMLR, 2018.

Andras Janosi, William Steinbrunn, Matthias Pfisterer, and Robert Detrano. Heart Disease. UCI Machine Learning Repository, 1988. DOI: https://doi.org/10.24432/C52P4X.

Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Michael C Mozer, and Becca Roelofs. Soft calibration objectives for neural networks. Advances in Neural Information Processing Systems , 34:29768-29779, 2021.

Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807 , 2016.

Volodymyr Kuleshov and Stefano Ermon. Estimating uncertainty online against an adversary. In Thirty-First AAAI Conference on Artificial Intelligence , 2017.

Volodymyr Kuleshov and Percy S Liang. Calibrated structured prediction. Advances in Neural Information Processing Systems , 28, 2015.

Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International conference on machine learning , pages 2796-2804. PMLR, 2018.

Meelis Kull and Peter Flach. Novel decompositions of proper scoring rules for classification: Score adjustment as precursor to calibration. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2015, Porto, Portugal, September 7-11, 2015, Proceedings, Part I 15 , pages 68-85. Springer, 2015.

Meelis Kull, Telmo Silva Filho, and Peter Flach. Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In Artificial Intelligence and Statistics , pages 623-631. PMLR, 2017.

Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In International Conference on Machine Learning , pages 2805-2814. PMLR, 2018.

Rachel Luo, Aadyot Bhatnagar, Yu Bai, Shengjia Zhao, Huan Wang, Caiming Xiong, Silvio Savarese, Stefano Ermon, Edward Schmerling, and Marco Pavone. Local calibration: metrics and recalibration. In Uncertainty in Artificial Intelligence , pages 1286-1295. PMLR, 2022.

Valery Manokhin. Multi-class probabilistic classification using inductive and cross venn-abers predictors. In Conformal and Probabilistic Prediction and Applications , pages 228-240. PMLR, 2017.

Charles Marx, Shengjia Zhao, Willie Neiswanger, and Stefano Ermon. Modular conformal calibration. In International Conference on Machine Learning , pages 15180-15195. PMLR, 2022.

Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in applied probability , 29(2):429-443, 1997.

Allan H Murphy and Robert L Winkler. Reliability of subjective probability forecasts of precipitation and temperature. Journal of the Royal Statistical Society Series C: Applied Statistics , 26(1):41-47, 1977.

Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence , volume 29, 2015.

Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning , pages 625-632, 2005.

United States. Bureau of the Census et al. Census of Population and Housing, 1990 [United States]: Summary Tape File 3A, Record Sequence Example File . 1992.

John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers , 10(3):61-74, 1999.

Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. On fairness and calibration. Advances in neural information processing systems , 30, 2017.

Teodora Popordanoska, Raphael Sayer, and Matthew Blaschko. A consistent and differentiable lp canonical calibration error estimator. Advances in Neural Information Processing Systems , 35:7933-7946, 2022.

Mihaela Rosca, Balaji Lakshminarayanan, and Shakir Mohamed. Distribution matching in variational inference. arXiv preprint arXiv:1802.06847 , 2018.

Roshni Sahoo, Shengjia Zhao, Alyssa Chen, and Stefano Ermon. Reliable decisions with threshold calibration. Advances in Neural Information Processing Systems , 34, 2021.

C. Sakar and Yomi Kastro. Online Shoppers Purchasing Intention Dataset. UCI Machine Learning Repository, 2018. DOI: https://doi.org/10.24432/C5F88Q.

Kamaljot Singh, Ranjeet Kaur Sandhu, and Dinesh Kumar. Comment volume prediction using neural networks and decision trees. In IEEE UKSim-AMSS 17th International Conference on Computer Modelling and Simulation, UKSim2015 (UKSim2015) , 2015.

Hao Song, Tom Diethe, Meelis Kull, and Peter Flach. Distribution calibration for regression. In International Conference on Machine Learning , pages 5897-5906. PMLR, 2019.

Zoltán Szabó and Bharath K Sriperumbudur. Characteristic and universal tensor product kernels. J. Mach. Learn. Res. , 18(233):1-29, 2017.

Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 3459-3467. PMLR, 2019.

Vladimir Vovk, Ivan Petej, Paolo Toccaceli, Alexander Gammerman, Ernst Ahlberg, and Lars Carlsson. Conformal calibrators. In Conformal and probabilistic prediction and applications , pages 84-99. PMLR, 2020.

David Widmann, Fredrik Lindsten, and Dave Zachariah. Calibration tests in multi-class classification: A unifying framework. Advances in Neural Information Processing Systems , 32, 2019.

David Widmann, Fredrik Lindsten, and Dave Zachariah. Calibration tests beyond classification. arXiv preprint arXiv:2210.13355 , 2022.

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8:229-256, 1992.

William Wolberg, Olvi Mangasarian, Nick Street, and W. Street. Breast Cancer Wisconsin (Diagnostic). UCI Machine Learning Repository, 1995. DOI: https://doi.org/10.24432/C5DW2B.

Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , pages 295-305, 2020.

Shengjia Zhao, Tengyu Ma, and Stefano Ermon. Individual calibration with randomized forecasting. In International Conference on Machine Learning , pages 11387-11397. PMLR, 2020.

Shengjia Zhao, Michael Kim, Roshni Sahoo, Tengyu Ma, and Stefano Ermon. Calibrating predictions to decisions: A novel approach to multi-class calibration. Advances in Neural Information Processing Systems , 34, 2021.

Johanna F Ziegel and Tilmann Gneiting. Copula calibration. 2014.

A Expressing Popular Forms of Calibration as Distribution Matching

A.1 Calibration in Regression

Here, we show how the forms of calibration listed in Table 1 can be expressed in terms of conditional distribution matching. Note that many of these forms of calibration are sensible in the classification setting as well (e.g., marginal, group, and decision calibration).

Quantile Calibration Let Y be a continuous random variable. A forecaster is quantile calibrated if

This is equivalent to stating that Q Y | X ( Y ) is uniform on the interval [0 , 1] . For continuous random variables, the probability integral transform P Y | X ( Y ) is uniform on the interval [0 , 1] . Thus, quantile calibration can be written as Q Y | X ( Y ) ≜ P Y | X ( Y ) , where ≜ denotes equality in distribution.

Threshold Calibration Let Y be a continuous random variable. A forecaster satisfies threshold calibration [39] if P ( Q Y | X ( Y ) ≤ t | Q Y | X ( y ) ≤ c ) = t for all t ∈ [0 , 1] , c ∈ [0 , 1] , and y ∈ Y . This states that Q Y | X ( Y ) is uniform, conditioned on the event Q Y | X ( y ) ≤ c . Noting that P Y | X ( Y ) is uniform, the constraint is then equivalent to the following:

Marginal Calibration A forecaster is marginally calibrated [11] if E [ Q Y | X ( y )] = P Y ( y ) for all y ∈ Y . Recall that the distribution of ̂ Y is defined by ( ̂ Y | X = x ) ∼ Q Y | x . Thus, the marginal distribution of ̂ Y is given by ̂ Y ∼ E [ Q Y | X ] . So marginal calibration states that ̂ Y and Y have the same marginal cdf or, equivalently, ̂ Y ≜ Y .

Decision Calibration Zhao et al. [52] introduce the concept of L K decision calibration for multiclass classification, where Y = { 0 , 1 } C is the set of onehot vectors over C classes.

Definition A.1 ( L K -Decision Calibration, Zhao et al. [52]) . Let L K be the set of all loss functions with K actions L K = { ℓ : Y × A → R , |A| = K } , we say that a prediction Q Y | X is L K -decision calibrated (with respect to P Y | X ) if ∀ ℓ ∈ L K and δ ∈ ∆ L K

where ∆ L K is the set of all Bayes decision rules over loss functions with K actions.

They show that L K decision calibration can equivalently be expressed as requiring that E [( ̂ Y -Y ) 1 { δ ( Q Y | X ) = a } ] = 0 for all δ ∈ ∆ L K . We can rewrite this as

note that E [ Y | δ ( Q Y | X ) = a ] gives the conditional distribution of ( Y | δ ( Q Y | X ) = a ) due to the onehot representation of Y . Thus, it is equivalent to require that

where ≜ denotes equality in distribution.

Group Calibration Group calibration [20] says that forecasts are marginally calibrated for each subgroup (e.g., subgroups defined by gender or age). We can write this as ̂ Y ≜ Y | Group( X ) , where Group( X ) is a variable indicating which subgroup includes the feature vector X .

Distribution Calibration A forecaster satisfies distribution calibration [42] if for all possible forecasts Q 0 and labels y ∈ Y , we have P ( Y ≤ y | Q Y | X = Q 0 ) = Q 0 ( y ) . This is equivalent to requiring P Y | X ( y ) | Q Y | X = Q Y | X ( y ) for all y ∈ Y . Since the cdfs take the same value at all y ∈ Y , the random variables Y and ̂ Y have the same distribution, giving us that ̂ Y ≜ Y | Q Y | X .

Individual Calibration Let Y be a real-valued continuous random variable. Individual calibration [51] states that P ( Q Y | x ( Y ) ≤ t ) = t , for all t ∈ [0 , 1] and x ∈ X . Letting U ∼ Unif(0 , 1) , we can write this as

Local Calibration A forecaster is locally calibrated [27] if Q Y | X and P Y | X are equal, on average, in any neighborhood of the feature space. Similarity in feature space is defined by a kernel k ( x, x ′ ) . In regression, local calibration for quantiles says that

This can equivalently be stated as

or, in terms of the canonical feature mapping,

Thus, it is sufficient for E [ Q Y | X ] and E [ P Y | X ] to have the same distribution conditioned on the feature mapping ϕ ( X ) . This can be written succinctly as

A.2 Calibration in Classification

Here, we express three popular forms of calibration from the classification literature-namely, canonical, top-label, and marginal calibration [44, Eq (1-3)]-in terms of distribution matching constraints. For the entirety of this section, Y is a discrete random variable taking values in the set Y = { 1 , 2 , . . . , m } , and q Y | x is the probability mass function of the forecast given the features X = x .

Canonical Calibration Aforecaster satisfies canonical calibration if P ( Y = y | q Y | X ) = q Y | X ( y ) for all y ∈ Y . This says that the label follows the same distribution under the forecast and the true distribution, conditioned on q Y | X . In other words, Y ≜ ̂ Y | q Y | X .

Top-Label Calibration A forecaster satisfies top-label calibration if P ( Y = Y ∗ | q Y | X ( Y ∗ ) ) = q Y | X ( Y ∗ ) , where Y ∗ := arg max y ∈Y q Y | X ( y ) is the mode of the forecast. This equation can be rewritten in terms of the indicators

and the indicator variables are equal in expectation if and only if they are equal in distribution. Thus, top-label calibration is equivalent to 1 { Y = Y ∗ } ≜ 1 { ̂ Y = Y ∗ } | q Y | X ( Y ∗ ) .

Marginal Calibration A forecaster satisfies marginal calibration if P ( Y = y | q Y | X ( y ) ) = q Y | X ( y ) for all y ∈ Y . Similar to top-label calibration, this can be written as

and the indicator variables are equal in expectation if and only if they are equal in distribution. Thus, marginal calibration can be written as 1 { Y = y } ≜ 1 { ̂ Y = y } | q Y | X ( y ) , for all y ∈ Y .

B Experimental Setup and Additional Results

B.1 Reproducibility

In order to reproduce the experiments and results obtained in this paper, we provide details about the hyperparameter tuning and required computational resources.

Hyperparameter Search: We use TPES [3] to perform hyperparameter optimization. For all experiments, we vary:

Layer sizes between 32 and 512

RBF kernel bandwidths between 0.001 and 200

Batch sizes between 16 and 512, with and without batch normalization

Learning rates between 10 -7 and 10 -1 .

The loss mixture weight λ (as in NLL+ λ · MMD and XE+ λ · MMD ) between 0.1 and 1000.

For each dataset, we test 100 hyperparameter settings for each training objective for regression { NLL , NLL+MMD } , and classification setting { XE , XE+MMCE , XE+KDE , XE+MMD } . For each run, we enable early-stopping if the validation loss does not improve for more than 50 epochs. In the regression setting, we pick the best performing run based on the validation NLL . For classification, we select the best performing run based on the validation accuracy, with validation ECE used for break ties. For each model and dataset, the best performing model is then re-run with 50 random seeds to gather information about standard errors and statistical significance. To capture variability across different seeds, we report the standard error of the mean in Tables 3, 4, and 5.

Kernel Bandwidth We select the RBF kernel bandwidth for training on each dataset using the aforementioned hyperparameter optimization. For each dataset in Tables 4 and 5, the KDE metric we report is for the bandwidth selected through a held-out validation set, scaled so that the error is of a comparable order of magnitude across datasets (since multiplying a kernel by a positive scalar preserves the kernel properties).

Baselines In the classification setting, we compare our approach to two state-of-the-art trainable calibration methods, MMCE [26] and KDE [37]. For MMCE , we follow the same experimental setup as the official author implementation, where the loss mixture weight λ and the Laplacian kernel bandwidth are chosen via a held-out validation set. For KDE , we use the L 1 canonical calibration using the ECE KDE estimator, calculated according to Equation (9) from Popordanoska et al. [37]. We follow the same experimental setup as the official author implementation, where the kernel bandwidth is chosen using leave-one-out-likelihood (LOO MLE), and the loss mixture weight λ is chosen via a held-out validation set.

Computational Requirements: All experiments were conducted on a single CPU machine (Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz), utilizing 8 cores per experiment. When run on a CPU, the training time for a single model ranges from 5 to 15 minutes, depending on the dataset size. It is possible to perform a full hyperparameter sweep using this setup in ∼ 24 hours.

To accelerate training, we have also run some experiments using an 11GB NVIDIA GeForce GTX 1080 Ti. When run on a GPU, the training time for a single model ranges from 2 to 5 minutes, depending on the dataset size, and a hyperparameter sweep takes ∼ 10 hours.

US Agriculture Dataset: We introduce a new CROP-YIELD dataset for predicting yearly wheat crop yield from geospatial weather data for different counties across the US. We use NOAA database which contains weather data from thousands of stations across the United States. For each county, we track the weather sequence of each year into a few summary statistics for each month (average/maximum/minimum temperatures, precipitation, cooling/heating degree days). To study how climate change and location affect crop yield, we match crop yield data from the NASS dataset to weather stations in the NOAA database, and learn a function to predict crop yield from weather data. We specifically considered the crop yield of wheat across different counties in the US between 1990-2007. Since crop yield data for 1995 is the least sparse, we hold out data from 1995 for visualization, and train our forecasters on the remaining data.

B.2 Additional Experimental Results

SUPERCONDUCTIVITY

NLL

QCE

DCE

3.369

3.35

3.311

3.333

3.298

3.345

3.318

3.291

0.053

0.062

0.038

0.036

0.055

0.041

0.036

0.034

0.182

0.197

0.216

0.208

0.262

0.208

0.181

0.182

FB-COMMENT

NLL

QCE

DCE

0.637

0.477

0.409

0.59

0.569

0.479

0.469

0.458

0.268

0.294

0.276

0.278

0.258

0.223

0.23

0.231

3.15

3.14

3.146

3.102

3.119

3.14

3.149

3.137

1

2

5

10

20

50

100

200

B.3 Experiment: Decision Calibration

̸

As an illustrative example of how trainable calibration metrics can improve calibration for a specific decision-making problem, we consider a common real-world decision problem, requiring us to make an action whose utility depends only on whether the label y is greater or less than a particular threshold c . Concretely, the decision loss ℓ : A×Y → R , for an action a ∈ {-1 , +1 } and a label y ∈ Y , is defined as ℓ ( a, y ) = 1 { a = sign( y -c ) } .

Setup In order to achieve decision calibration, we tailor our kernels k X , k Y to closely follow the decision problem. More concretely, we choose a kernel function that penalizes predicted labels which are on the wrong side of c , namely k Y ( y, y ′ ) = tanh( y -c ) · tanh( y ′ -c ) . Notice that k Y ( y, y ′ ) ≈ +1 when sign( y -c ) = sign( y -c ) , and -1 otherwise. Thus, the calibration metric gives the model training feedback that is well aligned with the decision problem. To encourage decision calibration across all features, we set k X to be the universal RBF kernel.

As part of this experiment, we investigate whether tailoring an MMD kernel to the decision problem will improve decision calibration over using a universal RBF kernel for both k X , k Y .

Metric To evaluate the effectiveness of methods, we compute the standard Decision Calibration Error ( DCE ) across all regression datasets, defined as

where P is the true distribution over X × Y , and Q Y | X is a forecaster.

Additional Results: Choice of kernel We present two sets of results. The first set of results is shown in Table 7, comparing models trained using our objective ( NLL + MMD ), where one uses a kernel k Y that is tailored to the decision problem ( tanh ), while the other uses a universal RBF kernel. As described in Section 6.3, we observe that using the tanh kernel clearly achieves better decision calibration across all dataset, with minimal degradation in sharpness, i.e. NLL . This demonstrates that we can improve decision calibration using our trainable calibration metrics by tailoring our kernels to closely follow the decision problem.

The second set of results, shown in Figure 2, further demonstrates the relationship between our trainable calibration metrics and decision calibration throughout model training. By tailoring our kernels to closely follow the decision problem, we observe that decision calibration is optimized throughout training. In contrast, using objectives that are agnostic to the decision problem (i.e. NLL , or MMD with RBF kernels) will lead to noticeably worse decision calibration, with little to no improvement throughout training. The same trend is observed on a held-out validation set, as shown in Figure 3.

0

20

40

60

80

Epochs

0.00

0.05

0.10

0.15

0.20

0.25

0.30

Train DCE

NLL

NLL + MMD: Tanh kernel

NLL + MMD: RBF kernel

(a) CRIME

0

50

100

150

Epochs

0.03

0.04

0.05

0.06

0.07

0.08

Train DCE

NLL

NLL + MMD: Tanh kernel

NLL + MMD: RBF kernel

(b) MEDICAL-EXPENDITURE

0

10

20

30

40

50

60

Epochs

0.35

0.40

0.45

0.50

0.55

Train DCE

(c) BLOG

0

25

50

75

100

125

150

Epochs

0.00

0.05

0.10

0.15

0.20

Train DCE

(d) SUPERCONDUCTIVITY

0

20

40

60

80

100

120

Epochs

0.38

0.39

0.40

0.41

0.42

0.43

Train DCE

(e) FB-COMMENT

0

20

40

60

80

Epochs

0.0

0.1

0.2

0.3

NLL

NLL + MMD: Tanh kernel

NLL + MMD: RBF kernel

(a) CRIME

0

50

100

150

Epochs

0.04

0.05

0.06

0.07

NLL

NLL + MMD: Tanh kernel

NLL + MMD: RBF kernel

(b) MEDICAL-EXPENDITURE

0

10

20

30

40

50

60

Epochs

0.35

0.40

0.45

0.50

(c) BLOG

0

25

50

75

100

125

150

Epochs

0.00

0.05

0.10

0.15

0.20

0.25

(d) SUPERCONDUCTIVITY

0

20

40

60

80

100

120

Epochs

0.38

0.39

0.40

0.41

(e) FB-COMMENT

Peer reviews:
comments: SUMMARY:
Accurate probability prediction is a crucial aspect of a trustworthy model, especially for deep neural networks. This paper focuses on the model calibration problem and proposes a unified framework via distribution matching. To leverage the distribution matching, they used 
Maximum Mean Discrepancy to estimate the miscalibration, where MMD is a popular distribution measure and used in many distribution alignment problems in recent years. The main usage of MMD in this paper is to treat it as a trainable regularizer. In the experiment section, 
they demonstrate their effectiveness in performing proper distribution alignments.

STRENGTHS:
1. They introduced a unified framework for calibration metrics as distribution-matching problems, which is a very interesting view for uncertainty calibration.
2. They proposed a training-time model calibration method by introducing kernel-based calibration metrics.

WEAKNESSES:
The paper does not have main weakness, if some, they are:

1. How to apply this method to multi-class classification? I did not see the related experiments. It seems that all the used datasets are regression datasets, but Line 6 said it is also suitable for classification tasks.

2. Deep comparisons with similar kernel methods, like [1]. I did not see the description of their difference. They also used the kernel method to perform model calibration.

[1] Trainable Calibration Measures For Neural Networks From Kernel Mean Embeddings. ICML 2018

3. What are the n and d in Table 2? Why do you use the different n and d regarding different datasets?

4. How to demonstrate the prediction performance improvement using the proposed methods? I did not find the clue in Table 2.

QUESTIONS:
Please refer to the weaknesses.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a general framework for train-time calibration based on kernel MMD. The authors demonstrate empirically that choosing good kernels (based on the task at hand) in their framework can lead to improvements in calibration while maintaining sharpness of predictions.\n\nUpdate: I have increased my score to a 6 in light of the authors' responses. I think the empirical portions of the paper could still be improved (ideally comparing to more training-time calibration methods), but I believe the sharpness improvements over MMCE are interesting and useful.\n\nSTRENGTHS:\n- **Originality:** While kernel-based calibration methods have been studied in the literature, the perspective provided in this work is original in that the authors show how various different types of calibration can be captured by their MMD framework.\n- **Quality:** The quality of the work is overall good, with appropriate support and examples provided for the claims in the paper.\n- **Clarity:** The paper is mostly well-written. However, there are implementation details regarding the MMD approach that I feel are important and missing (detailed further in weaknesses). \n- **Significance:** This paper adds to the growing body of work showing the benefits of kernel-based methods for improving calibration. However, it is difficult to assess the relative significance of this work (compared to, for example [1]) due to the limited experiments.\n\n[1] https://proceedings.mlr.press/v80/kumar18a.html\n\nWEAKNESSES:\n- **Motivation for Implementation Details:** It would be very helpful to accompany the discussion in Section 4 with an idea of how the conditional kernel MMD is implemented in practice and why a practitioner would make certain decisions. For example, is using a model parameterization like the one in the paper actually that necessary (i.e. predicting parameters of a Gaussian)? Can one not just take a similar approach to [1]?\n- **Experiments:** The experiments in the paper compare to a baseline of training using NLL (and post-hoc calibration), with no comparisons to other train-time calibration approaches. As a result, it is difficult to assess the relative worth of the proposed methodology compared to existing approaches. \n- **Contextualization:** The MMD framework proposed in this paper is very similar to the ideas in [1], and I think the paper would benefit from more direct comparisons to [1] and related prior work that highlight the contributions of this work.\n\nTo summarize, I feel that the lack of implementation details + lack of comparison to other train-time calibration approaches limit the contribution of the paper.\n\nQUESTIONS:\n- How much longer does it take to train (wall clock time) using the MMD regularization? \n- How important is the choice of output distribution for the model considered? In other words, did you consider predicting the parameters of a distribution other than Gaussian and comparing? \n- As asked in the weaknesses, can we relax the need to differentiably sample from the predictive distribution?\n- In the supplementary material, I believe under distribution calibration it should be $P_{Y \\mid X}(y) \\mid Q_{Y \\mid X} = Q_{Y \\mid X}(y)$.\n\nLIMITATIONS:\nWhile the authors do discuss limitations, I feel the limitations mentioned should have received further exposition in the main body of the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces differentiable kernel-based calibration metrics that unify and generalize popular forms of calibration.\nThe notions of calibration are presented as distribution matching constraints by using various kernels in the Maximum Mean Discrepancy (MMD) metric.\nThrough empirical evaluation, it is demonstrated that incorporating these metrics as regularizers enhances calibration, sharpness and facilitates better decision-making in regression scenarios.\n\nSTRENGTHS:\n- The proposed metrics can be integrated in gradient based optimization alongside proper scoring rules.\n- The paper extends on the discussion regarding the connection between calibration and decision making, which offers valuable insights for practical problems.  \n- The empirical results are sufficiently extensive and verify the effectiveness of integrating the proposed metrics into training objectives, as a means to mitigate calibration error.\n\nWEAKNESSES:\nThe related work section does not sufficiently discuss prior works, and fails to highlight the specific advancements made by this paper in comparison. As the current paper proposes a kernel-based differentiable metric to be used as a regularizer alongside a proper scoring rule, there has to be a discussion of existing trainable calibration methods, especially kernel-based methods like [Kumar et al., 2018, Zhang et al., 2020, Popordanoska et al., 2022]. \n\n\n*[Zhang et al., 2020] Jize Zhang, Bhavya Kailkhura, and T Han. Mix-n-Match: Ensemble and compositional methods for uncertainty calibration in deep learning. In ICML 2020*\n\n*[Popordanoska et al., 2022] Teodora Popordanoska, Raphael Sayer, Matthew B. Blaschko A Consistent and Differentiable Lp Canonical Calibration Error Estimator. In NeurIPS 2022*\n\nQUESTIONS:\n- The abstract claims that the proposed metric generalizes popular forms of calibration for classification and regression. Could you clarify the relationship of the metric with notions of calibration used in classification, i.e. top-label, marginal and canonical (see e.g. Eq. 1 2 and 3 in [Vaicenavicius et al., 2019, Kull et al., 2019]? \n- What is the computational overhead of the proposed method?\n- How is the bandwidth of the kernels chosen?\n\n*[Vaicenavicius et al. 2019] Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, Thomas B. Schön Evaluating model calibration in classification. In AISTATS 2019.*\n\n*[Kull et al. 2019] Meelis Kull, Miquel Perello-Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, Peter Flach Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration. In NeurIPS 2019*\n\nLIMITATIONS:\nLimitations are discussed in section 7.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCalibration as a training regularizer:\n\nThis paper focuses on investigating how to incorporate calibration metrics as training regularizers. It starts with summarizing different calibration metrics using a unified view from distribution matching. The paper then proposed a kernel-based general metric that maps labels and conditioning variables to an RKHS space and measures the maximum mean discrepancy on the kernel space. The paper then extends the metric to the decision-making space with two concrete examples. In experiments, combined with post-hoc calibration, using the proposed calibration kernel metric as a trainable regularizer can achieve a better evaluation of calibration in regression problems.\n\nSTRENGTHS:\nPros:\nThe paper provides a unified view of many different calibration metrics.\n\nThe kernel-based calibration regularizer is novel.\n\nThe paper is relatively clearly written.\n\nWEAKNESSES:\nCons:\n\n1. The decision calibration seems to be a natural extension of the section 4, as the decision calibration framework has been proposed before. So the main contribution is based on the kernel-MMD-based calibration metric, which is relatively straightforward.\n\n2. Also, the paper only suggests using the metric as a regularizer to incentivize decision calibration during training but does not really guarantee anything or analyze formally the impact on the decision calibration scenario.\n\n3. The trainable regularizer needs to be combined with post-hoc calibration method, which may raise the question how effective the method is. Table 2 shows that the MMD alone does not always provide enough improvement.\n\nQUESTIONS:\nI may have missed it, but I am a little confused by the correspondence between the trainable calibration metric and the evaluation. Table 2 shows training using NLL and the proposed method (individual calibration) and evaluating using both quantile calibration and decision calibration. Why quantile calibration is used here? Why not individual calibration in the evaluation? \n\nWhat happens if the problem is a classification problem? The evaluation of calibration is classification is usually tricky. How much the trainable regularize can benefit there?\n\nMinor: table caption is usually on top of the table.\n\n=====\nAfter reading the rebuttal, I raised my score to 5 but still concerned about the lack of theoretical guarantees and the requirement of additional calibration data and step.\n\nLIMITATIONS:\nThe paper discussed technical limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}