SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Loss estimation says that the expected loss of each action is equal under the true distribution and the forecast. This means we can estimate the loss on unlabeled data. No regret says that the Bayes optimal action according to the forecast  $\delta_{\ell}^{*}(Q_{Y|x})$  achieves expected loss less than or equal to any decision policy  $\delta \in \Delta (\mathcal{Z})$  that depends only on  $z$ . If we condition on the forecast by choosing  $z = Q_{Y|x}$ , then a decision-maker who only has access to the forecast is incentivized to act as if the forecast is correct. Additionally, we consider cases where the loss function is not known in advance. For example, when a vendor releases a foundation model to serve a large population of users, it is likely that different users have distinct loss functions. Thus, we also show settings in which we can achieve accurate loss estimation and no regret for all loss functions  $\ell$  included in a family of losses  $\mathcal{L}$ .
A General Recipe for Decision Calibration We now describe an approach to measure decision calibration using our kernel-based metrics. For each action  $a \in \mathcal{A}$  and loss  $\ell \in \mathcal{L}$ , we want to measure the discrepancy between the expectation of the loss  $\ell(a, Y)$  under  $P$  and  $Q$ . To achieve this, we define the feature map that computes the losses of all actions  $\phi(y) = (\ell(a, y))_{a \in \mathcal{A}, \ell \in \mathcal{L}}$ . This assigns to  $P$  and  $Q$  the mean embeddings  $\mu_P = \mathbb{E}_P[(\ell(a, Y))_{a \in \mathcal{A}, \ell \in \mathcal{L}}]$  and  $\mu_Q = \mathbb{E}_Q[(\ell(a, \widehat{Y}))_{a \in \mathcal{A}, \ell \in \mathcal{L}}]$ , respectively. Letting  $\mathcal{F}$  be the unit ball of the Hilbert space associated with this feature map, the mean embedding form of MMD gives us that  $\mathrm{MMD}(\mathcal{F}, P, Q)^2 = \| \mu_P - \mu_Q \|_2^2 = \sum_{a \in \mathcal{A}} \sum_{\ell \in \mathcal{L}} \left( \mathbb{E}_P[(\ell(a, Y)) - \mathbb{E}_Q[\ell(a, \widehat{Y})]] \right)^2$ . When either  $\mathcal{A}$  or  $\mathcal{L}$  is infinite, the corresponding sum is replaced by an integral. This metric measures whether each action has the same expected loss marginally under the true distribution and the forecasts. Thus, accurate loss estimation is only guaranteed marginally and the no regret result is relative to constant decision functions that choose the same action for all examples.
Next, we refine the decision calibration results to hold conditionally on a variable  $Z$ . Recall that the kernel over  $y$  is given by  $k_{y}(y,y^{\prime}) = \langle \phi (y),\phi (y^{\prime})\rangle$ . We lift this kernel into  $\mathcal{V}\times \mathcal{Z}$  by defining  $k((y,z),(y',z')) = k_{y}(y,y')k_{z}(z,z')$  where  $k_{z}$  is a universal kernel. The MMD metric corresponding to this kernel measures decision calibration conditioned on  $Z$ . If we choose  $Z$  to be the forecast  $Z = Q_{Y|X}$ , then the no regret guarantee says that acting as if the forecasts were correct is an optimal decision policy among all policies depending only on the forecast.
Note that the feature maps defined above are infinite dimensional when  $\mathcal{A}$  or  $\mathcal{L}$  are infinite. However, applying the kernel trick sometimes allows us to estimate the MMD metric efficiently. Still, optimizing the MMD metric may be difficult in practice, especially when the conditioning variable  $Z$  is very expressive. We suggest using the metric as a regularizer to incentivize decision calibration during training, and then still apply post-hoc recalibration if needed. We now give two concrete examples, one where the action space is infinite and one where the set of loss functions is infinite.
Example 1 (Point Estimate Decision). Consider a regression problem in which a decision-maker chooses a point estimate  $a \in \mathbb{R}$  and incurs the squared loss  $\ell(a, y) = (a - y)^2$ . The expected loss of an action  $a$  is  $\mathbb{E}_P[(a - Y)^2] = (a - \mathbb{E}_P[Y])^2 + \mathbb{E}_P[Y^2] - \mathbb{E}_P[Y]^2$ . Observe that the expected loss only depends on  $Y$  through the first two moments,  $\mathbb{E}_P[Y]$  and  $\mathbb{E}_P[Y^2]$ . We use the representation  $\phi(y) = (y, y^2)$ , so that the kernel is  $k(y, y') = \langle (y, y^2), (y', y'^2) \rangle$ . From the mean embedding form of MMD, we see that  $\mathrm{MMD}(\mathcal{F}, P, Q) = \| \mu_P - \mu_Q \|^2 = (\mathbb{E}[Y] - \mathbb{E}_Q[\widehat{Y}])^2 + (\mathbb{E}_P[Y^2] - \mathbb{E}_Q[\widehat{Y}^2])^2$ . Therefore, the metric is zero exactly when the expected loss of each action  $a$  is the same under the true distribution and the forecasts. The MMD metric incentivizes the forecasts to accurately reflect the marginal loss of each action. However, we often want to know the optimal action given a particular forecast, not marginally. We can now choose the conditioning variable to refine the guarantee. Let  $Z$  be the forecast  $Z = Q_{Y|X}$  and choose the kernel  $k((y, z), (y', z')) = k_y(y, y') k_z(z, z')$  where  $k_y(y, y') = \langle (y, y^2), (y', y'^2) \rangle$  as before and  $k_z(z, z')$  is universal. For this kernel, the MMD will equal zero if and only if  $\mathbb{E}_P[Y | Q_{Y|X}] = \mathbb{E}_Q[\widehat{Y} | Q_{Y|X}]$  and  $\mathbb{E}_P[Y^2 | Q_{Y|X}] = \mathbb{E}_Q[\widehat{Y}^2|$
$Q_{Y|X}]$  almost surely. The Bayes optimal action suggested by the forecast is optimal among decision policies that depend only on the forecast:
$$
\mathbb {E} _ {P} \left[ \ell (\delta^ {*} (Q _ {Y | X}), Y) \right] \leq \mathbb {E} _ {P} \left[ \ell (\delta (Q _ {Y | X}), Y) \right], \quad \forall \delta \in \Delta (Q _ {Y | X}) \tag {5}
$$
In other words, the decision maker is incentivized to behave as if the forecasts are correct.
Example 2 (Threshold Decision). The above example had an infinite action space and a single loss function. Here, we consider a problem where the action space is small but the set of loss functions is infinite. This setting applies for a model vendor who serves a model to multiple decision makers, each with a potentially different loss function. For example, suppose that  $Y \in [0,T]$  is a blood value for a patient for some  $T > 0$ , and the decision task is to determine whether  $Y$  exceeds a threshold  $t \in [0,T]$  that indicates eligibility for a clinical trial. Formally, the decision maker chooses a threshold  $a \in \{\pm 1\}$  to minimize the loss  $\ell_t(a,y) = \mathbb{1}\{a \neq \mathrm{sign}(y - t)\}$ . Our goal is to simultaneously calibrate our forecasts for all thresholds  $t \in [0,T]$ , i.e.  $\mathcal{L} = \{\ell_t : t \in [0,T]\}$ . In this case, we choose the feature map  $\phi(y) = (\mathbb{1}\{y \geq t\})_{t \in [0,T]}$ . The corresponding kernel is  $k(y,y') = \langle \phi(y), \phi(y') \rangle = \min\{y,y'\}$ . The mean embeddings are  $\mathbb{E}_P[\phi(Y)] = P_Y$  and  $\mathbb{E}_Q[\phi(\hat{Y})] = Q_Y$ , the marginal cdfs for  $Y$  and  $Y_Q$ . Thus, the MMD metric measures  $\| P_Y - Q_Y\|_2^2 = \int_{t=0}^{T} (P_Y(t) - Q_Y(t))^2 dt$ , which is zero exactly when  $P_Y = Q_Y$ . When  $\mathrm{MMD}(\mathcal{F}, P, Q) = 0$ , the expected loss of each action  $a$  is the same under the true distribution and the forecasts, for all  $\ell \in \mathcal{L}$ . Similar to the previous example, we can achieve conditional decision calibration by adding a universal kernel over the conditioning variable.
Discussion The connections between decision-making and calibration have been previously studied by Zhao et al. [52] and Sahoo et al. [39]. Here, we extend results from Zhao et al. [52] to include infinite action spaces and further refine the decision guarantees to depend on an arbitrary conditioning variable. In Example 2, we connect our methods to the special case of threshold calibration [39]. Notably, our framework provides a strategy to optimize decision calibration at training time.
The main goals of our experiments are to: (1) compare the performance our method with other trainable calibration metrics, and with standard post-hoc recalibration methods; (2) study the impact of tailoring the MMD kernel to a specific decision task; and (3) study whether trainable calibration metrics can be used to improve calibration across local neighborhoods of data.
6.1 Datasets
We consider standard benchmark datasets and datasets relating to decision-making tasks. For each dataset, we randomly assign  $70\%$  of the dataset for training,  $10\%$  for validation, and  $20\%$  for testing.
Regression Datasets We use four tabular UCI datasets (SUPERCONDUCTIVITY [16], CRIME [34], BLOG [6], FB-COMMENT [41]), as well as the Medical Expenditure Panel Survey dataset (MEDICALEXPENDITURE [7]). They are common benchmarks in uncertainty quantification literature [23, 39]. The number of features ranges from  $d = 53$  to 280. The total number of examples  $n$ , and features  $d$  for each dataset can be found in Table 3.
Classification Datasets We use five tabular UCI datasets: BREAST-CANCER [49], HEART-DISEASE [18], ONLINE-SHOPPERS [40], DRY-BEAN [1], and ADULT [2]. The datasets range from  $m = 2$  to 7 label classes, and  $d = 16$  to 104 features. The total number of examples  $n$ , features  $d$ , and classes  $m$  for each dataset can be found in Table 4.
Crop Yield Dataset We introduce a CROP-YIELD dataset, where the task is to predict yearly wheat crop yield from weather data for different counties across the US. We use the NOAA database to track summary weather statistics across each year (temperature, precipitation, cooling/heating degree days), and the NASS database to track annual wheat crop yield. The dataset spans from 1990-2007, where data from 1995 was held-out for visualization, and the remaining data was used for training. We use this dataset to evaluate local calibration with respect to the latitude and longitude coordinates.
$^{2}$ Code to reproduce experiments can be found at https://github.com/kernel-calibration/kernel-calibration/.
Table 3: Comparison of model performance on five different regression datasets. Models were trained with two objectives: NLL and NLL + MMD (Ours). We display the metrics on the test set for each training procedure, both with and without post-hoc Quantile Calibration [23] fit to the validation set.  $n$  is the number of examples in the dataset and  $d$  is the number of features.
Dataset	Training Objective	NLL ↓	Quantile Cal. ↓	Decision Cal. ↓
CRIME	NLL	-0.716 ± 0.007	0.220 ± 0.004	0.151 ± 0.009
+ Post-hoc	-0.387 ± 0.012	0.154 ± 0.003	0.089 ± 0.005
n = 1992	NLL + MMD (Ours)	-0.778 ± 0.008	0.164 ± 0.004	0.042 ± 0.011
d = 102	+ Post-hoc	-0.660 ± 0.014	0.105 ± 0.004	0.041 ± 0.005
BLOG	NLL	0.997 ± 0.040	0.402 ± 0.006	4.087 ± 0.004
+ Post-hoc	0.624 ± 0.021	0.053 ± 0.001	4.090 ± 0.001
n = 52397	NLL + MMD (Ours)	0.957 ± 0.008	0.302 ± 0.002	4.051 ± 0.002
d = 280	+ Post-hoc	0.945 ± 0.007	0.042 ± 0.001	4.067 ± 0.001
MEDICAL-EXPENDITURE	NLL	1.535 ± 0.000	0.078 ± 0.001	0.476 ± 0.002
+ Post-hoc	1.808 ± 0.002	0.059 ± 0.001	0.472 ± 0.002
n = 33005	NLL + MMD (Ours)	1.532 ± 0.000	0.059 ± 0.001	0.438 ± 0.002
d = 107	+ Post-hoc	1.780 ± 0.001	0.045 ± 0.000	0.431 ± 0.001
SUPERCONDUCTIVITY	NLL	3.375 ± 0.008	0.062 ± 0.003	0.211 ± 0.003
+ Post-hoc	3.707 ± 0.007	0.066 ± 0.001	0.202 ± 0.003
n = 21264	NLL + MMD (Ours)	3.269 ± 0.012	0.042 ± 0.003	0.182 ± 0.003
d = 81	+ Post-hoc	3.643 ± 0.010	0.050 ± 0.002	0.186 ± 0.003
FB-COMMENT	NLL	0.634 ± 0.016	0.334 ± 0.004	3.151 ± 0.003
+ Post-hoc	0.734 ± 0.015	0.063 ± 0.001	3.151 ± 0.002
n = 40949	NLL + MMD (Ours)	0.605 ± 0.004	0.258 ± 0.003	3.131 ± 0.003
d = 53	+ Post-hoc	0.639 ± 0.005	0.054 ± 0.001	3.138 ± 0.002
6.2 Experimental Setup and Baselines
Models To represent the model uncertainty, we use a feed-forward neural network with three fully-connected layers. In the classification setting, the model outputs the logits for all  $m$  classes. In the regression setting, the model outputs the predicted mean  $\mu(Y)$  and variance  $\sigma^2(Y)$ , which are used to parameterize a Gaussian probability distribution. This allows us to tractably compute the inverse cdf  $Q_{Y|X}^{-1}$  and perform differential sampling, enabling us to test our calibration metrics in multiple settings. See Appendix B for more details on model hyperparameter tuning.
Training Objectives In the regression setting, we consider two training objectives: Negative Log-likelihood (NLL), and our trainable calibration metric as a regularizer (NLL + MMD). For MMD, we enforce a notion of individual calibration (Table 1) by choosing the conditioning variable  $Z = X$  to match the distributions of  $(X, \hat{Y})$  and  $(X, Y)$ . More specifically, we optimize an MMD objective where the kernel  $k$  is decomposed among the pair of random variables as  $k((x, y), (x', y')) = k_X(x, x') \cdot k_Y(y, y')$ . The form of  $k_X$  and  $k_Y$  varies across experiments. In the classification setting, we consider cross-entropy (XE) loss, and three trainable calibration metrics as regularizers: MMCE [26], KDE [37], and MMD (Ours).
Baselines In the regression setting, we compare our approach against an uncalibrated forecaster trained with NLL, and with post-hoc recalibration using isotonic regression [23]. In the classification setting, we compare our approach against an uncalibrated forecaster trained with XE loss. Furthermore, we compare our approach to two trainable calibration metrics (MMCE [26] and KDE [37]). We compare all approaches against temperature-scaling [15], a standard post-hoc recalibration method. For both classification and regression, we train the post-hoc recalibration methods on a validation set, and subsequently evaluate on the held-out test set.
Metrics To quantify predictive performance in regression, we evaluate the negative log-likelihood, quantile calibration error (QCE) [23], and decision calibration error (see Section 6.3). In classification, we compute accuracy, expected calibration error (calibration), and average Shannon entropy (sharpness) of the forecast. We also report kernel calibration error (KCE), defined as the MMD
Table 4: Experimental results for five tabular classification tasks, comparing MMCE regularization [26], KDE regularization [37], and MMD regularization (Ours), alongside a standard cross-entropy (XE) loss without post-hoc recalibration. Here,  $n$  is the number of examples in the dataset,  $d$  is the number of features, and  $m$  is the number of classes.
Dataset	Training Objective	Accuracy ↑	ECE ↓	Entropy ↓	KCE ↓
BREAST-CANCER	XE	95.372 ± 0.160	0.194 ± 0.003	0.453 ± 0.004	6.781 ± 0.601
n = 569	XE + MMCE	94.770 ± 0.147	0.060 ± 0.001	0.076 ± 0.002	0.392 ± 0.086
d = 30	XE + ECE KDE	94.351 ± 0.163	0.062 ± 0.001	0.065 ± 0.001	0.225 ± 0.083
m = 2	XE + MMD (Ours)	95.789 ± 0.060	0.052 ± 0.000	0.006 ± 0.000	0.014 ± 0.000
HEART-DISEASE	XE	55.904 ± 0.196	0.373 ± 0.003	1.512 ± 0.007	0.581 ± 0.016
n = 921	XE + MMCE	60.787 ± 0.208	0.267 ± 0.002	0.947 ± 0.003	0.097 ± 0.002
d = 23	XE + ECE KDE	50.036 ± 2.801	0.304 ± 0.012	1.392 ± 0.016	0.450 ± 0.050
m = 5	XE + MMD (Ours)	61.516 ± 0.255	0.261 ± 0.003	0.945 ± 0.006	0.077 ± 0.002
ONLINE-SHOPPERS	XE	89.816 ± 0.037	0.129 ± 0.001	0.221 ± 0.003	0.022 ± 0.001
n = 12330	XE + MMCE	89.933 ± 0.036	0.128 ± 0.001	0.220 ± 0.002	-0.019 ± 0.001
d = 28	XE + ECE KDE	90.019 ± 0.034	0.127 ± 0.000	0.225 ± 0.002	-0.022 ± 0.001
m = 2	XE + MMD (Ours)	89.976 ± 0.031	0.127 ± 0.001	0.218 ± 0.002	-0.026 ± 0.000
DRY-BEAN	XE	92.071 ± 0.025	0.113 ± 0.000	0.264 ± 0.001	0.061 ± 0.002
n = 13612	XE + MMCE	92.772 ± 0.035	0.099 ± 0.000	0.224 ± 0.002	0.048 ± 0.004
d = 16	XE + ECE KDE	92.760 ± 0.037	0.097 ± 0.000	0.232 ± 0.001	0.041 ± 0.003
m = 7	XE + MMD (Ours)	92.894 ± 0.035	0.089 ± 0.000	0.174 ± 0.002	0.040 ± 0.004
ADULT	XE	84.528 ± 0.040	0.186 ± 0.000	0.320 ± 0.002	-0.014 ± 0.002
n = 32561	XE + MMCE	84.203 ± 0.042	0.191 ± 0.000	0.340 ± 0.002	-0.015 ± 0.002
d = 104	XE + ECE KDE	84.187 ± 0.045	0.178 ± 0.001	0.414 ± 0.002	-0.020 ± 0.003
m = 2	XE + MMD (Ours)	84.565 ± 0.035	0.178 ± 0.001	0.315 ± 0.002	-0.018 ± 0.001
objective for an RBF kernel over  $\mathcal{X} \times \mathcal{Y}$ . Each metric is computed on a held-out test set. We repeat all experiments across 50 random seeds and report the mean and standard error for each metric.
Results The results for regression and classification are shown in Table 3 and Table 4, respectively. In regression, our objective  $(\mathrm{NLL} + \mathrm{MMD})$  achieves better QCE than the NLL objective across all datasets while maintaining comparable (or better) NLL scores on the test set. On most datasets, we find that our method also reduces the sharpness penalty induced by post-hoc calibration. Importantly, we also observe that post-hoc calibration and our method are complementary for calibrating forecasters; post-hoc methods can guarantee calibration on held-out data, and calibration regularization mitigates the sharpness penalty. For classification, we find that our method improves accuracy, ECE, and entropy relative to the baselines we tested for calibration regularization. These trends also generally hold with post-hoc calibration, where we observe that our method achieves better calibration and accuracy across most datasets (See Table 5).
Computational Requirements Relative to training with an unregularized NLL objective, our method with MMD regularization requires an average wall clock time per epoch of  $1.2\mathrm{x}$  for the 5 regression datasets, and  $1.3\mathrm{x}$  for the 5 classification datasets.
6.3 Experiment: Decision Calibration
One of the key advantages of our trainable calibration metrics is their flexibility in enforcing multiple notions of calibration. We consider a decision-making task to study the effectiveness of our method in improving decision calibration. The decision task is to choose an action  $a \in \{\pm 1\}$ , whose loss  $\ell(a, y) = \mathbb{1}\{a \neq \mathrm{sign}(y - c)\}$  depends only on whether the label  $y$  exceeds a threshold  $c \in \mathbb{R}$ . Given a true distribution  $P$  and a forecaster  $Q_{Y|X}$ , the Decision Calibration Error (DCE) is  $\mathrm{DCE}^2(P, Q) = \sum_{a \in \mathcal{A}} \| \mathbb{E}_P[\ell(Y, a)] - \mathbb{E}_X \mathbb{E}_{\widehat{Y} \sim Q_{Y|X}}[\ell(\widehat{Y}, a)]\|_2^2$ .
MMD Objective To optimize for decision calibration, we tailor the kernels  $k_{X}, k_{Y}$  to reflect the decision problem. More concretely, we choose a kernel function that penalizes predicted labels which are on the wrong side of  $c$ , namely  $k_{Y}(y, y') = +1$  if  $\mathrm{sign}(y - c) = \mathrm{sign}(y' - c)$ , and  $-1$  otherwise.
(a) NLL
(b) Ours (NLL + MMD)
Figure 1: We visualize Local Calibration Error for a forecaster trained to predict annual wheat crop-yield based on climate and geospatial data. The standard NLL objective (Left) leads to a forecaster that is miscalibrated across local geospatial neighbourhoods, as seen by areas of brighter color. Our kernel-based calibration objective (Right) leads to better calibration across local neighborhoods.
To allow for gradient based optimization, we relax the objective to yield the differentiable kernel  $k_{Y}(y,y^{\prime}) = \tanh (y - c)\cdot \tanh (y^{\prime} - c)$ . Notice that  $k_{Y}(y,y^{\prime})\approx +1$  when  $\mathrm{sign}(y - c) = \mathrm{sign}(y - c)$ , and  $-1$  otherwise. To encourage decision calibration for all features, we let  $k_{X}$  be an RBF kernel.
Results The results in Table 3 demonstrate that the MMD training objective tailored to the decision problem achieves the best decision calibration across all datasets. Additional results in Appendix B.3 show that using an MMD kernel tailored to a decision problem (i.e. the tanh kernel) improves decision calibration scores across all datasets.
6.4 Experiment: Local Calibration
Setup To provide intuition on how our metrics can facilitate calibration in local neighbourhoods of features, we study how location affects the reliability of forecasts on the CROP-YIELD regression dataset. We tailor the kernels  $k_{X}, k_{Y}$  to local calibration for geospatial neighborhoods. Namely, we select  $\phi(x)$  which extracts the geospatial features (latitude, longitude) from the full input vector  $x$ . We then define  $k_{X}(x, x') = k_{\mathrm{RBF}}(\phi(x), \phi(x'))$  and  $k_{Y}(y, y') = k_{\mathrm{RBF}}(y, y')$ .
Metric Extending prior work on local calibration [27], given a cdf forecaster  $Q_{Y|X}$ , a feature embedding  $\phi(\cdot)$  and kernel  $k$ , we define Local Calibration Error (LCE) for regression is  $\mathrm{LCE}(x;c) = \left( \sum_{i} \mathbb{1}\{y_i \leq Q_{Y|x_i}^{-1}(c)\} \cdot k(\phi(x), \phi(x_i)) \right) / (\sum_{i} k(\phi(x), \phi(x_i))) - c$ . The total LCE is computed as  $\mathrm{LCE}_{\mathrm{total}}(x) = \frac{1}{B} \sum_{i=1}^{B} \mathrm{LCE}(x; c_i)^2$ , where  $c_i = \frac{i}{B}$ . Intuitively, LCE measures the QCE across local regions determined by  $\phi(\cdot)$ . We visualize the results in Figure 1 for  $B = 20$ .
We introduced a flexible class of trainable calibration metrics based on distribution matching using MMD. These methods can be effectively combined with existing post-hoc calibration methods to achieve strong calibration guarantees while preserving sharpness.
Limitations Optimizing the MMD metric requires that samples from the forecast are differentiable with respect to the model parameters. This restricts the family of applicable forecasters, but differentiable relaxations such as Gumbel-softmax can mitigate this challenge. Furthermore, in practice it is often not possible to achieve zero calibration error via regularization due to computational and data limitations. To mitigate this limitation, we suggest pairing our calibration regularizers with post-hoc calibration methods. Lastly, the scale of the MMD metric is sensitive to the chosen kernel (e.g., the bandwidth of a RBF kernel), so it can be difficult to understand how calibrated a model is in an absolute sense based on the MMD metric.
CM is supported by the NSF GRFP. This research was supported in part by NSF (#1651565), ARO (W911NF-21-1-0125), ONR (N00014-23-1-2159), and CZ Biohub.