SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\boldsymbol {\theta} _ {\delta_ {i}} \leq \boldsymbol {\theta} _ {\delta_ {j}}, \quad \text {i f} \delta_ {i} <   \delta_ {j} \text {a n d} \beta_ {\delta_ {i}} \geq 0,
$$
$$
\boldsymbol {\theta} _ {\delta_ {i}} \geq \boldsymbol {\theta} _ {\delta_ {j}}, \quad \text {i f} \delta_ {i} <   \delta_ {j} \text {a n d} \beta_ {\delta_ {j}} \leq 0.
$$
Fit a continuous, monotonically increasing interpolation model  $\hat{\beta}(\delta)$  and a continuous model  $\hat{\theta}(\delta)$  using the training data  $\{\delta_i, \beta_{\delta_i}, \pmb{\theta}_{\delta_i}\}_{i=1,\dots,N}$
Output:  $\hat{\beta} (\delta),\hat{\theta} (\delta)$
desired calibration, provided that we interpolate between two strictly increasing or decreasing sets of hyperparameters. Formally, this is achieved by solving (7)  $N_{\mathrm{cal}} + 1$  times to obtain  $\beta_0,\beta_{\delta_1},\ldots ,\beta_{\delta_N}\in$ $\mathbb{R}$  and  $\pmb {\theta}_0,\pmb {\theta}_{\delta_1},\dots ,\pmb{\theta}_{\delta_N}\in \Theta$  , subject to two additional constraints. First, the calibration scaling parameters  $\beta_{\delta}$  must be monotonically increasing with  $\delta$  , i.e.,
$$
\beta_ {\delta_ {i}} \leq \beta_ {\delta_ {j}} \quad \text {i f} \delta_ {i} <   \delta_ {j},
$$
and the calibration hyperparameters  $\theta_{\delta}$  must be decreasing with  $\delta$  if  $\beta_{\delta}$  is negative, and increasing if  $\beta_{\delta}$  is positive, i.e.,
In other words, the entries of  $\theta_{\delta}$  are strictly decreasing with  $\beta_{\delta}$  up until the sign of  $\beta_{\delta}$  switches, after which they are increasing. The reason why we impose these restrictions is that we can then confidently interpolate between any values of  $\beta_{\delta}$  and  $\theta_{\delta}$ , since Assumption 4.1 implies that the quantile stipulated by  $\beta_{\delta} \sigma_{\mathcal{D}_{\mathrm{tr}}}(\pmb{\theta}, \cdot)$  is monotonically increasing with  $\delta$ . We then train simple piecewise linear interpolation models  $\hat{\beta}: [0,1] \to \mathbb{R}$  and  $\hat{\pmb{\theta}}: [0,1] \to \Theta$ , such that  $\hat{\beta}(\delta_i) = \beta_{\delta_i}$  and  $\hat{\pmb{\theta}}(\delta_i) = \pmb{\theta}_{\delta_i}$ , with the additional constraint that  $\hat{\pmb{\theta}}(\delta)$  reaches a minimum whenever  $\hat{\beta}(\delta) = 0$ , which can be potentially achieved by adding an artificial vector of training hyperparameters  $\pmb{\theta}_{\delta}$  for computing  $\hat{\pmb{\theta}}(\delta)$ . Note, however, that any other form of monotone interpolation is also acceptable for obtaining  $\hat{\beta}(\delta)$  and  $\hat{\pmb{\theta}}(\delta)$ . The procedure is summarized in Algorithm 1.
Remark 5.3. While lengthscale constraints of the form  $\theta_{\delta_j} \leq \theta_{\delta_{j+1}}$  can be easily enforced when solving (7) by substituting  $\pmb{\theta}$  with  $\pmb{\theta}_{\delta_j} + \Delta \pmb{\theta}$  and minimizing over the logarithm of  $\Delta \pmb{\theta}$ , the constraint  $\beta_{\delta_j} \leq \beta_{\delta_{j+1}}$  is not enforced in (7). However, in practice we were often able to find local minima of (7) that satisfy this requirement.
We can then easily show that our approach achieves an arbitrary calibration level as the amount of data grows, provided that we choose the confidence levels  $\delta$  accordingly.
Theorem 5.4. Let  $y$  be absolutely continuous conditioned on  $\pmb{x}$ , let  $\mu_{\mathcal{D}_{tr}}(\pmb{\theta}^R, \cdot)$  be a posterior GP mean, and let  $\sigma_{\mathcal{D}_{tr}}(\cdot, \cdot)$  be a GP posterior variance conditioned on  $\mathcal{D}_{tr}$ . Then, for any calibration data set  $\mathcal{D}_{cal} = \{\pmb{x}_{cal}^i, y_{cal}^i\}$ , choose
$$
\delta_ {1} = \frac {1}{N _ {c a l} + 1}, \quad \delta_ {2} = \frac {2}{N _ {c a l} + 1}, \dots , \quad \delta_ {N _ {c a l}} = \frac {N _ {c a l}}{N _ {c a l} + 1},
$$
and let  $\hat{\beta} (\cdot)$  and  $\hat{\pmb{\theta}} (\cdot)$  be interpolation models obtained with Algorithm 1 and confidence levels  $\delta_{1},\ldots ,\delta_{N_{cal}}$  . Then
$$
\mathbb {P} _ {\boldsymbol {x}, y \sim \Pi} \Big (y - \mu_ {\mathcal {D} _ {t r}} (\hat {\boldsymbol {\theta}} (\delta), \boldsymbol {x}) \leq \hat {\beta} (\delta) \sigma_ {\mathcal {D} _ {t r}} (\hat {\boldsymbol {\theta}} (\delta), \boldsymbol {x}) \Big) \in \left[ \delta - \frac {1}{N _ {c a l} + 1}, \delta + \frac {1}{N _ {c a l} + 1} \right].
$$
Proof. The proof can be found in the supplementary material.
Note that Theorem 5.4 also implies that a single set of calibration parameters  $\beta_{\delta}$  and  $\pmb{\theta}_{\delta}$  obtained by solving (7), since we can substitute  $\hat{\beta} (\delta) = \beta_{\delta}$  and  $\hat{\pmb{\theta}} (\delta) = \pmb{\theta}_{\delta}$  into (8).
(a) Our approach.
(b) Kuleshov et al. (2018).
(c) Vovk et al. (2020).
(d) Mean  $\pm$  constant (Marx et al., 2022).
(e) Neural network-based recalibrator (Kuleshov & Deshpande, 2022).
(f) Base model (vanilla GP).
Figure 2: Centered  $99\%$  confidence intervals (shaded regions) obtained with our method, vanilla GPs recalibrated using the approaches of Kuleshov et al. (2018) and Vovk et al. (2020), a naive vanilla GP, the point-predictor (variance-free) approach proposed in Marx et al. (2022), and a naive fully Bayesian GP. Solid lines represent the predictive mean, crosses represent data used to train the base model, and squares represent calibration (holdout) data. Our approach yields a model that is both well-calibrated and sharp. This is because of the added flexibility that comes from being able to also change the lengthscale to design a calibrated model.
Computational Complexity. Much like hyperparameter optimization for standard GPs (Rasmussen & Williams, 2006), the major driver behind the computational complexity in our approach stems from the need to invert the covariance matrix, an operation that scales cubically with the amount of data. In order to alleviate the computational cost of our approach, we can resort to different tools that improve scalability (Liu et al., 2020). One approach consists of employing only a subset of the training data  $\mathcal{D}_{\mathrm{tr}}$  to choose the calibration hyperparameters  $\theta_{\delta}$ , and then the full data set to choose  $\beta_{\delta}$ . While this potentially leads to a loss in sharpness compared to when using the full data set, it still guarantees a calibrated model. Our technique is also readily applicable to sparse GPs (Snelson & Ghahramani, 2005; Titsias, 2009), as Assumption 4.1 typically still holds. This option is also explored in numerical experiments, in Section 7. Moreover, in many settings a specific level of calibration is often required, as opposed to several different ones, e.g., in stochastic model predictive control, where chance constraints corresponding to a fixed risk have to be satisfied (Mesbah, 2016). In such settings, we potentially only have to train a single vector of calibration hyperparameters  $\theta_{\delta}$  which reduces computational cost.
Initialization and Solution. The choice of initial hyperparameters can affect the optimization results considerably, and choosing a good hyperparameter initialization can be challenging, as is true when choosing the hyperparameters for the predictive mean. While this can be partially addressed by employing random restarts, we can also reuse trained models for similar calibration levels, since it is reasonable to expect that only small changes to the calibration hyperparameters are required to achieve a slight increase or decrease in confidence level. Furthermore, we can also simplify the problem by considering only a scaled version of the regression hyperparameters  $\theta^R$  to compute  $\theta_{\delta}$  which would reduce the optimization problem to a line search.
In this section, we apply and analyze our approach using a toy data set and different regression benchmark data sets from the UCI repository. In the supplementary material, we also compare our
approach to that of Capone et al. (2022) when used to obtain uniform error bounds and apply our method to two different Bayesian optimization problems.
The goal is for our approach to obtain a sharp calibrated regression model for each data set in the calibrated regression experiments. We test our approach on various data sets and compare it to the state-of-the-art recalibration approaches by Kuleshov et al. (2018) and Vovk et al. (2020), the point predictor (posterior variance-free) approach proposed in Marx et al. (2022), as well as the check-score-based approach of Kuleshov & Deshpande (2022). The technique proposed by Kuleshov et al. (2018) essentially multiplies the vanilla posterior standard deviation  $\sigma_{\mathcal{D}_{\mathrm{tr}}}(\pmb{\theta}^R, \cdot)$  with the recalibrated  $z$ -score, such that the confidence level observed on the calibration data matches that of the desired confidence level. Vovk et al. (2020) employ a similar approach, except that random interpolation is employed to compute new scaling values. The point predictor-based method proposed in Marx et al. (2022) discards the posterior standard deviation  $\sigma_{\mathcal{D}_{\mathrm{tr}}}(\pmb{\theta}^R, \cdot)$  and computes a constant scalar that is added to the predictive mean and used to compute quantiles everywhere within the input space. The method of Kuleshov & Deshpande (2022) trains a neural network using a quantile loss, which takes base quantiles as inputs and returns new, recalibrated quantiles. As a recalibrator for the method of Kuleshov & Deshpande (2022), we employ the same neural network architecture suggested in their paper, trained over 200 epochs, with additional pretraining over 2000 epochs using a single dataset for the UCI experiments. In all experiments except kin8nm and Facebook comment volume 2, we employ standard GPs with automatic relevance determination squared-exponential (ARD-SE) kernels and zero prior mean as base models, trained using log-likelihood maximization. For the kin8nm and Facebook comment volume 2 datasets, we employ sparse GPs (Titsias, 2009) with zero prior mean, ARD-SE kernels, and 300 inducing points.
7.1 Toy Data Set
The first regression data set corresponds to a one-dimensional synthetic data set, where the results can be easily displayed visually. The main purpose of this section is to give an intuition as to how our approach computes confidence intervals compared to other techniques. We investigate the performance of our approach and compare it to other methods when employed to compute centered  $99\%$  confidence intervals. We observe that the confidence intervals obtained with our approach peak less strongly far away from the data while being tight near the data compared to all other approaches except the one of Kuleshov & Deshpande (2022). This is because we allow the lengthscale to change to obtain a calibrated model. In contrast, all other methods except that of Kuleshov & Deshpande (2022) scale the standard GP posterior variance without changing hyperparameters. The method of Kuleshov & Deshpande (2022), which uses a neural network as a recalibrator, offers additional flexibility, resulting in sharper confidence intervals. However, calibration is not explicitly enforced during training, resulting in poor calibration. The results are depicted in Figure 2.
7.2 Benchmark Data Sets
We now experiment with seven different regression data sets from the UCI repository, two containing over eight thousand data points and requiring sparse GP approximations. The training/calibration/test split is 0.6, 0.2, and 0.2 for all data sets except the Facebook comment volume 2 data set, which contains over 80000 data points, and where the split is 0.08, 0.02, and 0.9. For the approach of Kuleshov & Deshpande (2022), we follow the steps in their paper and limit the calibration data size to 500.
We assess performance by employing diagnostic tools commonly used to assess calibration and sharpness (Kuleshov et al., 2018; Marx et al., 2022; Gneiting et al., 2007)). The score used to quantify calibration is the calibration error (Kuleshov et al., 2018), given by
$$
\left. \operatorname {c a l} \left(\mu_ {\mathcal {D} _ {\mathrm {t r}}} \left(\boldsymbol {\theta} ^ {R}, \cdot\right), \beta , \sigma_ {\mathcal {D} _ {\mathrm {t r}}} (\cdot , \cdot)\right) = \sum_ {j = 1} ^ {m} \left(p _ {j} - \hat {p} _ {j}\right) ^ {2}, \right. \tag {8}
$$
where  $p_j$  corresponds to the  $j$ -th desired confidence level, chosen, e.g., evenly spaced between 0 and 1, and  $\hat{p}_j$  is the observed confidence level, i.e.,
$$
\hat {p} _ {j} = \frac {\left| \left\{y _ {t} ^ {*} \mid \Delta y _ {t} ^ {*} \leq \beta_ {p _ {j}} \sigma_ {\mathcal {D} _ {\mathrm {u}}} \left(\boldsymbol {\theta} _ {p _ {j}} \boldsymbol {x} _ {t} ^ {*}\right) , t = 1 , \dots , T \right\} \right|}{T}. \tag {9}
$$
Table 1: Expected calibration error and sharpness of different methods over 100 repetitions per experiment. We report the expected calibration error (ECE), the average predictive standard deviation (STD), negative log-likelihood (NLL) and  $95\%$  confidence interval width  $(95\% \mathrm{CI})$  obtained with our approach, vanilla GPs recalibrated using the methods of Kuleshov et al. (2018) (RK) and Vovk et al. (2020) (RV), the variance-free approach proposed in Marx et al. (2022) (RM), and the neural network-based recalibrator of Kuleshov & Deshpande (2022) (NN). We additionally report performance for the base model (B), which corresponds to a vanilla GP without the holdout data. Lower is better for all metrics. In all experiments except the Facebook2 dataset, our method is sharpest compared to all other methods except that of Kuleshov & Deshpande (2022). However, Kuleshov & Deshpande (2022) performs more poorly in terms of expected calibration error.
DATA SET	METRIC	OURS	RK	RV	RM	NN	B
BOSTON	ECE	0.003	0.0029	0.0029	0.0029	0.0056	0.041
STD	0.16	0.31	0.3	0.33	0.22	1.9
NLL	0.21	0.39	0.4	0.42	-0.24	1.6
95% CI	0.76	1.4	1.4	1.4	0.73	7.4
Yacht	ECE	0.0044	0.0043	0.0044	0.0043	0.0081	0.039
STD	0.16	0.5	0.47	0.5	0.14	2.8
NLL	0.26	0.68	0.69	0.68	-2	2
95% CI	0.76	2.3	2.3	2.3	0.3	11
MPG	ECE	0.0036	0.0035	0.0035	0.0035	0.0053	0.044
STD	0.13	0.38	0.38	0.38	0.29	2.8
NLL	0.032	0.63	0.64	0.63	0.02	2
95% CI	0.6	1.7	1.8	1.7	0.96	11
WINE	ECE	0.00047	0.00047	0.00047	0.00047	0.0067	0.0058
STD	0.54	1	1	0.88	0.72	1.4
NLL	1.2	1.3	1.3	1.3	-0.36	1.4
95% CI	2.1	3.8	3.8	3.9	2.8	5.4
CONCRETE	ECE	0.00071	0.00064	0.00064	0.00064	0.00076	0.032
STD	0.25	0.64	0.64	0.64	0.57	2.8
NLL	0.72	1.1	1.1	1.1	0.85	2
95% CI	0.93	2.5	2.5	2.5	2.1	11
KIN8NM	ECE	0.00016	0.00016	0.00016	0.00016	0.00053	0.028
STD	0.074	0.12	0.12	0.12	0.098	0.4
NLL	-0.54	-0.65	-0.65	-0.63	-0.76	0.1
95% CI	0.26	0.47	0.47	0.48	0.44	1.6
FACEBOOK2	ECE	0.00044	0.00043	0.00043	0.00045	0.0089	0.044
STD	0.068	0.18	0.18	0.18	0.18	1.2
NLL	3.6	-1.3	-1.3	-1.2	-2.3	1.2
95% CI	0.6	1.7	1.7	1.7	3.4	4.6
Here the superscript  $*$  denotes test inputs and outputs,  $T$  denotes the total number of test points, and  $\Delta y_{t}^{*} \coloneqq \mu_{\mathcal{D}_{\mathrm{ir}}}(\pmb{\theta}^{R}, \pmb{x}_{t}^{*}) - y_{t}^{*}$ . We employ  $m = 21$  evenly spaced values between 0 and 1 for  $p_j$ . To measure sharpness, we employ the average length of the  $95\%$  confidence interval, the average standard deviation of the predictive distribution, and the average negative log-likelihood of the predictions (Gneiting et al., 2007; Marx et al., 2022). Note that since every model outputs a quantile for any desired calibration level, the corresponding negative log-likelihood and average standard deviation are well specified. These are computed by employing the cumulative distribution function, obtained by inverting the quantile function specified by each model.
We carried out each experiment 100 times and report the resulting average expected calibration error, standard deviation, negative log-likelihood, and length of the centered  $95\%$  confidence intervals in Table 1. Our approach performs best or marginally worse than all other calibration approaches regarding expected calibration error. This is to be expected from Theorem 5.4. Furthermore, it outperforms all approaches except that of Kuleshov & Deshpande (2022) in sharpness. However, the improved sharpness of the method of Kuleshov & Deshpande (2022) comes at the expense of calibration.
We have presented a calibration method for Gaussian process regression that leverages the monotonicity properties of the kernel hyperparameters to obtain sharp calibrated models. We show that, under reasonable assumptions, our method yields an accurately calibrated model as the size of data used for calibration increases. When applied to different regression benchmark data sets, our approach was shown to be competitive in sharpness compared to state-of-the-art recalibration methods without sacrificing calibration performance. It is worth stressing that, though the tools presented here emerge naturally from a Gaussian process setting, we do not require our predictor to be a Gaussian process to obtain theoretical guarantees. In future work, we aim to leverage similar monotonicity characteristics to get sharply calibrated models using tools different from Gaussian processes. Furthermore, we aim to experiment with inducing variables as hyperparameters when optimizing the models for sharpness.
This work was supported in part by the European Research Council Consolidator Grant Safe data-driven control for human-centric systems (CO-MAN) under grant agreement number 864686.
Berkenkamp, F., Schoellig, A. P., and Krause, A. No-regret bayesian optimization with unknown hyperparameters. Journal of Machine Learning Research, 20(50):1-24, 2019. URL http://jmlr.org/papers/v20/18-213.html.
Capone, A., Lederer, A., and Hirche, S. Gaussian process uniform error bounds with unknown hyperparameters for safety-critical applications. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 2609-2624. PMLR, 2022. URL https://proceedings.mlr.press/v162/capone22a.html.
Chowdhury, S. R. and Gopalan, A. On kernelized multi-armed bandits. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 844-853, International Convention Centre, Sydney, Australia, 2017. PMLR. URL http://proceedings.mlr.press/v70/chowdhury17a.html.
Deisenroth, M. P., Fox, D., and Rasmussen, C. E. Gaussian processes for data-efficient learning in robotics and control. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(2): 408-423, 2015.
Deringer, V. L., Bartók, A. P., Bernstein, N., Wilkins, D. M., Ceriotti, M., and Csányi, G. Gaussian process regression for materials and molecules. Chemical Reviews, 121(16):10073-10141, 2021. doi: 10.1021/acs.chemrev.1c00022. URL https://doi.org/10.1021/acs.chemrev.1c00022. PMID: 34398616.
Fong, E. and Holmes, C. C. On the marginal likelihood and cross-validation. Biometrika, 107(2):489-496, 2020. doi: 10.1093/biomet/asz077. URL https://doi.org/10.1093/biomet/asz077.
Gal, Y., Hron, J., and Kendall, A. Concrete dropout. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/84ddfb34126fc3a48ee38d7044e87276-Paper.pdf.
Gneiting, T., Balabdaoui, F., and Raftery, A. E. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69(2):243-268, 2007.
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1321-1330. PMLR, 2017. URL https://proceedings.mlr.press/v70/guo17a.html.
Kocijan, J. Modelling and control of dynamic systems using Gaussian process models. Springer, 2016.
Vovk, V., Petej, I., Toccaceli, P., Gammerman, A., Ahlberg, E., and Carlsson, L. Conformal calibrators. In Proceedings of the Ninth Symposium on Conformal and Probabilistic Prediction and Applications, volume 128 of Proceedings of Machine Learning Research, pp. 84-99. PMLR, 2020. URL https://proceedings.mlr.press/v128/vovk20a.html.
Wendland, H. Scattered Data Approximation. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, 2004. doi: 10.1017/CBO9780511617539.