SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alexandre Capone
alexandre.capone@tum.de
hirc@cit.tum.de
Geoff Pleiss
geoff.pleiss@stat.ubc.ca
While Gaussian processes are a mainstay for various engineering and scientific applications, the uncertainty estimates don't satisfy frequentist guarantees and can be miscalibrated in practice. State-of-the-art approaches for designing calibrated models rely on inflating the Gaussian process posterior variance, which yields confidence intervals that are potentially too coarse. To remedy this, we present a calibration approach that generates predictive quantiles using a computation inspired by the vanilla Gaussian process posterior variance but using a different set of hyperparameters chosen to satisfy an empirical calibration constraint. This results in a calibration approach that is considerably more flexible than existing approaches, which we optimize to yield tight predictive quantiles. Our approach is shown to yield a calibrated model under reasonable assumptions. Furthermore, it outperforms existing approaches in sharpness when employed for calibrated regression.
Gaussian process (GP) regression offers an ambitious proposition: by conditioning a model on measurement data, we are provided with a Gaussian probability distribution for the unseen data. Assuming that the posterior probability distribution holds, we can then directly calibrate our model using the inverse error function. Though the distribution of unseen data seldom follows the Gaussian prior distribution, and the GP generally does not adapt adequately to the observed distributions after being conditioned on the data, GPs have become one of the most powerful and established regression techniques. Besides having found widespread use in machine learning (Deisenroth et al., 2015; Srinivas et al., 2012), their good generalization properties have motivated applications in the fields of control (Kocijan, 2016), astrophysics (Roberts et al., 2013) and chemistry (Deringer et al., 2021), to name a few. Furthermore, the Bayesian paradigm offers a powerful tool to analyze the theoretical properties of different regression techniques (Srinivas et al., 2012; Capone et al., 2022).
In this paper, we present a novel approach to obtaining sharp calibrated Gaussian processes, i.e., Gaussian processes that provide concentrated predictive distributions that accurately match the observed data. Instead of computing confidence intervals by inflating the Gaussian process posterior variance, our approach discards it and computes a new quantity inspired by the computation of the posterior variance, where all hyperparameters are chosen in a way that results in both accurate and sharp calibration. In other words, we train two separate Gaussian processes: one for the predictive mean and one for obtaining predictive quantiles, which is exclusively used for calibration purposes. By doing so, we reach considerably more flexibility than existing calibration approaches, which enables us to additionally optimize the sharpness of the calibration. Our approach outperforms several
state-of-the-art calibration approaches in terms of sharpness while still yielding similar calibration performance. Furthermore, it is competitive compared to a neural network-based method in sharpness without sacrificing calibration performance.
Notation. We use  $\mathbb{R}_+$  to denote the non-negative real numbers. Boldface lowercase/uppercase characters denote vectors/matrices. For two vectors  $\pmb{a}$  and  $\pmb{a}'$  in  $\mathbb{R}^d$ , we employ the notation  $\pmb{a} \leq \pmb{a}'$  to denote componentwise inequality, i.e.,  $a_i \leq a_i'$ ,  $i = 1, \dots, d$ . For a square matrix  $\pmb{K}$ , we use  $|\pmb{K}|$  to denote its determinant, and  $[\pmb{K}]_{ij}$  to denote the entry corresponding to the  $i$ -th row and  $j$ -th column.
Calibration of Classification Models. There has been extensive work on obtaining calibrated models in the domain of classification. While there are many methods that do not employ post-processing, we only focus here on methods that employ some form of post-processing. Most forms of post-processing-based calibration for classification fall into the category of conformal methods (Vovk et al., 2005), which, given an input, aim to produce sets of labels that contain the true label with a pre-specified probability. Arguably the two most common forms of calibration are isotonic regression (Niculescu-Mizil & Caruana, 2005) and Platt scaling (Platt et al., 1999). In Niculescu-Mizil & Caruana (2005), Platt scaling and isotonic regression are analyzed extensively for different types of predictive models. In Guo et al. (2017), a modified form of Platt scaling for modern classification neural networks is proposed.
Calibration of Regression Models. Though initially developed for classification, conformal calibration has been extended to regression settings. In Lakshminarayanan et al. (2017), a calibration approach was proposed for deep ensembles. Gal et al. (2017) propose a dropout-based technique for calibrating deep neural networks. However, these approaches require changing the regressor, potentially deteriorating its predictive performance. It should be noted that Bayesian neural networks (MacKay, 1995), while being able to provide credible sets for the output, fully trust the posterior, resulting in a naively calibrated model that seldom reflects the data's distribution. As a remedy for this, Kuleshov et al. (2018) present a recalibration approach that scales a model's predictive quantiles to satisfy the observed data's distribution. In Vovk et al. (2020), a similar approach is presented, where interpolation between scaling factors is randomized, and a theoretical analysis is provided. An extension of both Kuleshov et al. (2018) and Vovk et al. (2020) and other recalibration is proposed in Marx et al. (2022), along with corresponding theoretical guarantees. While these methods have been shown to yield well-calibrated models, the resulting predictive quantiles are potentially much too crude, resulting in predictions that perform poorly in terms of sharpness, i.e., the corresponding confidence intervals will overestimate the model error by a very large margin. To remedy this, Song et al. (2019) and Kuleshov & Deshpande (2022) propose optimizing the parameters of a recalibration model by obtaining calibration on a distribution level. However, while Song et al. (2019) relies on complex approximations and provides no theoretical guarantees, Kuleshov & Deshpande (2022) does not allow to optimize for sharpness directly, and calibration is only guaranteed asymptotically as the number of data grows.
Consider a compact input space  $\mathcal{X} \subset \mathbb{R}^d$ , and output space  $\mathcal{V} \subset \mathbb{R}$ , and an unknown data distribution  $\Pi$  on  $\mathcal{X} \times \mathcal{V}$ . Consider a model conditioned on training data  $\mathcal{D}_{\mathrm{tr}} \sim \Pi$  that, for every  $\pmb{x} \in \mathcal{X}$  and confidence level  $\delta \in [0,1]$ , returns a base prediction  $\mu_{\mathcal{D}_{\mathrm{tr}}}(\pmb{x})$  and an additive cut-point term  $\beta_{\delta} \sigma_{\mathcal{D}_{\mathrm{tr}}}(\delta, \pmb{x})$  where  $\sigma_{\mathcal{D}_{\mathrm{tr}}}(\delta, \pmb{x}) \geq 0$  and  $\beta_{\delta} \in \mathbb{R}$  is potentially negative, such that  $\mu_{\mathcal{D}_{\mathrm{tr}}}(\pmb{x}) + \beta_{\delta} \sigma_{\mathcal{D}_{\mathrm{tr}}}(\delta, \pmb{x})$  corresponds to a predictive  $\delta$ -quantile. The model is then said to be calibrated if
$$
\mathbb {P} _ {\boldsymbol {x}, y \sim \Pi} \left(y - \mu_ {\mathcal {D} _ {\mathrm {t r}}} (\boldsymbol {x}) \leq \beta_ {\delta} \sigma_ {\mathcal {D}} (\delta , \boldsymbol {x})\right) = \delta \tag {1}
$$
holds for every  $\delta \in [0,1]$ . Furthermore, the calibrated model is also said to be sharp if the corresponding predictive distributions are concentrated (Gneiting et al., 2007), i.e., if the centered confidence intervals induced by the predictive quantiles
$$
\left| \beta_ {\delta} \sigma_ {\mathcal {D} _ {\mathrm {t r}}} (\delta , \boldsymbol {x}) - \beta_ {1 - \delta} \sigma_ {\mathcal {D} _ {\mathrm {t r}}} (1 - \delta , \boldsymbol {x}) \right| \tag {2}
$$
are as small as possible for every  $\delta \in [0,1]$ . Our goal is to find a sharply calibrated model based on GP regression.
4 Gaussian Process Regression
In this section, we briefly review GP regression, with particular focus on the choice and influence of hyperparameters.
A GP is formally defined as a collection of random variables, any finite subset of which is jointly Gaussian (Rasmussen & Williams, 2006). It is fully specified by a prior mean function, which we set to zero without loss of generality, and a hyperparameter-dependent covariance function, called kernel  $k: \Theta \times \mathcal{X} \times \mathcal{X} \to \mathbb{R}$ , where  $\Theta$  denotes the hyperparameter space. The core concept behind GP regression lies in assuming that any finite number of measurements of an unknown function  $f: \mathcal{X} \to \mathbb{R}$  at arbitrary inputs  $x_{1}, \ldots, x_{N} \in \mathcal{X}$  are jointly Gaussian with mean zero and covariance  $K(\theta)$ , where
$$
[ \boldsymbol {K} (\boldsymbol {\theta}) ] _ {i j} = k (\boldsymbol {\theta}, \boldsymbol {x} _ {i}, \boldsymbol {x} _ {j})
$$
consists of kernel evaluations at the test inputs.
Given a set of noisy observations  $\mathcal{D} = \{\pmb{x}_i, y_i\}_{i=1}^N$ , where  $y_i := f(\pmb{x}_i) + \varepsilon_i$  and  $\varepsilon_i \sim \mathcal{N}(0, \sigma_0^2)$  is iid Gaussian measurement noise, we can condition the GP on them to obtain the posterior distribution. The posterior distribution for a new observation  $y^*$  at an arbitrary input  $\pmb{x}^*$  is again Gaussian distributed, with mean and variance
$$
\mu_ {\mathcal {D} _ {\mathrm {t r}}} (\boldsymbol {\theta}, \boldsymbol {x} ^ {*}) = \boldsymbol {k} (\boldsymbol {\theta}) \left(\boldsymbol {K} (\boldsymbol {\theta}) + \sigma_ {0} ^ {2} \boldsymbol {I}\right) ^ {- 1} \boldsymbol {y}, \tag {3a}
$$
$$
\sigma_ {\mathcal {D} _ {\mathrm {t r}}} ^ {2} (\boldsymbol {\theta}, \boldsymbol {x} ^ {*}) = k (\boldsymbol {\theta}, \boldsymbol {x} ^ {*}, \boldsymbol {x} ^ {*}) - k (\boldsymbol {\theta}) \left(\boldsymbol {K} (\boldsymbol {\theta}) + \sigma_ {0} ^ {2} \boldsymbol {I}\right) ^ {- 1} \boldsymbol {k} (\boldsymbol {\theta}) + \sigma_ {0} ^ {2}, \tag {3b}
$$
with  $\pmb{k}(\pmb{\theta}) \coloneqq (k(\pmb{\theta},\pmb{x}^{*},\pmb{x}_{1}),\dots,k(\pmb{\theta},\pmb{x}^{*},\pmb{x}_{N}))$ ,  $\pmb{y} = (y_{1},\dots,y_{N})$ , and  $\pmb{I}$  denoting the identity matrix. In this paper, we restrict ourselves to kernels  $k(\cdot ,\cdot ,\cdot)$  that yield a posterior variance that is monotonically increasing with respect to the hyperparameters  $\pmb{\theta}$ , as specified in the following assumption.
Assumption 4.1. The posterior variance  $\sigma_{\mathcal{D}_{\mathrm{tr}}}^2 (\pmb {\theta},\pmb{x}^*)$  is a continuous function of  $\pmb{\theta}$ . Furthermore, for all hyperparameters  $\pmb {\theta},\pmb{\theta}^{\prime}\in \Theta$  with  $\pmb {\theta}\leq \pmb{\theta}^{\prime}$ , it holds that  $\sigma_{\mathcal{D}_{\mathrm{tr}}}^2 (\pmb {\theta},\pmb{x}^*)\leq \sigma_{\mathcal{D}_{\mathrm{tr}}}^2 (\pmb {\theta}^\prime ,\pmb{x}^*)$ .
Assumption 4.1 holds trivially for the signal variance of a kernel. Moreover, it holds for any hyperparameters that lead to a monotonous increase in the Fourier transform, which is the case, e.g., for the inverse lengthschale of stationary kernels up to a multiplicative factor corresponding to the ratio of the lengthscales (Capone et al., 2022). Furthermore, several results that employ the so-called fill-distance indicate that Assumption 4.1 holds for the inverse lengthscale of a broad class of stationary kernels as the lengthscale becomes very large or very small (Wendland, 2004). In our experiments, we observed that Assumption 4.1 was never violated for the inverse lengthscale of the squared-exponential kernel. Assumption 4.1 will be leveraged to define a cumulative density function by also changing the hyperparameters corresponding to the posterior covariance, as opposed to simply scaling it.
Arguably one of the most challenging aspects of GP regression lies in the choice of hyperparameters  $\theta$ , as they ultimately determine various characteristics of the posterior, e.g., smoothness and amplitude. In practice, the most common way of choosing  $\theta$  is by maximizing the log marginal likelihood
$$
\log p (\boldsymbol {y} \mid \boldsymbol {X}, \boldsymbol {\theta}) = - \frac {1}{2} \log | \boldsymbol {K} (\boldsymbol {\theta}) + \sigma_ {0} ^ {2} \boldsymbol {I} | - \frac {N}{2} \log (2 \pi) - \frac {1}{2} \boldsymbol {y} ^ {\top} \left(\boldsymbol {K} (\boldsymbol {\theta}) + \sigma_ {0} ^ {2} \boldsymbol {I}\right) ^ {- 1} \boldsymbol {y}. \tag {4}
$$
In terms of posterior mean quality, i.e., predictive performance of  $\mu_{\mathcal{D}_{\mathrm{tr}}}(\pmb{\theta},\pmb{x}^*)$ , choosing the hyperparameters in this manner is often the most promising option, since it seeks a trade-off between model complexity and data fit, and has repeatedly been shown to yield a satisfactory mean square error when applied to test data (Rasmussen & Williams, 2006). However, even when employing log-likelihood maximization, the posterior GP distribution is seldom well calibrated, and, in practice, the data often has a significantly different distribution. As a result, quantiles obtained with a purely Bayesian approach are either too confident or too conservative in practice (Capone et al., 2022; Fong & Holmes, 2020). Furthermore, the restrictions imposed by the GP prior often produce a posterior variance  $\sigma^2 (\pmb {\theta},\pmb{x}^*)$  that is grossly conservative. See Figure 1 for an illustration.
(a) Bayesian approach.
(b) Our approach.
Figure 1: Confidence interval of  $95\%$  (shaded regions) obtained with purely Bayesian approach (a), where the inverse error function is employed to compute  $\beta_{\delta}$ , and our approach (b). Solid lines represent the predictive mean. The confidence interval obtained with the Bayesian approach is not only grossly overconfident (contains less than  $80\%$  of the total data) but also partially extremely loose, exhibiting unnecessarily large confidence intervals far away from the data. By contrast, our approach is both accurate and tight.
In this section, we present our approach to obtaining calibrated GPs by discarding the posterior variance and obtaining alternative predictive quantiles using a quantity inspired by the posterior variance and new hyperparameters. As we will demonstrate, our approach has numerous advantages. First, by varying the hyperparameters used to obtain predictive quantiles, the resulting quantiles are sharper than what can be obtained simply by multiplying the posterior variance with an appropriate constant. Secondly, by exploiting the monotonicity of the hyperparameters, our approach can be used to obtain intervals for multiple confidence levels  $\delta$  in a very efficient manner. Finally, our method is backed by tight theoretical guarantees, obtained by exploiting its connection to conformal prediction.
5.1 Sharp Calibrated GP for Single Confidence Level  $\delta$
In the following, we describe how to obtain a sharply calibrated GP for a fixed desired calibration level  $\delta$ . Instead of scaling the GP posterior variance to meet the desired calibration level  $\delta$ , we propose computing predictive quantiles by training a new quantity similar to the posterior variance but with new hyperparameters. In other words, we discard the posterior variance of the first GP and replace it by a quantity that corresponds to the posterior variance of a different GP, which we train separately. This way, we allow for more degrees of freedom during calibration. We then leverage this additional freedom to minimize the distance of the quantiles to the predictive mean, which yields a sharply calibrated model.
We assume to have a data set  $\mathcal{D}$ , which we split into training data  $\mathcal{D}_{\mathrm{tr}}$  and calibration (holdout) data  $\mathcal{D}_{\mathrm{cal}}$ , with  $\mathcal{D} = \mathcal{D}_{\mathrm{tr}} \cup \mathcal{D}_{\mathrm{cal}}$ . The training data  $\mathcal{D}_{\mathrm{tr}}$  will be used to compute the posterior, whereas  $\mathcal{D}_{\mathrm{cal}}$  will be used to calibrate the model. Note that while not splitting the data might be reasonable when using other types of regression models, e.g., Bayesian or ensemble neural networks, splitting the data in the case of GPs is beneficial for providing accurate quantiles for data out of distribution. This is because, for many commonly used kernels, the GP posterior distribution is considerably more concentrated for test points close to the training data  $\mathcal{D}_{\mathrm{tr}}$  than for those far away from  $\mathcal{D}_{\mathrm{tr}}$ . Since we wish to obtain a model that is calibrated for data both close and far away from the training data, we take this into account during training by splitting the data. We also assume to have a predictive mean  $\mu_{\mathcal{D}_{\mathrm{tr}}}(\pmb{\theta}^R, \cdot)$ , corresponding to GP posterior mean function, where the regressor hyperparameters  $\pmb{\theta}^R$  were obtained, e.g., via log-likelihood maximization, as discussed in Section 4. However, note that any other way of choosing the posterior mean hyperparameters  $\pmb{\theta}^R$  is permitted.
Given  $\mu_{\mathcal{D}_{\mathrm{tr}}}(\pmb{\theta}^R, \cdot)$  and  $\mathcal{D} = \mathcal{D}_{\mathrm{tr}} \cup \mathcal{D}_{\mathrm{cal}}$ , we then follow the convention of other recalibration approaches (Kuleshov et al., 2018; Marx et al., 2022) and aim to obtain, for an arbitrary  $0 \leq \delta \leq 1$ , a scalar  $\beta_{\delta}$  and vector of hyperparameters  $\pmb{\theta}_{\delta}$ , such that the corresponding predictive quantile  $\mu_{\mathcal{D}_{\mathrm{tr}}}(\pmb{\theta}^R, \cdot)$
$\times$  Training data
$\spadesuit$  Calibration (holdout) data
$\beta_{\delta}\sigma_{\mathcal{D}_{\mathrm{tr}}}(\pmb{\theta}_{\delta},\cdot)$  contains  $\delta$  times the total amount of data points. For this reason, we henceforth refer to  $\pmb{\theta}_{\delta}$  as calibration hyperparameters.
In order to obtain a sharply calibrated GP model, ideally we would like to choose  $\beta_{\delta}$  and  $\theta_{\delta}$  such that they minimize the expected length of the centered intervals (2) subject to calibration. However, this optimization problem is hard to solve. Hence, we instead attempt to improve model sharpness by concentrating the predictive distribution around the predictive mean  $\mu_{\mathcal{D}_{\mathrm{tr}}}(\pmb{\theta}^R,\cdot)$ , i.e., by minimizing the deviation of the quantiles from  $\mu_{\mathcal{D}_{\mathrm{tr}}}(\pmb{\theta}^R,\cdot)$ . This corresponds to solving the optimization problem
$$
\min  _ {\boldsymbol {\theta} _ {\delta} \in \boldsymbol {\Theta}} \sum_ {i = 1} ^ {N _ {\mathrm {c a l}}} \beta_ {\delta} ^ {2} \sigma_ {\mathcal {D} _ {\mathrm {t r}}} ^ {2} \left(\boldsymbol {\theta} _ {\delta}, \boldsymbol {x} _ {\mathrm {c a l}} ^ {i}\right) \tag {5}
$$
$$
\text {s . t .} \sum_ {i = 1} ^ {N _ {\mathrm {c a l}}} \frac {\mathbb {I} _ {\geq 0} \left(\Delta y _ {\mathrm {c a l}} ^ {i} - \beta_ {\delta} \sigma_ {\mathcal {D} _ {\mathrm {t r}}} \left(\boldsymbol {\theta} _ {\delta} , \boldsymbol {x} _ {\mathrm {c a l}} ^ {i}\right)\right)}{N _ {\mathrm {c a l}} + 1} = \delta
$$
where  $\{\pmb{x}_{\mathrm{cal}}^i,y_{\mathrm{cal}}^i\} \in \mathcal{D}_{\mathrm{cal}}$  are samples from the calibration data set,  $\Delta y_{\mathrm{cal}}^{i}\coloneqq y_{\mathrm{cal}}^{i} - \mu_{\mathcal{D}_{\mathrm{tr}}}(\pmb{\theta}^{R},\pmb{x}_{\mathrm{cal}}^{i})$  corresponds to the difference between predicted and measured output, and  $N_{\mathrm{cal}} = |\mathcal{D}_{\mathrm{cal}}|$  to the number of data points used for calibration.
Remark 5.1. Note that we employ  $N_{\mathrm{cal}} + 1$  in the denominator in (5) instead of  $N_{\mathrm{cal}}$ . Though this choice makes little difference in practice, we require it for theoretical guarantees.
The equality constraint in (5) is generally infeasible, e.g., if  $N_{\mathrm{cal}} = 2$  and  $\delta = 0.5$ , and is discontinuous, making it hard to solve. As it turns out, this can be easily remedied without any detriment to sharpness or calibration, and the problem can be rendered considerably easier to solve by substituting the equality constraint in (5) with
$$
\beta_ {\delta} = q _ {\text {l i n}} \left(\delta , \boldsymbol {\Sigma} _ {\mathcal {D} _ {\mathrm {t r}}} ^ {- 1} \boldsymbol {\Delta} \boldsymbol {y} _ {\text {c a l}}\right), \tag {6}
$$
where  $q_{\mathrm{lin}}(\delta, \pmb{\Sigma}_{\mathcal{D}_{\mathrm{tr}}}^{-1}\pmb{\Delta}\pmb{y}_{\mathrm{cal}})$  is a monotonically increasing piecewise linear function<sup>1</sup> that maps  $\delta = j / (N_{\mathrm{cal}} + 1)$  to the  $j$ -th smallest entry of  $\pmb{\Sigma}_{\mathcal{D}_{\mathrm{tr}}}^{-1}\pmb{\Delta}\pmb{y}_{\mathrm{cal}}$ , and the entries of the vector
$$
\boldsymbol {\Sigma} _ {\mathcal {D} _ {\mathrm {t r}}} ^ {- 1} \boldsymbol {\Delta} \boldsymbol {y} _ {\mathrm {c a l}} = \left(\frac {\Delta y _ {\mathrm {c a l}} ^ {1}}{\sigma_ {\mathcal {D} _ {\mathrm {t r}}} (\boldsymbol {\theta} _ {\delta} , \boldsymbol {x} _ {\mathrm {c a l}} ^ {1})}, \dots , \frac {\Delta y _ {\mathrm {c a l}} ^ {N _ {\mathrm {c a l}}}}{\sigma_ {\mathcal {D} _ {\mathrm {t r}}} (\boldsymbol {\theta} _ {\delta} , \boldsymbol {x} _ {\mathrm {c a l}} ^ {N _ {\mathrm {c a l}}})}\right) ^ {\top}
$$
correspond to the z-scores of the data under the calibration standard deviation  $\sigma_{\mathcal{D}_{\mathrm{tr}}}(\pmb{\theta}_{\delta},\cdot)$ . The original problem (5) then becomes
$$
\min  _ {\boldsymbol {\theta} \in \boldsymbol {\Theta}} \sum_ {i = 1} ^ {N _ {\mathrm {c a l}}} \left[ q _ {\text {l i n}} (\delta , \boldsymbol {\Sigma} _ {\mathcal {D} _ {\mathrm {t r}}} ^ {- 1} \boldsymbol {\Delta} \boldsymbol {y} _ {\mathrm {c a l}}) \sigma_ {\mathcal {D} _ {\mathrm {t r}}} \left(\boldsymbol {\theta} _ {\delta}, \boldsymbol {x} _ {\mathrm {c a l}} ^ {i}\right) \right] ^ {2}, \tag {7}
$$
which is considerably easier to solve due to the lack of constraints, and enables us to use gradient-based approaches, since  $q_{\mathrm{lin}}(\delta, \Sigma_{\mathcal{D}_{\mathrm{tr}}}^{-1} \Delta \boldsymbol{y}_{\mathrm{cal}})$  is differentiable with respect to  $\sigma_{\mathcal{D}_{\mathrm{tr}}}(\pmb{\theta}_{\delta}, \pmb{x}_{\mathrm{cal}}^i)$ .
Remark 5.2. The choice of interpolant (6) is due to its simplicity. However, other forms of monotone interpolation are also possible. For high data sizes, the choice of interpolant becomes of little relevance, since we only perform small interpolation steps.
5.2 Calibrated GP for Arbitrary Confidence Level  $\delta$
While (7) is useful for obtaining a sharply calibrated model for a single confidence level  $\delta$ , solving (7) multiple times whenever we want to obtain sharply calibrated models for different confidence levels  $\delta$  can be time-consuming. Furthermore, interpolating between any two arbitrary solutions of (7) won't necessarily yield a result close to the desired calibration. Fortunately, we can leverage Assumption 4.1 to show that interpolating between two solutions of (7) will yield a result close to the
For  $\pmb{a} \in \mathbb{R}^{N_{\mathrm{cal}}}$ , a permutation  $i_1, \ldots, i_{N_{\mathrm{cal}}} \in [1, \ldots, N_{\mathrm{cal}}]$  where  $a_{i_j} \leq a_{i_{j+1}}$  for all  $j$ , and  $\delta \in [l / (N_{\mathrm{cal}} + 1), (l + 1) / (N_{\mathrm{cal}} + 1)]$ ,
$q_{\mathrm{lin}}(\delta ,\pmb {a}) = a_{i_l} + (\delta (N + 1) - l)\left(a_{i_{l + 1}} - a_{i_l}\right).$
Algorithm 1 Training Calibration Hyperparameters for Arbitrary Confidence Level
Input: kernel  $k(\cdot ,\cdot ,\cdot)$  , predictor  $\mu_{\mathcal{D}_{\mathrm{tr}}}(\pmb {\theta}^R,\cdot)$  , calibration data  $\mathcal{D}_{\mathrm{cal}}$  , confidence levels  $\delta_1,\ldots \delta_N$  for  $i = 1$  to  $M$  do
Compute  $\beta_{\delta_1}, \pmb{\theta}_{\delta_1}, \dots, \beta_{\delta_{N_{\mathrm{cal}}}}, \pmb{\theta}_{\delta_{N_{\mathrm{cal}}}}$  by solving (6) and (7) subject to