SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The theoretical property of Algorithm 2 can be established by combining the results from Section 3.1 with Proposition 1. In Algorithm 2, the quantile calibration allows full flexibility in choosing the regression model  $\hat{f}$  and does not require an associated uncertainty model to proceed. The motivation for calibrating the error  $U_{i}$  instead of the original  $Y_{i}$  is two-fold: First, if one treats  $Y_{i}$  itself as  $U_{i}$  and applies Algorithm 1, then it essentially restricts the prediction model to be a nonparametric one. Second, the reduction from  $Y_{i}$  to  $U_{i}$  gives a better smoothness of the conditional distribution (a
Algorithm 2 Nonparametric Regression Calibration
Input: Dataset  $\{(X_i,Y_i)\}_{i = 1}^n$  , kernel choice  $\kappa_{h}(\cdot ,\cdot),\tau$
1: Split the dataset into half:  $\mathcal{D}_1 = \{(X_i,Y_i)\}_{i = 1}^{n_1}$ ,  $\mathcal{D}_2 = \{(X_i,Y_i)\}_{i = n_1 + 1}^n$ .
2: Use  $\mathcal{D}_1$  to train a regression model  $\hat{f}$ .
3: Calculate the estimation error of  $\hat{f}$  on  $\mathcal{D}_2$ :  $U_i = Y_i - \hat{f}(X_i)$ ,  $i = n_1 + 1, \dots, n$ .
4: Run Algorithm 1 on the data  $\{(X_i, U_i)\}_{i=n_1+1}^n$  and obtain  $\hat{Q}_\tau^{\mathrm{SNQ}}(\cdot)$ .
5: Return  $\hat{f} (\cdot) + \hat{Q}_{\tau}^{\mathrm{SNQ}}(\cdot)$ .
smaller Lipschitz constant in Assumption 1 (a)). And thus it will give a faster convergence rate. We remark that Algorithm 2 induces an independence between the training of the prediction model and that of the calibration model. The design is intuitive and important because otherwise, it may result in predicting smaller confidence intervals both theoretically and empirically.
3.3 Implications from the nonparametric perspective
The nonparametric approach gives a positive result in terms of the possibility of individual calibration, but it pays a price with respect to the dimensionality  $d$ . The following theorem states that such a dimensionality problem can be addressed under conditional independence. And more generally, it provides a guideline on which variables one should use to perform calibration tasks.
Theorem 3. Suppose  $U = Y - \hat{f}(X)$ . Suppose  $Z = m(X) \in \mathcal{Z}$ , where  $m$  is some measurable function, and  $\mathcal{Z} \subset \mathcal{X}$  is a  $d_0$ -dimensional subspace of  $\mathcal{X}$ . If  $X$  and  $U$  are mutually independent conditioned on  $Z$  (i.e.  $X \to Z \to U$  is a Markov chain), then it is lossless to perform calibration using only  $(U_i, Z_i)$ 's. In particular, applying Algorithm 1 on  $(U_i, Z_i)$ 's will yield the same consistency but with a faster convergence rate of  $O(n^{-\frac{2}{d_0 + 2}})$ .
Theoretically, one can identify such  $Z$  by independence test for each component of  $X$  and  $Y - \hat{f}(X)$  on the validation set. Some other practical methods such as random projection and correlation screening selection are shown in Section 4. When the conditional independence in Theorem 3 does not hold, it is still not the end of the world if one calibrates  $(U_i, Z_i)$ . The following theorem tells that conditional calibration with respect to  $Z$ , as an intermediate between marginal calibration and individual calibration, can be achieved.
Theorem 4. Suppose  $U = Y - \hat{f}(X)$  and  $Z$  is a sub-feature of  $X$ . Without loss of generality, we assume that  $Z = \Pi_{d_0}(X)$ , where  $\Pi_{d_0}$  projects from  $\mathcal{X}$  onto a  $d_0$ -dimensional subspace  $\mathcal{Z}$ . Then
$$
\mathbb {P} (U \leq Q _ {\tau} (U | Z) \mid Z) = \tau ,
$$
which implies that  $\mathbb{P}(Y\leq \hat{f} (X) + Q_{\tau}(U|Z)\mid Z) = \tau$
A notable implication from the above theorem is that if we want to calibrate the model against certain sub-features such as age, gender, geo-location, etc., we can summarize these features in  $Z$  and use Algorithm 1 with respect to  $(U_i, Z_i)$ .
However, if we return to the original pinball loss, the following theorem inherits from Proposition 2 and states that the inclusion of more variables will always be helpful in terms of improving the pinball loss. It highlights a trade-off between the representation ability of the quantile calibration model (inclusion of more variables) and its generalization ability (slowing down the convergence rate with higher dimensionality).
Theorem 5. Suppose  $U = Y - \hat{f}(X)$ . Suppose  $Z^{(d_1)}$  and  $Z^{(d_2)}$  are the first  $d_1$  and  $d_2$  components of  $X$ . Suppose  $Q_{\tau}(F_{Z^{(d_i)}})$  is the true  $\tau^{-th}$  quantile of  $U$  conditioned on  $Z^{(d_i)}$ ,  $i = 1, 2$ . Suppose  $Q_{\tau}(F_{margin})$  is the true  $\tau^{-th}$  marginal quantile of  $U$ . If  $0 < d_1 < d_2 < d$ , then
$$
\begin{array}{l} \mathbb {E} \left[ l _ {\tau} (\hat {f} (X) + Q _ {\tau} (U), Y) \right] \geq \mathbb {E} \left[ l _ {\tau} (\hat {f} (X) + Q _ {\tau} (U | Z ^ {(d _ {1})}), Y) \right] \\ \geq \mathbb {E} [ l _ {\tau} (\hat {f} (X) + Q _ {\tau} (U | Z ^ {(d _ {2})}), Y) ] \geq \mathbb {E} [ l _ {\tau} (\hat {f} (X) + Q _ {\tau} (U | X), Y) ]. \\ \end{array}
$$
In this section, we evaluate our methods against a series of benchmarks. We first introduce the evaluation metrics that are widely considered in the literature of uncertainty calibration and conformal prediction, including Mean Absolute Calibration Error (MACE) which calculates the average absolute error for quantile predictions from 0.01 to 0.99; Adversarial Group Calibration Error (AGCE) which finds the sub-group of the test data with the largest MACE; Check Score which shows the empirical pinball loss; Length which measures the average length for the constructed 0.05-0.95 intervals; and Coverage which reflects the empirical coverage rate for those 0.05-0.95 intervals. Formal definitions of these calibration measurements are provided in Appendix E.1.
The benchmark methods are listed as follows (the details can be found in Appendix E.4). For the Gaussian-based methods, we implement the vanilla Heteroscedastic Neural Network (HNN) [Kendall and Gal, 2017], Deep Ensembles (DeepEnsemble) [Lakshminarayanan et al., 2017], and Maximum Mean Discrepancy method (MMD) [Cui et al., 2020]. We also implement MC-Dropout (MCDrop) [Gal and Ghahramani, 2016] and Deep Gaussian Process model (DGP) [Damianou and Lawrence, 2013]. For models that do not rely on Gaussian assumption, we implement the combined calibration loss (CCL) introduced in Sec. 3.2 of [Chung et al., 2021]. We also implement post-hoc calibration methods, such as isotonic regression (ISR) suggested by [Kuleshov et al., 2018] with a pre-trained HNN model. Another post-hoc method named Model Agnostic Quantile Regression (MAQR) method in Sec. 3.1 of Chung et al. [2021] is also implemented. For conformal prediction algorithms, we implement Conformalized Quantile Regression (CQR) [Romano et al., 2019] and Orthogonal Quantile Regression (OQR) [Feldman et al., 2021].
As our proposed nonparametric regression calibration (NRC) algorithm is agnostic with respect to the underlying prediction model, we implement a feed-forward neural network and a random forest model as the prediction model for NRC. Also, we apply the dimension reduction idea (called NRC-DR) including random projection and covariate selection where all technical details are given in Appendix E.3 and E.5. All codes are available at https://github.com/ZhongzeCai/NRC.
In the following subsections, we summarize our results on UCI datasets, time series, and high-dimensional datasets. Additional experiments on covariate shift are deferred to E.7.
4.1 UCI dataset experiments
We evaluate our NRC algorithms on the standard 8 UCI datasets Dua and Graff [2017]. Part of the experiment results on representative benchmark models is given in Table 1, and the full results are presented in Appendix E.5 due to space limit. Additional details on dataset description, hyperparameter selection, and early stopping criteria are given in Appendix E.5.
The result supports the competitiveness of our algorithm. In terms of MACE, our methods achieve the minimum loss on 6 out of 8 datasets, with the non-optimal results close to the best performance seen on other benchmarks. The rest metrics also witness our algorithms outperforming the benchmark models, with a superior result on 6 datasets in terms of AGCE, and 4 in terms of CheckScore. Note that even for the rest 4 datasets where our NRC algorithm does not achieve the best CheckScore (the empirical pinball loss), our performance is still competitive compared to the best score, while for some specific datasets (such as Energy and Yacht) we achieve a significant improvement over the CheckScore, which verifies the potential of our methods for the related downstream tasks. As for the Length metric (which evaluates the average interval length of a symmetric  $90\%$  confidence interval), it is not easy to trade-off between the sharpness requirement and the coverage rate due to the fact that one can trivially sacrifice one for the other, so we omit the direct comparison. However, we can still observe that our NRC algorithm behaves reasonably well, with a stable coverage rate performance and a relatively sharp interval length amongst the rest benchmarks.
4.2 Bike-sharing time series dataset
This experiment explores the potential of our algorithm to quantify the uncertainty of the time series objective, where independent variables are aggregated in a sliding window fashion from the past few days. The Bike-Sharing dataset provided in Fanaee-T and Gama [2014] is used to visually evaluate our proposed algorithm. Due to the limitations in the representation ability of the feed-forward structure, we instead deploy LSTM networks for the time series data. We design LSTM-HNN,
which is an ordinary LSTM network followed by a fully connected layer with 2 output units; and LSTM-NRC, whose regression model shares the same LSTM setting as LSTM-HNN except for the ending layer having only one output unit (without variance/uncertainty unit). For more details, see Appendix E.6.
Figure 2: Experiments on the Bike-Sharing dataset. The demonstrated time stamps are consecutively taken from the test set, thus temporally separated from the training set. The top figure gives the predicted  $95\%$  confidence intervals of an LSTM-HNN model, and the bottom figure is produced from a random projection variant of our NRC method (LSTM-NRC). The target variable shows strong periodicity, which both two models capture. Nevertheless, our NRC method delivers sharper confidence intervals and is better adapted to the underlying distribution shift. In particular, our method also has a better coverage rate on the extreme values at time stamps between 100 and 200.
Figure 2 visually compares the two models on the  $95\%$  confidence interval prediction task on the test set (see Appendix E.6 for how the test set is split out). A first observation is that our LSTM-NRC is producing sharper and better calibrated  $95\%$  intervals. Also, in spite of the satisfying coverage rate of both the methods on time range 0 to 100 (time stamp -1 is exactly the terminating point of the training set, thus time range 0 to 100 roughly denotes the near future of the historical events), on time range 100 to 200 the behaviors of the two models diverge. After an evolution of 100 time stamps the target variable (rental count per hour) is under a shift in distribution: the peak values get higher than historical records would foresee. Confidence intervals produced by LSTM-HNN fail to cover these "extremer" peaks, while our LSTM-NRC method could easily adapt to it, taking advantage of a high-quality regression model. Note that both LSTM-HNN and LSTM-NRC share the same regression architecture, we argue that the additional variance unit of LSTM-HNN may hurt the mean prediction performance of the model.
4.3 Higher-dimensional datasets
One major drawback of the non-parametric type methods is its unsatisfying dependence on the dimension of the data. Our paper attacks such an obstacle from two aspects simultaneously: on the one hand, we subtract a regression model from the original data to gain a smaller Lipschitz constant  $L$ ; on the other hand, we apply several dimensional reduction methods to reduce the data dimension manually. In this paper, we mainly implement three of them: random projection, covariate selection, and second-to-last layer feature extraction, of which the details can be found in Appendix E.3. We call the variants of our NRC algorithm NRC-RP, NRC-Cov, and NRC-Embed, respectively. We examine them against several high-dimensional datasets: the medical expenditure panel survey (MEPS) datasets [panel 19, 2017, panel 20, 2017, panel 21, 2017], as suggested in [Romano et al., 2019]. The datasets' details are also deferred to Appendix E.3.
Dataset	Metric	DeepEnsemble	ISR	CQR	OQR	NRC-RP	NRC-Embed
meps_19	MACE	0.26±0.04	0.12±0.05	0.05±0.01	0.055±0.02	0.024±0.01	0.0087±0.006
AGCE	0.26±0.04	0.14±0.04	0.079±0.01	0.091±0.03	0.062±0.02	0.04±0.02
CheckScore	25±7	10±6	2.4±0.2	2.5±0.2	3.2±0.4	2.4±0.7
Length(Coverage)	230±140(99%)	50±6(93%)	28±2(89%)	25±2(85%)	35±8(90%)	24±0.5(90%)
meps_20	MACE	0.25±0.02	0.16±0.08	0.046±0.01	0.057±0.024	0.0052±0.004	0.0058±0.005
AGCE	0.25±0.02	0.18±0.08	0.061±0.01	0.08±0.03	0.033±0.01	0.041±0.007
CheckScore	23±8	7.4±1	2.5±0.1	2.5±0.1	3±0.4	2.4±0.08
Length(Coverage)	190±50(99%)	69±38(94%)	28±2(89%)	27±4(85%)	32±5(90%)	27±3(91%)
meps_21	MACE	0.25±0.02	0.13±0.1	0.05±0.02	0.0087±0.004	0.012±0.002	0.0088±0.005
AGCE	0.26±0.02	0.15±0.1	0.063±0.008	0.075±0.02	0.045±0.02	0.041±0.01
CheckScore	36±20	23±10	2.5±0.3	2.6±0.3	3.5±0.7	2.7±1
Length(Coverage)	160±40(99%)	43±20(93%)	27±3(86%)	29±3(89%)	39±3(90%)	24±2(91%)
Table 2: Experiments on MEPS Datasets
In Table 2 we summarize the dimension reduction variants of our NRC algorithm against several benchmarks. The result sees a substantial advantage of our method over the selected benchmark methods, while most of the unshown benchmarks either are too computationally expensive to run or do not deliver competitive results on the high-dimensional dataset.
In this paper, we propose simple nonparametric methods for regression calibration, and through its lens, we aim to gain a better theoretical understanding of the problem. While numerical experiments show the advantage, we do not argue for a universal superiority of our methods. Rather, we believe the contribution of our work is first to give a positive result for individual calibration and an awareness of its importance, and second to show the promise of returning to simple statistical estimation approaches than designing further complicated loss functions without theoretical understanding. Future research includes (i) How to incorporate the uncertainty output from a prediction model with our nonparametric methods; (ii) While we do not observe the curse of dimensionality for our methods numerically and we explain it by the reduction of Lipschitzness constant, it deserves more investigation on how to adapt the method for high-dimensional data.
Tim Pearce, Alexandra Brintrup, Mohamed Zaki, and Andy Neely. High-quality prediction intervals for deep learning: A distribution-free, ensembled approach. In International conference on machine learning, pages 4075-4084. PMLR, 2018.
Joaquin Quinonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. Mit Press, 2008.
Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. Advances in neural information processing systems, 32, 2019.
Hugh Salimbeni and Marc Deisenroth. Doubly stochastic variational inference for deep gaussian processes. Advances in neural information processing systems, 30, 2017.
Hao Song, Tom Diethe, Meelis Kull, and Peter Flach. Distribution calibration for regression. In International Conference on Machine Learning, pages 5897-5906. PMLR, 2019.
Ingo Stainwart and Andreas Christmann. Estimating conditional quantiles with the help of the pinball loss. Bernoulli, 17(1):211-225, 2011.
Balazs Szorenyi, Róbert Busa-Fekete, Paul Weng, and Eyke Hüllermeier. Qualitative multi-armed bandits: A quantile-based approach. In International Conference on Machine Learning, pages 1660–1668. PMLR, 2015.
Ichiro Takeuchi, Quoc V Le, Timothy D Sears, and Alexander J Smola. Nonparametric quantile estimation. Journal of Machine Learning Research, 7(45):1231-1264, 2006.
Jayaraman J Thiagarajan, Bindya Venkatesh, Prasanna Sattigeri, and Peer-Timo Bremer. Building calibrated deep models via uncertainty matching with auxiliary interval predictors. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6005-6012, Apr. 2020.
Léonard Torossian, Aurélien Garivier, and Victor Picheny.  $\mathcal{X}$ -armed bandits: Optimizing quantiles, cvar and other risks. In *Asian Conference on Machine Learning*, pages 252-267. PMLR, 2019.
Vladimir Vovk. Conditional validity of inductive conformal predictors. In *Asian conference on machine learning*, pages 475-490. PMLR, 2012.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world, volume 29. Springer, 2005.
Junfeng Wen, Chun-Nam Yu, and Russell Greiner. Robust learning under uncertain test distributions: Relating covariate shift to model misspecification. In International Conference on Machine Learning, pages 631-639. PMLR, 2014.
Jonathan Wenger, Hedvig Kjellström, and Rudolph Triebel. Non-parametric calibration for classification. In International Conference on Artificial Intelligence and Statistics, pages 178-190. PMLR, 2020.
Jia Yuan Yu and Evdokia Nikolova. Sample complexity of risk-averse bandit-arm selection. In *IJCAI*, pages 2576–2582, 2013.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Mengyan Zhang and Cheng Soon Ong. Quantile bandits for best arms identification. In International Conference on Machine Learning, pages 12513-12523. PMLR, 2021.
Shengjia Zhao, Tengyu Ma, and Stefano Ermon. Individual calibration with randomized forecasting. In International Conference on Machine Learning, pages 11387-11397. PMLR, 2020.
A Related Literature
A.1 Estimating the whole distribution
The most straightforward thought is to estimate the whole distribution which accomplishes the task of mean regression and uncertainty quantification simultaneously. Generally, there are two ways to fit a distribution: either fitting a parametric model or turning to nonparametric methods. Representative parametric methods are Bayesian Neural Networks [MacKay, 1992] and Deep Gaussian Process (DGP) [Damianou and Lawrence, 2013], of which the latter assumes the data is generated according to a Gaussian process of which the parameters are further generated by another Gaussian process. The major drawbacks of these Bayesian-type methods are their high computation cost during the training phase and statistical inconsistency under model misspecification. To resolve the computational issue, [Gal and Ghahramani, 2016] proposes a simple MC-Dropout method that captures the model uncertainty without changing the network structure. Despite the view of Bayesian, the task of distribution estimation also falls into the range of frequentists: by assuming the underlying distribution family (say, Gaussian), one can fulfill the task by estimating the moments or maximizing the likelihood. For example, the Heteroscedastic Neural Network (HNN) [Kendall and Gal, 2017] gives an estimation of both mean and variance at the final layer by assuming the Gaussian distribution. An ensemble method called Deep Ensemble [Lakshminarayanan et al., 2017] is constructed on the basis of HNN. [Cui et al., 2020] designs an algorithm that finds the best Gaussian distribution to minimize the empirical Maximum Mean Discrepancy (MMD) from the observed data so as to ensure the marginal calibration property of quantiles. [Zhao et al., 2020] raises a randomized forecasting principle that minimizes the randomized calibration error with a negative log-likelihood regularization term to ensure the so-called sharpness. Although they claim the construction of (randomized) individual calibration (named PAIC) from the existence of mPAIC predictors, they do not provide any non-trivial mPAIC predictor except for the one that permanently predicts the uniform distribution.
The other approach to estimating the entire distribution is the nonparametric way without assuming the underlying distribution. For example, Kernel Density Estimation (KDE) utilizes kernel methods to estimate the conditional probability density function. Although the method enjoys certain theoretical consistency guarantees [Lei and Wasserman, 2014, Lei et al., 2018, Bilodeau et al., 2021], it is often computationally expensive and not sample-efficient in high dimensional cases, which makes it impractical for uncertainty quantification. A similar idea is applied in [Song et al., 2019] via a post-hoc way. The method is to improve on some given distribution estimation via the Gaussian process and Beta link function. However, unlike the KDE literature, the method in [Song et al., 2019] lacks a theoretical guarantee of its consistency.
A.2 Estimating quantiles
Due to the excessive and sometimes unnecessary difficulties in estimating the distribution, some researchers suggest estimating the quantiles only, since a satisfying quantile estimation suffices in many downstream tasks. The quantile prediction is now regarded as a new task, where one natural idea is to combine the average calibration error on quantiles with a sharpness regularization term. [Pearce et al., 2018] appends a calibration error (with respect to coverage and sharpness) to the loss functions to obtain the calibrated prediction intervals at a pre-specified confidence level. [Thiagarajan et al., 2020] induces an extra interval predictor with a similar combined loss. But their methods require the pre-determined confidence or quantile level and the whole model needs to be retrained if one wants another level of prediction. Such methods in [Pearce et al., 2018, Thiagarajan et al., 2020] are only evaluated empirically without theoretical guarantee for even the marginal distribution. [Chung et al., 2021] proposes an average calibration loss combined with a similar sharpness regularization term, which achieves an average (but not individually, as we will discuss later) calibration. Along with these models that aim to give a quantile prediction out of scratch, [Kuleshov et al., 2018] develops a post-hoc way of calibrating (on average) a pre-trained quantile prediction model via isotonic regression methods. The aim is to choose a specific level of quantile prediction that reaches the desired marginal calibration of which the consistency relies solely on the pre-trained model in contrast to our model that applies a model-agnostic post-hoc calibration.
Another line of research is to use Kernelized Support Vector Machines (KSVM) to predict the conditional quantiles in one step, which fixes the inconsistency issue [Takeuchi et al., 2006, Stainwart and Christmann, 2011]. Essentially, methods from both papers search for a quantile prediction
function over the Reproducing Kernel Hilbert Space (RKHS), say  $\mathcal{H}$ . Theoretically, Takeuchi et al. [2006] derive a performance guarantee for the conditional quantile estimator based on the assumption of finite  $\mathcal{H}$ -norm, which is in parallel with our Lipschitz class (Assumption 1 (a)). The Lipschitz function class and the bounded  $\mathcal{H}$ -norm function class overlap, but neither one is contained in the other. Stainwart and Christmann [2011] analyze the same algorithm but adopt a different approach. Instead of imposing a bounded  $\mathcal{H}$ -norm, they impose a decay rate for the eigenvalues of the kernel integral operator, alongside some other assumptions on the underlying distribution. Those works focus on the RKHS space while we focus on the Lipschitz space. For the Lipschitz function class, to the best of our knowledge, our work is the first result to provide an individual guarantee. From the practical point of view, both papers rely on solving a kernelized learning problem that cannot scale well with the sample size or dimension, while our algorithm is simple to implement and of low computational cost.