SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2 reports the sample mean and sample standard deviation of MSE from our proposed PNC predictor and other baseline approaches with 10 experimental repetition times. As shown, the proposed PNC achieves notably smaller MSE than baselines with similar computational costs (the base network and deep ensemble with two networks) in all experiments. Moreover, it achieves better or analogous results compared with the deep ensemble with 5 networks, but the running time of the latter is 2.5 times as much as that of PNC. It also has relatively small sample standard deviations, showing its capacity to reduce variability in networks effectively. These results verify that our proposed PNC can successfully produce better prediction and simultaneously bears significantly reduced computational costs compared to the classical deep ensemble.
In this study, we propose a systematic epistemic uncertainty assessment framework with simultaneous statistical coverage guarantees and low computation costs for over-parametrized neural networks in regression. Benefiting from our proposed PNC approach, our study shows promise to characterize and eliminate procedural uncertainty by introducing only one artificial-label-trained network. Integrated with suitable light-computation resampling methods, we provide two effective approaches to construct asymptotically exact-coverage confidence intervals using few model retrainings. Our evaluation results in different settings corroborate our theory and show that our approach can generate confidence intervals that are narrow and have satisfactory coverages simultaneously. In the future, we will extend our current framework to general loss functions that can potentially handle classification tasks to open up opportunities for broader applications.
This work has been supported in part by the National Science Foundation under grants CAREER CMMI-1834710 and IIS-1849280, and the Cheung-Kong Innovation Doctoral Fellowship. The authors thank the anonymous reviewers for their constructive comments which have helped greatly improve the quality of our paper.
[1] Y. Abbasi-Yadkori, D. Pál, and C. Szepesvári. Improved algorithms for linear stochastic bandits. Advances in Neural Information Processing Systems, 24:2312-2320, 2011.
[2] M. Abdar, F. Pourpanah, S. Hussain, D. Rezazadegan, L. Liu, M. Ghavamzadeh, P. Fieguth, X. Cao, A. Khosravi, U. R. Acharya, et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76:243-297, 2021.
[3] A. Alaa and M. Van Der Schaar. Frequentist uncertainty in recurrent neural networks via blockwise influence functions. In International Conference on Machine Learning, pages 175-190. PMLR, 2020.
[4] Z. Allen-Zhu, Y. Li, and Y. Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. Advances in Neural Information Processing Systems, 2019.
[5] Z. Allen-Zhu, Y. Li, and Z. Song. A convergence theory for deep learning via overparameterization. In International Conference on Machine Learning, pages 242-252. PMLR, 2019.
[6] Z. Allen-Zhu, Y. Li, and Z. Song. On the convergence rate of training recurrent neural networks. Advances in Neural Information Processing Systems, 32:6676-6688, 2019.
[7] S. Arora, S. S. Du, W. Hu, Z. Li, R. R. Salakhutdinov, and R. Wang. On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems, 32:8141-8150, 2019.
[8] A. Ashukha, A. Lyzhov, D. Molchanov, and D. Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. arXiv preprint arXiv:2002.06470, 2020.
[9] P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2):235-256, 2002.
[10] Y. Bai, S. Mei, H. Wang, and C. Xiong. Understanding the under-coverage bias in uncertainty estimation. arXiv preprint arXiv:2106.05515, 2021.
[11] R. F. Barber, E. J. Candes, A. Ramdas, and R. J. Tibshirani. The limits of distribution-free conditional predictive inference. arXiv preprint arXiv:1903.04684, 2019.
[12] R. F. Barber, E. J. Candes, A. Ramdas, and R. J. Tibshirani. Predictive inference with the jackknife+. arXiv preprint arXiv:1905.02928, 2019.
[13] S. Basu, P. Pope, and S. Feizi. Influence functions in deep learning are fragile. arXiv preprint arXiv:2006.14651, 2020.
[14] A. Berlinet and C. Thomas-Agnan. Reproducing kernel Hilbert spaces in probability and statistics. Springer Science & Business Media, 2011.
[15] A. Bietti and J. Mairal. On the inductive bias of neural tangent kernels. Advances in Neural Information Processing Systems, 32, 2019.
[16] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, pages 1613-1622. PMLR, 2015.
[17] L. Breiman. Bagging predictors. Machine Learning, 24:123-140, 1996.
[118] Y. Zhang, Z.-Q. J. Xu, T. Luo, and Z. Ma. A type of generalization error induced by initialization in deep neural networks. In Mathematical and Scientific Machine Learning, pages 144-164. PMLR, 2020.
[119] D. Zhou, L. Li, and Q. Gu. Neural contextual bandits with ucb-based exploration. In International Conference on Machine Learning, pages 11492-11502. PMLR, 2020.
[120] M. Zhou, R. Ge, and C. Jin. A local convergence theory for mildly over-parameterized two-layer neural network. In Conference on Learning Theory, pages 4577-4632. PMLR, 2021.
[121] X. Zhou, Y. Jiao, J. Liu, and J. Huang. A deep generative approach to conditional sampling. Journal of the American Statistical Association, pages 1-12, 2022.
[122] D. Zou, Y. Cao, D. Zhou, and Q. Gu. Gradient descent optimizes over-parameterized deep relu networks. Machine Learning, 109(3):467-492, 2020.
We provide further results and discussions in this supplementary material. Section A discusses aleatoric uncertainty and predictive uncertainty. Section B discusses statistical inference for kernel-based regression and, in particular, the asymptotic normality of kernel-based regression. Section C discusses the training of the linearized neural networks based on the NTK theory. In particular, we show in Proposition 3.1 that the shifted kernel ridge regressor using NTK with a shift from an initial function  $s_{\theta^b}$  is exactly the linearized neural network regressor that starts from the initial network  $s_{\theta^b}$ . Section D provides additional remarks to explain some of the details in the main paper. Section E presents the proofs for results in the paper. Section F presents experimental details and more experimental results.
Appendix A Other Types of Uncertainty
Back in 1930, [39] first introduced a formal distinction between aleatory and epistemic uncertainty in statistics [52], while in modern machine learning, their distinction and connection were investigated in [101] and further extended to deep learning models [67, 33]. In [94], the differences between procedural and data variabilities in epistemic uncertainty were intuitively described; however, no rigorous definition or estimation method was provided. Our previous draft [60] discussed intuitive approaches for quantifying procedural and data variabilities, which some of the ideas in this work originate from. In the following, we present an additional discussion on other types of uncertainty for the sake of completeness. Section A.1 presents aleatoric uncertainty and Section A.2 presents predictive uncertainty.
A.1 Aleatoric Uncertainty
We note that if we could remove all the epistemic uncertainty, i.e., letting  $\mathrm{UQ}_{EU} = 0$ , the best predictor we could get is the Bayes predictor  $h_B^*$ . However,  $h_B^*$ , as a point estimator, is not able to capture the randomness in  $\pi_{Y|X}$  if it is not a point mass distribution.
The easiest way to think of aleatoric uncertainty is that it is captured by  $\pi_{Y|X}$ . At the level of the realized response or label value, this uncertainty is represented by
$$
\mathbf {U Q} _ {A U} = y - h _ {B} ^ {*} (x)
$$
If the connection between  $X$  and  $Y$  is non-deterministic, the description of a new prediction problem involves a conditional probability distribution  $\pi_{Y|X}$ . Standard neural network predictors can only provide a single output  $y$ . Thus, even given full information of the distribution  $\pi$ , the uncertainty in the prediction of a single output  $y$  remains. This uncertainty cannot be removed by better modeling or more data.
There are multiple work aiming to estimate the aleatoric uncertainty, i.e., to learn  $\pi_{Y|X}$ . For instance, conditional quantile regression aims to learn each quantile of the distribution  $\pi_{Y|X}$  [70, 89, 108]. Conditional density estimation aims to approximately describe the density of  $\pi_{Y|X}$  [57, 36, 62, 29, 42, 63]. However, we remark that these approaches also face their own epistemic uncertainty in the estimation. See [26, 108, 10] for recent studies on the epistemic bias in quantile regression.
A.2 Predictive Uncertainty
Existing work on uncertainty measurement in deep learning models mainly focuses on prediction sets (predictive uncertainty), which captures the sum of epistemic and aleatoric uncertainties [94, 97, 12, 3, 22].
In certain scenarios, it is not necessary to estimate each uncertainty separately. A distinction between aleatoric and epistemic uncertainties might appear less significant. The user may only concern about the overall uncertainty related to the prediction, which is called predictive uncertainty and can be thought as the summation of the epistemic and aleatoric uncertainties.
The most common way to quantify predictive uncertainty is the prediction set: We aim to find a map  $\hat{C}:\mathcal{X}\to 2^{\mathcal{Y}}$  which maps an input to a subset of the output space so that for each test point
$x_0 \in \mathcal{X}$ , the prediction set  $\hat{C}(x_0)$  is likely to cover the true outcome  $y_0$  [111, 11, 12, 84, 83, 82]. This prediction set is also called a prediction interval in the case of regression [68, 69]. The prediction set communicates predictive uncertainty via a statistical guarantee on the marginal coverage, i.e.,
$$
\mathbb {P} (y _ {0} \in \hat {C} (x _ {0})) \geq 1 - \delta
$$
for a small threshold  $\delta > 0$  where the probability is taken with respect to both training data  $\mathcal{D}_{tr}$  (epistemic uncertainty) for learning  $\hat{C}$  and the test data  $(x_0, y_0)$  (aleatoric uncertainty). It is more tempting to obtain a statistical guarantee with only aleatoric uncertainty by considering the probability conditional on the prediction set, i.e.,
$$
\mathbb {P} \left(y _ {0} \in \hat {C} \left(x _ {0}\right) \mid \hat {C} ^ {\prime}\right) \geq 1 - \delta . \tag {9}
$$
However, this guarantee is in general very hard to achieve in the finite-sample case. Even asymptotically, (9) is not easy to achieve unless we have a very simple structure on the data distribution [98, 116]. A recent study shows that (9) could hold in the finite-sample sense if we could leverage a set of validation data [22].
Appendix B Statistical Inference for Kernel-Based Regression
B.1 Classical Statistical Learning Framework and Kernel Ridge Regression
Following Section 2, assuming that the input-output pair  $(X,Y)$  is a random vector following an unknown probability distribution  $\pi$  on  $\mathcal{X} \times \mathcal{Y}$  where  $X \in \mathcal{X} \subset \mathbb{R}^d$  is an input and  $Y \in \mathcal{Y} \subset \mathbb{R}$  is the corresponding output. Let the marginal distribution of  $X$  be  $\pi_X(x)$  and the conditional distribution of  $Y$  given  $X$  be  $\pi_{Y|X}(y|x)$ .
Suppose a learner has access to a set of training data  $\mathcal{D} = \{(x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)\}$ , which is assumed to be independent and identically distributed (i.i.d.) according to the data distribution  $\pi$ . Let  $\pmb{x} = (x_1, \dots, x_n)^T$  and  $\pmb{y} = (y_1, \dots, y_n)^T$  for short. Let  $\pi_{\mathcal{D}}$  or  $\pi_n$  denote the empirical distribution associated with the training data  $\mathcal{D}$ , where  $\pi_n$  is used to emphasize the sample size dependence in asymptotic results:
$$
\pi_ {\mathcal {D}} = \pi_ {n} = \frac {1}{n} \sum_ {i = 1} ^ {n} \delta_ {(x _ {i}, y _ {i})}.
$$
The goal of the learner is to obtain a function  $h: \mathcal{X} \to \mathcal{Y}$  such that  $h(x)$  is a "good" predictor for the response  $y$  if  $X = x$  is observed for any  $x \in \mathcal{X}$ . This is typically done by minimizing the following (ground-truth) population risk
$$
R _ {\pi} (h) := \mathbb {E} _ {(X, Y) \sim \pi} [ \mathcal {L} (h (X), Y) ]
$$
where  $\mathcal{L}:\mathcal{Y}\times \mathbb{R}\to \mathbb{R}$  is the loss function. For instance,  $\mathcal{L}$  can be the square error in regression or cross-entropy loss in classification. If  $h$  is allowed to be any possible functions, the best predictions in the sense of minimizing the risk are described by the Bayes predictor  $h_B^*$  [61, 93]:
$$
h_{B}^{*}(X):= \operatorname *{arg  min}_{\hat{y}\in \mathcal{Y}}\mathbb{E}_{Y\sim \pi_{Y|X}}\left[\mathcal{L}(\hat{y},Y)|X\right].
$$
Note that  $h_B^*$  cannot be obtained in practice, since the conditional distribution  $\pi_{Y|X}(y|x)$  is unknown. In particular, the least squares loss  $\mathcal{L}(h(X), Y) = (h(X) - Y)^2$  yields the (ground-truth) regression function defined by
$$
g _ {\pi} ^ {*} (x) := \mathbb {E} _ {(X, Y) \sim \pi} [ Y | X = x ].
$$
As the ground-truth distribution  $\pi$  is unknown, we can neither compute nor minimize the population risk  $R_{\pi}(h)$  directly. We define the risk under a general distribution  $P$ ,
$$
R _ {P} (h) := \mathbb {E} _ {(X, Y) \sim P} [ \mathcal {L} (h (X), Y) ].
$$
In particular, for  $P = \pi_{n}$  (the empirical distribution associated with the training data  $\mathcal{D}$ ), an empirical risk is derived:
$$
R _ {\pi_ {n}} (h) := \mathbb {E} _ {(X, Y) \sim \pi_ {n}} [ \mathcal {L} (h (X), Y) ] = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathcal {L} \left(h \left(x _ {i}\right), y _ {i}\right)
$$
In practice, minimizing the risk over  $h$  is restricted to a certain function class. Let  $\mathcal{H}$  be a hypothesis class, which is a set of functions  $\{h : \mathcal{X} \to \mathcal{Y} | h \in \mathcal{H}\}$ . For instance, 1)  $\mathcal{H}$  could be a nonparametric class such as a reproducing kernel Hilbert space (RKHS). The resulting method is known as kernel-based regression; See below. 2)  $\mathcal{H}$  could be a parametric class  $\{h_{\theta} : \mathcal{X} \to \mathcal{Y} | \theta \in \Theta\}$  where  $\theta$  is the parameter, and  $\Theta$  is the set of all possible parameters, e.g., the linear coefficients in a linear regression model, or the set of all network parameters in a neural network model.
In classical statistical learning, one is interested in finding a hypothesis  $g_{\pi} \in \mathcal{H}$  that minimizes the population risk
$$
g _ {\pi} := \underset {h \in \mathcal {H}} {\arg \min } R _ {\pi} (h). \tag {10}
$$
which is called the true risk minimizer. We remark that  $h_{\pi}$  is the best choice in the sense of minimizing the risk within the hypothesis set  $\mathcal{H}$  and different choices of  $\mathcal{H}$  may lead to different  $h_{\pi}$ . As  $\pi$  is unknown, the learner may consider minimizing the empirical risk:
$$
g _ {\pi_ {n}} := \underset {h \in \mathcal {H}} {\arg \min } R _ {\pi_ {n}} (h) \tag {11}
$$
which is called the empirical risk minimizer. More generally, to avoid overfitting in the finite sample regime, the learner may consider a regularized empirical risk minimization problem:
$$
g _ {\pi_ {n}, \lambda_ {n}} := \underset {h \in \mathcal {H}} {\arg \min } R _ {\pi_ {n}} (h) + \lambda_ {n} \| h \| _ {\mathcal {H}} ^ {2} \tag {12}
$$
and its corresponding population problem
$$
g _ {\pi , \lambda_ {0}} := \underset {h \in \mathcal {H}} {\arg \min } R _ {\pi} (h) + \lambda_ {0} \| h \| _ {\mathcal {H}} ^ {2} \tag {13}
$$
which are called the true regularized risk minimizer and the empirical regularized risk minimizer, respectively. Here,  $\lambda_{n} \geq 0$  is the regularization parameter, which may depend on the data size  $n$ , and we assume it has a limit  $\lambda_0 = \lim_{n\to \infty}\lambda_n$ . In general, the target  $g_{\pi}$  is not equal to  $g_{\pi_n,\lambda_n}$  or  $g_{\pi_n}$ . We omit 0 in the subscript if  $\lambda = 0$  and write  $g_{\pi_n} \coloneqq g_{\pi_n,0}$  and  $g_{\pi} = g_{\pi,0}$  for simplicity.
The above framework fits classical machine learning approaches such as linear regression or kernel-based convex regression (such as kernel ridge regression), since (12) as well as (13) has a unique solution in this setting. However, this framework is not suitable for deep learning: the empirical (regularized) risk minimizer  $g_{\pi_n,\lambda_n}$  is not unique and cannot be precisely obtained due to the nonconvex nature of neural networks. Therefore, the gradient-based methods in deep learning can only find an approximate solution  $g_{\pi_n,\lambda_n}$ , subjected to procedural variability.
Kernel ridge regression. Next, we apply the above framework to kernel ridge regression and review some basic existing results about kernel ridge regression, which can be found, e.g., in [14, 78, 104-106, 27, 109].
The kernel-based regression means that the hypothesis class  $\mathcal{H}$  in statistical learning is chosen to be an RKHS. Formally, a Hilbert space  $\mathcal{H}$  consisting of functions  $h: \mathcal{X} \to \mathbb{R}$  with an inner product  $\langle \cdot, \cdot \rangle: \mathcal{H} \times \mathcal{H} \to \mathbb{R}$  is a RKHS if there exists a symmetric positive definite function  $k: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$ , called a (reproducing) kernel, such that for all  $x \in \mathcal{X}$ , we have  $k(\cdot, x) \in \mathcal{H}$  and for all  $x \in \mathcal{X}$  and  $h \in \mathcal{H}$ , we have  $h(x) = \langle h(\cdot), k(\cdot, x) \rangle$ . We use the above  $k$  to denote the kernel associated with  $\mathcal{H}$ . Note that for any symmetric positive definite function  $k$ , we can naturally construct, from  $k$ , an RKHS  $\mathcal{H}$  whose kernel is exactly  $k$  [14].
When  $\mathcal{H}$  is a RKHS,  $g_{\pi_n,\lambda_n}$  in (12) can be computed for a number of convex loss functions  $\mathcal{L}$ . In particular, for the least squares loss  $\mathcal{L}(h(X),Y) = (h(X) - Y)^2$ , the resulting problem (12) is well known as the kernel ridge regression, and there exists a closed-form expression for both  $g_{\pi_n,\lambda_n}$  and  $g_{\pi ,\lambda_0}$ , as we discuss below.
Formally, the kernel ridge regression problem is given by
$$
g _ {\pi_ {n}, \lambda_ {n}} (x) := \underset {g \in \mathcal {H}} {\arg \min } \left\{\frac {1}{n} \sum_ {j = 1} ^ {n} (y _ {j} - g (x _ {j})) ^ {2} + \lambda_ {n} \| g \| _ {\mathcal {H}} ^ {2} \right\}
$$
where  $\lambda_{n} > 0$  is a regularization hyperparameter that may depend on the cardinality of the training data set. There is a closed-form formula for its solution, as stated below.
Proposition B.1. Let
$$
\boldsymbol {k} (\boldsymbol {x}, \boldsymbol {x}) = (k (x _ {i}, x _ {j})) _ {n \times n} \in \mathbb {R} ^ {n \times n}
$$
be the kernel Gram matrix and
$$
\boldsymbol {k} (x, \boldsymbol {x}) = \left(k (x, x _ {1}), \dots , k (x, x _ {n})\right) ^ {T}.
$$
Then the (unique) kernel ridge regression solution is given as
$$
g _ {\pi_ {n}, \lambda_ {n}} (x) = \boldsymbol {k} (x, \boldsymbol {x}) ^ {T} (\boldsymbol {k} (\boldsymbol {x}, \boldsymbol {x}) + \lambda_ {n} n \boldsymbol {I}) ^ {- 1} \boldsymbol {y}
$$
It is worth mentioning the linearity of kernel ridge regression: When we have two outputs  $\pmb{y} = (y_1, \dots, y_n)^T$  and  $\pmb{y}' = (y_1', \dots, y_n')^T$ , we have
$$
\begin{array}{l} \boldsymbol {k} (\boldsymbol {x}, \boldsymbol {x}) ^ {T} (\boldsymbol {k} (\boldsymbol {x}, \boldsymbol {x}) + \lambda_ {n} n \boldsymbol {I}) ^ {- 1} (\boldsymbol {y} + \boldsymbol {y} ^ {\prime}) \\ = \boldsymbol {k} (\boldsymbol {x}, \boldsymbol {x}) ^ {T} (\boldsymbol {k} (\boldsymbol {x}, \boldsymbol {x}) + \lambda_ {n} n \boldsymbol {I}) ^ {- 1} \boldsymbol {y} + \boldsymbol {k} (\boldsymbol {x}, \boldsymbol {x}) ^ {T} (\boldsymbol {k} (\boldsymbol {x}, \boldsymbol {x}) + \lambda_ {n} n \boldsymbol {I}) ^ {- 1} \boldsymbol {y} \\ \end{array}
$$
which means the solution to
$$
\underset {g \in \mathcal {H}} {\arg \min } \left\{\frac {1}{n} \sum_ {j = 1} ^ {n} \left(y _ {j} + y _ {j} ^ {\prime} - g \left(x _ {j}\right)\right) ^ {2} + \lambda_ {n} \| g \| _ {\mathcal {H}} ^ {2} \right\}
$$
is the summation of the solution to
$$
\underset {g \in \mathcal {H}} {\arg \min } \left\{\frac {1}{n} \sum_ {j = 1} ^ {n} \left(y _ {j} - g \left(x _ {j}\right)\right) ^ {2} + \lambda_ {n} \| g \| _ {\mathcal {H}} ^ {2} \right\}
$$
and the solution to
$$
\underset {g \in \mathcal {H}} {\arg \min } \left\{\frac {1}{n} \sum_ {j = 1} ^ {n} \left(y _ {j} ^ {\prime} - g (x _ {j})\right) ^ {2} + \lambda_ {n} \| g \| _ {\mathcal {H}} ^ {2} \right\}.
$$
Define  $\mathcal{O}_k:L^2 (\pi_X)\to L^2 (\pi_X)$  as the kernel integral operator
$$
\left(\mathcal {O} _ {k} g\right) (x) := \int_ {\mathcal {X}} k \left(x, x ^ {\prime}\right) g \left(x ^ {\prime}\right) \pi_ {X} \left(x ^ {\prime}\right) d x ^ {\prime}, x \in \mathcal {X}, g \in L ^ {2} \left(\pi_ {X}\right).
$$
where  $L^2 (\pi_X)\coloneqq \{f:\int_{\mathcal{X}}f(x)^2\pi_X(x)dx <   \infty \}$
[109] shows that  $\mathcal{O}_k$  is a compact and positive self-adjoint linear operator on  $L^2 (\pi_X)$ . Note that since  $L_{k}$  is positive:  $\mathcal{O}_k\geq 0$ , we have that  $\mathcal{O}_k + \lambda_0I$  is strictly positive for any  $\lambda_0 > 0$ :  $\mathcal{O}_k + \lambda_0I > 0$ , and thus its inverse operator  $(\mathcal{O}_k + \lambda_0I)^{-1}$  exists and is unique. Note that  $(\mathcal{O}_k + \lambda_0I)^{-1}$  is also a linear operator on  $L^2 (\pi_X)$ .
Next, consider the population risk minimization problem corresponding to  $g_{\pi_n,\lambda_n}$  as follows:
$$
g _ {\pi , \lambda_ {0}} := \underset {g \in \mathcal {H}} {\arg \min } \left\{\mathbb {E} _ {\pi} [ (Y - g (X)) ^ {2} ] + \lambda_ {0} \| g \| _ {\mathcal {H}} ^ {2} \right\}.
$$
It is easy to see that this problem is equivalent to
$$
g _ {\pi , \lambda_ {0}} := \underset {g \in \mathcal {H}} {\arg \min } \left\{\mathbb {E} _ {\pi} \left[ (g _ {\pi} ^ {*} (X) - g (X)) ^ {2} \right] + \lambda_ {0} \| g \| _ {\mathcal {H}} ^ {2} \right\}.
$$
where  $g_{\pi}^{*}(x) = \mathbb{E}_{(X,Y)\sim \pi}[Y|X = x]$  is the (ground-truth) regression function. This problem has the following explicit closed-form expression of the solution (a proof can be found in [27]):
Proposition B.2. The solution of  $g_{\pi, \lambda_0}$  is given as  $g_{\pi, \lambda_0} = (\mathcal{O}_k + \lambda_0 I)^{-1} \mathcal{O}_k g_{\pi}^*$ .
The linearity of kernel ridge regression also holds for the population-level solution:
$$
\begin{array}{l} (g + g ^ {\prime}) _ {\pi , \lambda_ {0}} = (\mathcal {O} _ {k} + \lambda_ {0} I) ^ {- 1} \mathcal {O} _ {k} (g + g ^ {\prime}) _ {\pi} ^ {*} \\ = \left(\mathcal {O} _ {k} + \lambda_ {0} I\right) ^ {- 1} \mathcal {O} _ {k} \left(g _ {\pi} ^ {*} + g _ {\pi} ^ {* *}\right) \\ = \left(\mathcal {O} _ {k} + \lambda_ {0} I\right) ^ {- 1} \mathcal {O} _ {k} g _ {\pi} ^ {*} + \left(\mathcal {O} _ {k} + \lambda_ {0} I\right) ^ {- 1} \mathcal {O} _ {k} g _ {\pi} ^ {*} \\ = g _ {\pi , \lambda_ {0}} + g _ {\pi , \lambda_ {0}} ^ {\prime} \\ \end{array}
$$
for any two functions  $g, g' \in L^2(\pi_X)$  since both  $(\mathcal{O}_k + \lambda_0 I)^{-1}$  and  $\mathcal{O}_k$  are linear operators on  $L^2(\pi_X)$ .
B.2 Asymptotic Normality of Kernel-Based Regression
In this section, we review existing results on the asymptotic normality of kernel-based regression, which is established using the influence function concept [25, 24, 50].
Let  $\mathcal{H}$  be a generic RKHS with the associated kernel  $k(x,x^{\prime})$ . Let  $\| \cdot \|_{\mathcal{H}}$  denote the norm on  $\mathcal{H}$ . Note that the feature map  $k_{x} = k(x,\cdot)$  is a function in  $\mathcal{H}$ . We consider the connection between the true regularized risk minimizer  $g_{\pi ,\lambda_0}$  and the empirical regularized risk minimizer  $g_{\pi_n,\lambda_n}$  introduced in (11) and (12) when  $\mathcal{H}$  is a RKHS.