SlowGuess/NeurIPS_2023_OCR huggingface dataset:
be the NTK Gram matrix evaluated on data  $\pmb{x}$ . For an arbitrary point  $x\in \mathbb{R}^d$ , we let  $\pmb {K}(x,\pmb {x})\in \mathbb{R}^{n}$  be the kernel value evaluated between the point  $x$  and data  $\pmb{x}$ , i.e.,
$$
\boldsymbol {K} (x, \boldsymbol {x}) := \left(K (x, x _ {1}), K (x, x _ {2}),..., K (x, x _ {n})\right) ^ {T} = J (x) J (\boldsymbol {x}) ^ {\top} \in \mathbb {R} ^ {1 \times n}.
$$
Training dynamics of the linearized network. Based on Assumption C.1, the training dynamics of the linearized network can be derived as follows.
We consider the regression problem with the following regularized empirical loss
$$
\hat {R} \left(f _ {\theta}\right) = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathcal {L} \left(f _ {\theta} \left(x _ {i}\right), y _ {i}\right) + \lambda_ {n} \| \theta - \theta^ {b} \| _ {2} ^ {2}. \tag {17}
$$
where  $\theta^b = \theta(0)$  is the initialization of the network. In the following, we use  $f_{\theta(t)}$  to represent the network with parameter  $\theta(t)$  that evolves with the time  $t$ . Let  $\eta$  be the learning rate. Suppose the network parameter is trained via continuous-time gradient flow. Then the evolution of the parameters  $\theta(t)$  and  $f_{\theta(t)}$  can be written as
$$
\frac {d \theta (t)}{d t} = - \eta \left(\frac {1}{n} \nabla_ {\theta} f _ {\theta (t)} (\boldsymbol {x}) ^ {\top} \nabla_ {f _ {\theta (t)} (\boldsymbol {x})} \mathcal {L} + 2 \lambda_ {n} (\theta (t) - \theta (0))\right) \tag {18}
$$
$$
\frac {d f _ {\theta (t)} (\boldsymbol {x})}{d t} = \nabla_ {\theta} f _ {\theta (t)} (\boldsymbol {x}) \frac {d \theta (t)}{d t} \tag {19}
$$
where  $\theta (t)$  and  $f_{\theta (t)}(x)$  are the network parameters and network output at time  $t$
Under Assumption C.1, we have that
$$
f _ {\theta (t)} (x) = f _ {\theta (0)} (x) + J (x) (\theta (t) - \theta (0)),
$$
$$
\nabla_ {\theta} f _ {\theta (t)} (x) = J (x).
$$
Suppose the loss function is an MSE loss, i.e.,  $\mathcal{L}(\tilde{y}, y) = (\tilde{y} - y)^2$ . Then
$$
\nabla_ {f _ {\theta (t)} (\boldsymbol {x})} \mathcal {L} \left(f _ {\theta (t)} (\boldsymbol {x}), \boldsymbol {y}\right) = 2 \left(f _ {\theta (t)} (\boldsymbol {x}) - \boldsymbol {y}\right)
$$
Therefore, under Assumption C.1, (18) becomes
$$
\begin{array}{l} \frac {d \theta (t)}{d t} = - \eta \left(\frac {1}{n} J (\boldsymbol {x}) ^ {\top} \nabla_ {f _ {\theta (t)} (\boldsymbol {x})} \mathcal {L} + 2 \lambda_ {n} (\theta (t) - \theta (0))\right) \\ = - \eta \left(\frac {2}{n} J (\boldsymbol {x}) ^ {\top} \left(f _ {\theta (0)} (\boldsymbol {x}) + J (\boldsymbol {x}) (\theta (t) - \theta (0)) - \boldsymbol {y}\right) + 2 \lambda_ {n} (\theta (t) - \theta (0))\right) \\ = - 2 \eta \left(\frac {1}{n} J (\boldsymbol {x}) ^ {\top} \left(f _ {\theta (0)} (\boldsymbol {x}) - J (\boldsymbol {x}) \theta (0) - \boldsymbol {y}\right) - \lambda_ {n} \theta (0) + \left(\frac {1}{n} J (\boldsymbol {x}) ^ {\top} J (\boldsymbol {x}) + \lambda_ {n} \boldsymbol {I}\right) \theta (t)\right) \\ \end{array}
$$
Note that  $\frac{1}{n} J(\pmb{x})^\top (f_{\theta(0)}(\pmb{x}) - J(\pmb{x})\theta(0) - \pmb{y}) - \lambda_n\theta(0)$  and  $\left(\frac{1}{n} J(\pmb{x})^\top J(\pmb{x}) + \lambda_n\pmb{I}\right)$  are both independent of  $t$ . Solving this ordinary differential equation, we obtain
$$
\begin{array}{l} \theta (t) = \theta (0) - \frac {1}{n} J (\boldsymbol {x}) ^ {\top} \left(\frac {1}{n} J (\boldsymbol {x}) ^ {\top} J (\boldsymbol {x}) + \lambda_ {n} I\right) ^ {- 1} \\ \left(\boldsymbol {I} - \exp \left(- 2 \eta t \left(\frac {1}{n} J (\boldsymbol {x}) ^ {\top} J (\boldsymbol {x}) + \lambda_ {n} \boldsymbol {I}\right)\right)\right) \left(f _ {\theta (0)} (\boldsymbol {x}) - \boldsymbol {y}\right) \\ \end{array}
$$
Hence the network output at time  $t$  becomes
$$
\begin{array}{l} f _ {\theta (t)} (x) - f _ {\theta (0)} (x) \\ = J (x) (\theta (t) - \theta (0)) \\ = - \frac {1}{n} J (x) J (\boldsymbol {x}) ^ {\top} \left(\frac {1}{n} J (\boldsymbol {x}) ^ {\top} J (\boldsymbol {x}) + \lambda_ {n} \boldsymbol {I}\right) ^ {- 1} \\ \left(\boldsymbol {I} - \exp \left(- 2 \eta t \left(\frac {1}{n} J (\boldsymbol {x}) ^ {\top} J (\boldsymbol {x}) + \lambda_ {n} \boldsymbol {I}\right)\right)\right) \left(f _ {\theta (0)} (\boldsymbol {x}) - \boldsymbol {y}\right) \\ = - \frac {1}{n} \boldsymbol {K} (x, \boldsymbol {x}) \left(\frac {1}{n} \boldsymbol {K} (\boldsymbol {x}, \boldsymbol {x}) + \lambda_ {n} \boldsymbol {I}\right) ^ {- 1} \left(\boldsymbol {I} - \exp \left(- 2 \eta t \left(\frac {1}{n} \boldsymbol {K} (\boldsymbol {x}, \boldsymbol {x}) + \lambda_ {n} \boldsymbol {I}\right)\right)\right) \left(f _ {\theta (0)} (\boldsymbol {x}) - \boldsymbol {y}\right) \\ \end{array}
$$
where the last inequality used the notation  $\pmb {K}(\pmb {x},\pmb {x}) = J(\pmb {x})J(\pmb {x})^{\top}$  and  $\pmb {K}(x_0,\pmb {x}) = J(\pmb {x})J(\pmb {x})^{\top}$
Therefore, with sufficient time of training  $(t\to \infty)$ , the final trained network is
$$
\begin{array}{l} f _ {\theta (\infty)} (x) = \lim  _ {t \rightarrow \infty} f _ {\theta (t)} (x) \\ = f _ {\theta (0)} (x) - \frac {1}{n} \boldsymbol {K} (x, \boldsymbol {x}) \left(\frac {1}{n} \boldsymbol {K} (\boldsymbol {x}, \boldsymbol {x}) + \lambda_ {n} \boldsymbol {I}\right) ^ {- 1} \left(f _ {\theta (0)} (\boldsymbol {x}) - \boldsymbol {y}\right) \\ = f _ {\theta (0)} (x) + \boldsymbol {K} (x, \boldsymbol {x}) \left(\boldsymbol {K} (\boldsymbol {x}, \boldsymbol {x}) + n \lambda_ {n} \boldsymbol {I}\right) ^ {- 1} \left(\boldsymbol {y} - f _ {\theta (0)} (\boldsymbol {x})\right) \\ = s _ {\theta^ {b}} (x) + \boldsymbol {K} (x, \boldsymbol {x}) (\boldsymbol {K} (\boldsymbol {x}, \boldsymbol {x}) + n \lambda_ {n} \boldsymbol {I}) ^ {- 1} (\boldsymbol {y} - s _ {\theta^ {b}} (\boldsymbol {x})) \\ \end{array}
$$
where in the last inequality, we use  $s_{\theta^b}$  to represent the network initialized with the parameter  $\theta^b$  as in the main paper. Summarizing the above discussion, we conclude that
Assumption C.2. Suppose that the network training is specified as follows:
1. The network adopts the NTK parametrization and its parameters are randomly initialized using He initialization.
2. The network is sufficiently (infinitely) wide so that the linearized neural network assumption (Assumption C.1) holds.
3. The network is trained using the loss function in (1) via continuous-time gradient flow by feeding the entire training data and using sufficient training time  $(t\to \infty)$ .
Proposition C.3. Suppose that Assumption C.2 holds. Then the final trained network is given by
$$
\hat {h} _ {n, \theta^ {b}} (x) = s _ {\theta^ {b}} (x) + K (x, \boldsymbol {x}) \left(\boldsymbol {K} (\boldsymbol {x}, \boldsymbol {x}) + \lambda_ {n} n \boldsymbol {I}\right) ^ {- 1} (\boldsymbol {y} - s _ {\theta^ {b}} (\boldsymbol {x})). \tag {20}
$$
For the rest part of Proposition 3.1, please refer to Lemma C.4 below.
Shifted kernel ridge regression. On the other hand, we build a shifted kernel ridge regression based on the NTK as follows.
Let  $\tilde{\mathcal{H}}$  be the reproducing kernel Hilbert space constructed from the kernel function  $K(x,x^{\prime})$  (the population NTK); See Appendix B. Consider the following kernel ridge regression problem on the RKHS  $\tilde{\mathcal{H}}$ :
$$
s _ {\theta^ {b}} + \underset {g \in \mathcal {H}} {\arg \min } \frac {1}{n} \sum_ {i = 1} ^ {n} \left(y _ {i} - s _ {\theta^ {b}} \left(x _ {i}\right) - g \left(x _ {i}\right)\right) ^ {2} + \lambda_ {n} \| g \| _ {\mathcal {H}} ^ {2} \tag {21}
$$
where  $\lambda_{n}$  is a regularization hyper-parameter and  $s_{\theta^b}$  is a known given function.
Lemma C.4. (20) is the solution to the following kernel ridge regression problem
$$
s _ {\theta^ {b}} + \underset {g \in \mathcal {H}} {\arg \min } \frac {1}{n} \sum_ {i = 1} ^ {n} \left(y _ {i} - s _ {\theta^ {b}} \left(x _ {i}\right) - g \left(x _ {i}\right)\right) ^ {2} + \lambda_ {n} \| g \| _ {\mathcal {H}} ^ {2}. \tag {22}
$$
Proof of Lemma C.4. We first let  $\tilde{y}_i = y_i - s_{\theta^b}(x_i)$  be the shifted label. Now consider the kernel ridge regression problem
$$
\underset {g \in \mathcal {H}} {\arg \min} \frac {1}{n} \sum_ {i = 1} ^ {n} (\tilde {y} _ {i} - g (x _ {i})) ^ {2} + \lambda_ {n} \| g \| _ {\mathcal {\bar {H}}} ^ {2}.
$$
Standard theory in kernel ridge regression (Proposition B.1) shows that the closed-form solution to the above problem is given by:
$$
\pmb {K} (\pmb {x}, \pmb {x}) ^ {T} (\pmb {K} (\pmb {x}, \pmb {x}) + \lambda_ {n} n \pmb {I}) ^ {- 1} \tilde {\pmb {y}} = \pmb {K} (\pmb {x}, \pmb {x}) ^ {T} (\pmb {K} (\pmb {x}, \pmb {x}) + \lambda_ {n} n \pmb {I}) ^ {- 1} (\pmb {y} - s _ {\theta^ {b}} (\pmb {x}))
$$
as  $K$  is the kernel function of  $\bar{\mathcal{H}}$ . This equation corresponds to the definition of  $\hat{h}_{n,\theta^b}(x) - s_{\theta^b}(x)$  in (20).
From Proposition C.3 and Lemma C.4, we immediately conclude that
Appendix D Additional Remarks
We make the following additional remarks to explain some of the details in the main paper:
1. Regarding Proposition 3.1 (Proposition C.3) in Section 3. Proposition 3.1 is the theoretical foundation of this paper to develop our framework for efficient uncertainty quantification and reduction for neural networks. We recognize the limitation of this proposition: The linearized neural network assumption (Assumption C.1) must hold to guarantee that the exact equality (2) holds in Proposition 3.1. In reality, the network can never be infinitely wide, and thus an additional error will appear in (2). Unfortunately, with finite network width, this error might be roughly estimated up to a certain order but still involves some unknown constants that hide beneath the data and network [34, 7, 80], which is extremely difficult to quantify precisely in practice. This work does not deal with such an error from finite network width. Instead, we assume that the linearized neural network assumption readily holds as the starting point for developing our uncertainty quantification framework. Equivalently, one may view our work as uncertainty quantification for linearized neural networks.
2. Regarding the regularization parameters in Section 3. We introduce the regularization parameters  $\lambda_{n} > 0$  with the regularization term centering at the randomly initialized network parameter in (1) throughout this paper. This form of regularized loss is also adopted in [119, 117]. It has some advantages: 1) A common assumption in NTK theory is to assume that the smallest eigenvalue of the NTK Gram matrix is bounded away from zero [34, 7, 80]. With  $\lambda_{n} > 0$  is introduced, there is an additional term  $\lambda_{n} n I$  in the NTK Gram matrix so that this assumption is always valid naturally. 2) It helps to stabilize the inversion of the NTK Gram matrix if any computation is needed. 3) It can be naturally linked to the well-known kernel ridge regression, which we leverage in this work to develop uncertainty quantification approaches.
On the other hand, we note that with  $\lambda_0 = \lim_{n\to \infty}\lambda_n > 0$ , the "best" predictor  $h^*$  in (6) given in our framework is not exactly the same as the ground-truth regression function  $g_{\pi}^{*} \coloneqq \mathbb{E}_{(X,Y)\sim \pi}[Y|X = x]$ . In fact, Proposition B.2 shows that  $h^* = \bar{s} + (\mathcal{O}_K + \lambda_0I)^{-1}\mathcal{O}_K(g_{\pi}^* - \bar{s}) = (\mathcal{O}_K + \lambda_0I)^{-1}\mathcal{O}_K(g_{\pi}^*) \neq g_{\pi}^*$ . However, previous work [109, 106, 112] shows that with some conditions,  $h^* - g_{\pi}^* \rightarrow 0$  in some metrics when  $\lambda_0 \rightarrow 0$ . Therefore, we may use a very small limit value  $\lambda_0$  in practice to make the error  $h^* - g_{\pi}^*$  negligible, as we did in our experiments.
3. Regarding naive resampling approaches such as "bootstrap on a deep ensemble" mentioned in Section 3.1. We show that bootstrap on the deep ensemble estimator  $\hat{h}_n^m (x)$  face computational challenges in constructing confidence intervals. Since  $\hat{h}_n^m (x)$  involves two randomnesses, data variability and procedural variability, we need to bring up two asymptotic normalities to handle them. One approach is first to consider procedural variability given data and then to consider data variability, as described as follows. Another approach is first to consider data variability given procedure and then to consider procedural variability. This will encounter similar challenges as the first approach, so we only discuss the first approach here.
We intuitively present the technical discussions to explain the computational challenges. First, conditional on the training data of size  $n$ , by the central limit theorem, we have  $\sqrt{m} (\hat{h}_n^m (x) - \hat{h}_n^* (x)) \stackrel {d}{\approx} \mathcal{N}(0,\xi_n^2)$  as  $m \to \infty$  where  $\stackrel {d}{\approx}$  represents "approximately equal in distribution" and  $\xi_{n}^{2} = \operatorname{Var}_{P_{\theta^{b}}}(\hat{h}_{n,\theta^{b}}(x))$  which depends on the training data. Second, we assume that the central limit theorem holds for  $\hat{h}_n^* (x)$  (which is rigorously justified in Section 3.3):  $\sqrt{n} (\hat{h}_n^* (x) - h^* (x)) \Rightarrow \mathcal{N}(0,\zeta^2)$  as  $n \to \infty$ . Moreover, we assume for simplicity that we can add up the above two convergences in distribution to obtain
$$
\sqrt {n} \left(\hat {h} _ {n} ^ {m} (x) - h ^ {*} (x)\right) \stackrel {d} {\approx} \mathcal {N} \left(0, \zeta^ {2} + \frac {n}{m} \xi_ {n} ^ {2}\right).
$$
Note that  $\hat{h}_n^m (x) - \hat{h}_n^* (x)$  and  $\hat{h}_n^* (x) - h^* (x)$  are clearly dependent, both depending on the training data of size  $n$ . Therefore, the statistical correctness of the above summation of two convergences in distribution needs to be justified, e.g., using techniques in [48, 75, 77]. However, even if we assume that all the technical parts can be addressed and the asymptotic normality of  $\hat{h}_n^m (x)$  is established, computation issues still arise in the "bootstrap on a deep ensemble" approach: 1) " $\frac{n}{m}\xi_n^{2}$ " is meaningful in the limit if we have the limit  $\frac{n}{m}\xi_n^2\to \alpha$ . This typically requires the ensemble size  $m$  to be large, at least in the order  $\omega (1)$ , which increases the computational cost of the deep ensemble predictor. However, this is not the only computationally demanding part. 2) The estimation of the asymptotic variance  $\zeta^2 +\frac{n}{m}\xi_n^{2}$ " requires additional computational effort. When using resampling techniques such as the bootstrap, we need to resample  $R$  times to obtain  $(\hat{h}_n^m (x))^{*j}$  where  $j\in [R]$ . This implies that we need  $Rm$  network training times in total, and according to Point 1), it is equivalent to at least  $R\times \omega (1)$  network training times, which is extremely computationally demanding. Therefore, naive use of the "bootstrap on a deep ensemble" approach is barely feasible in neural networks.
Appendix E Proofs
In this section, we provide technical proofs of the results in the paper.
Proof of Theorem 3.2. In terms of estimation bias, we note that
$$
\mathbb {E} [ \hat {h} _ {n} ^ {m} (x) | \mathcal {D} ] = \frac {1}{m} \sum_ {i = 1} ^ {m} \mathbb {E} [ \hat {h} _ {n, \theta_ {i} ^ {b}} (x) | \mathcal {D} ] = \mathbb {E} [ \hat {h} _ {n, \theta^ {b}} (x) | \mathcal {D} ] = \hat {h} _ {n} ^ {*} (x).
$$
as shown in (3). This implies that the deep ensemble predictor and the single network predictor have the same conditional procedural mean given the data. In terms of variance, note that  $\hat{h}_{n,\theta_1^b}(x),\dots,\hat{h}_{n,\theta_m^b}(x)$ , derived in (2), are conditionally independent given  $\mathcal{D}$ . Therefore the deep ensemble predictor gives
$$
\operatorname {V a r} (\hat {h} _ {n} ^ {m} (x) | \mathcal {D}) = \frac {1}{m ^ {2}} \sum_ {i = 1} ^ {m} \operatorname {V a r} (\hat {h} _ {n, \theta_ {i} ^ {b}} (x) | \mathcal {D}) = \frac {1}{m} \operatorname {V a r} (\hat {h} _ {n, \theta^ {b}} (x) | \mathcal {D})
$$
while the single model prediction gives  $\operatorname{Var}(\hat{h}_{n,\theta^b}(x)|\mathcal{D})$ . Now using conditioning, we have
$$
\begin{array}{l} \mathrm {V a r} (\hat {h} _ {n} ^ {m} (x)) \\ = \operatorname {V a r} \left(\mathbb {E} \left[ \hat {h} _ {n} ^ {m} (x) | \mathcal {D} \right]\right) + \mathbb {E} \left[ \operatorname {V a r} \left(\hat {h} _ {n} ^ {m} (x) | \mathcal {D}\right) \right] \\ = \operatorname {V a r} \left(\hat {h} _ {n} ^ {*} (x)\right) + \frac {1}{m} \mathbb {E} \left[ \operatorname {V a r} \left(\hat {h} _ {n, \theta^ {b}} (x) | \mathcal {D}\right) \right] \\ \leq \operatorname {V a r} \left(\hat {h} _ {n} ^ {*} (x)\right) + \mathbb {E} \left[ \operatorname {V a r} \left(\hat {h} _ {n, \theta^ {b}} (x) | \mathcal {D}\right) \right] \\ = \operatorname {V a r} \left(\hat {h} _ {n, \theta^ {b}} (x)\right) \\ \end{array}
$$
Proof of Theorem 3.3. Recall that  $\hat{\phi}_{n,\theta^b}^\prime (\cdot)$  is obtained by training an auxiliary neural network with data  $\{(x_1,\bar{s} (x_1)),(x_2,\bar{s} (x_2)),\dots,(x_n,\bar{s} (x_n))\}$  and the initialization parameters  $\theta^b$ . Then Proposition 3.1 immediately implies that
$$
\hat {\phi} _ {n, \theta^ {b}} ^ {\prime} (x) = s _ {\theta^ {b}} (x) + K (x, \pmb {x}) ^ {T} (K (\pmb {x}, \pmb {x}) + \lambda_ {n} n I) ^ {- 1} (\bar {s} (\pmb {x}) - s _ {\theta^ {b}} (\pmb {x})).
$$
Note that this auxiliary network starts from the same initialization  $\theta^b$  as in  $\hat{h}_{n,\theta^b}(x)$ . Then we have
$$
\begin{array}{l} \hat {\phi} _ {n, \theta^ {b}} (x) = \hat {\phi} _ {n, \theta^ {b}} ^ {\prime} (x) - \bar {s} (x) \\ = s _ {\theta^ {b}} (x) - \bar {s} (x) + \boldsymbol {K} (x, \boldsymbol {x}) ^ {T} (\boldsymbol {K} (\boldsymbol {x}, \boldsymbol {x}) + \lambda_ {n} n \boldsymbol {I}) ^ {- 1} (\bar {s} (\boldsymbol {x}) - s _ {\theta^ {b}} (\boldsymbol {x})) \\ = \hat {h} _ {n, \theta^ {b}} (x) - \hat {h} _ {n} ^ {*} (x) \\ \end{array}
$$
by (2) and (3). Hence,
$$
\hat {h} _ {n} ^ {*} (x) = \hat {h} _ {n, \theta^ {b}} (x) - (\hat {\phi} _ {n, \theta^ {b}} ^ {\prime} (x) - \bar {s} (x)) = \hat {h} _ {n, \theta^ {b}} (x) - \hat {\phi} _ {n, \theta^ {b}} (x)
$$
Proof of Theorem 3.5. Theorem 3.3 implies that
$$
\hat {h} _ {n, \theta^ {b}} (x) - \hat {\phi} _ {n, \theta^ {b}} (x) = \hat {h} _ {n} ^ {*} (x)
$$
which is the solution to the following problem
$$
\hat {h} _ {n} ^ {*} (\cdot) = \bar {s} (\cdot) + \min _ {g \in \mathcal {H}} \frac {1}{n} \sum_ {i = 1} ^ {n} [ (y _ {i} - \bar {s} (x _ {i}) - g (x _ {i})) ^ {2} ] + \lambda_ {n} \| g \| _ {\mathcal {H}} ^ {2}
$$
Therefore, its corresponding population risk minimization problem is:
$$
h ^ {*} (\cdot) = \bar {s} (\cdot) + \min _ {g \in \bar {\mathcal {H}}} \mathbb {E} _ {\pi} [ (Y - \bar {s} (X) - g (X)) ^ {2} ] + \lambda_ {0} \| g \| _ {\mathcal {H}} ^ {2}
$$
Applying Proposition 3.4 to  $\hat{h}_n^*$  and  $h^*$ , we have that
$$
\sqrt {n} \left((\hat {h} _ {n} ^ {*} (x) - \bar {s} (x)) - (h ^ {*} (x) - \bar {s} (x))\right) \Rightarrow \mathcal {N} (0, \sigma^ {2} (x))
$$
$$
\sqrt {n} \left(\hat {h} _ {n, \theta^ {b}} (x) - \hat {\phi} _ {n, \theta^ {b}} (x) - h ^ {*} (x)\right) \Rightarrow \mathcal {N} (0, \sigma^ {2} (x))
$$
This shows that asymptotically as  $n\to \infty$  we have
$$
\mathbb {P} \left(- q _ {1 - \frac {\alpha}{2}} \leq \frac {\hat {h} _ {n , \theta^ {b}} (x) - \hat {\phi} _ {n , \theta^ {b}} (x) - h ^ {*} (x)}{\sigma (x) / \sqrt {n}} \leq q _ {1 - \frac {\alpha}{2}}\right)\rightarrow 1 - \alpha
$$
where  $q_{\alpha}$  is the  $\alpha$ -quantile of standard Gaussian distribution  $\mathcal{N}(0,1)$ . Hence
$$
\left[ \hat {h} _ {n, \theta^ {b}} (x) - \hat {\phi} _ {n, \theta^ {b}} (x) - \frac {\sigma (x)}{\sqrt {n}} q _ {1 - \frac {\alpha}{2}}, \hat {h} _ {n, \theta^ {b}} (x) - \hat {\phi} _ {n, \theta^ {b}} (x) + \frac {\sigma (x)}{\sqrt {n}} q _ {1 - \frac {\alpha}{2}} \right]
$$
is an asymptotically exact  $(1 - \alpha)$ -level confidence interval of  $h^*(x)$ .
Proof of Theorem 3.6. Note that the statistics
$$
\hat {h} _ {n ^ {\prime}, \theta^ {b}} ^ {j} (x) - \hat {\phi} _ {n ^ {\prime}, \theta^ {b}} ^ {j} (x)
$$
from the  $j$ -th batch is i.i.d. for  $j \in [m']$ . Moreover, by Theorem 3.5, we have the asymptotic normality:
$$
\sqrt {n ^ {\prime}} \left(\hat {h} _ {n ^ {\prime}, \theta^ {b}} (x) - \hat {\phi} _ {n ^ {\prime}, \theta^ {b}} (x) - h ^ {*} (x)\right) \Rightarrow \mathcal {N} (0, \sigma^ {2} (x)),
$$
as  $n\to \infty$  meaning  $n^{\prime} = n / m^{\prime}\rightarrow \infty$ . Therefore by the property of Gaussian distribution and the principle of batching, we have
$$
\frac {\sqrt {n ^ {\prime}}}{\sigma (x)} \left(\psi_ {B} (x) - h ^ {*} (x)\right) \Rightarrow \frac {1}{m ^ {\prime}} \sum_ {i = 1} ^ {m ^ {\prime}} Z _ {i}
$$
$$
\frac {\psi_ {B} (x) - h ^ {*} (x)}{S _ {B} (x) / \sqrt {m ^ {\prime}}} \Rightarrow \frac {\frac {1}{m ^ {\prime}} \sum_ {i = 1} ^ {m ^ {\prime}} Z _ {i}}{\sqrt {\frac {1}{m ^ {\prime} (m ^ {\prime} - 1)} \sum_ {j = 1} ^ {m ^ {\prime}} \left(Z _ {j} - \frac {1}{m ^ {\prime}} \sum_ {i = 1} ^ {m ^ {\prime}} Z _ {i}\right) ^ {2}}}
$$
for i.i.d.  $\mathcal{N}(0,1)$  random variables  $Z_{1},\ldots,Z_{m'}$ , where we use the continuous mapping theorem to deduce the weak convergence. Note that
$$
\frac {\frac {1}{m ^ {\prime}} \sum_ {i = 1} ^ {m ^ {\prime}} Z _ {i}}{\sqrt {\frac {1}{m ^ {\prime} (m ^ {\prime} - 1)} \sum_ {j = 1} ^ {m ^ {\prime}} \left(Z _ {j} - \frac {1}{m ^ {\prime}} \sum_ {i = 1} ^ {m ^ {\prime}} Z _ {i}\right) ^ {2}}} \stackrel {\mathrm {d}} {=} t _ {m ^ {\prime} - 1}
$$
Hence, asymptotically as  $n\to \infty$  we have
$$
\mathbb {P} \left(- q _ {1 - \frac {\alpha}{2}} \leq \frac {\psi_ {B} (x) - h ^ {*} (x)}{S _ {B} (x) / \sqrt {m ^ {\prime}}} \leq q _ {1 - \frac {\alpha}{2}}\right)\rightarrow 1 - \alpha
$$
where  $q_{\alpha}$  is the  $\alpha$ -quantile of the t distribution  $t_{m' - 1}$  with degree of freedom  $m' - 1$ . Hence
$$
\left[ \psi_ {B} (x) - \frac {S _ {B} (x)}{\sqrt {m ^ {\prime}}} q _ {1 - \frac {\alpha}{2}}, \psi_ {B} (x) + \frac {S _ {B} (x)}{\sqrt {m ^ {\prime}}} q _ {1 - \frac {\alpha}{2}} \right]
$$
Proof of Theorem 3.7. Note that for  $j \in [R]$ , the statistics
$$
\hat {h} _ {n, \theta^ {b}} ^ {* j} (x) - \hat {\phi} _ {n, \theta^ {b}} ^ {* j} (x)
$$
from the  $j$ -th bootstrap replication are i.i.d. conditional on the dataset  $\mathcal{D}$ . By Theorem 3.5, we have the asymptotic normality:
$$
\sqrt {n} \left(\hat {h} _ {n, \theta^ {b}} (x) - \hat {\phi} _ {n, \theta^ {b}} (x) - h ^ {*} (x)\right) \Rightarrow \mathcal {N} (0, \sigma^ {2} (x)),
$$
as  $n\to \infty$  . By Theorem 2 in [24], we have the following asymptotic normality: For  $j\in [R]$
$$
\sqrt {n} \left(\hat {h} _ {n, \theta^ {b}} ^ {* j} (x) - \hat {\phi} _ {n, \theta^ {b}} ^ {* j} (x) - \left(\hat {h} _ {n, \theta^ {b}} (x) - \hat {\phi} _ {n, \theta^ {b}} (x)\right)\right) \Rightarrow \mathcal {N} (0, \sigma^ {2} (x)),
$$
as  $n\to \infty$  conditional on the training data  $\mathcal{D}$ . Therefore Assumption 1 in [75] is satisfies and thus Theorem 1 in [75] holds, showing that asymptotically as  $n\rightarrow \infty$ , we have
$$
\mathbb {P} \left(- q _ {1 - \frac {\alpha}{2}} \leq \frac {\psi_ {C} (x) - h ^ {*} (x)}{S _ {C} (x)} \leq q _ {1 - \frac {\alpha}{2}}\right)\rightarrow 1 - \alpha
$$
where  $q_{\alpha}$  is the  $\alpha$ -quantile of the t distribution  $t_R$  with degree of freedom  $R$ . Hence
$$
\left[ \psi_ {C} (x) - S _ {C} (x) q _ {1 - \frac {\alpha}{2}}, \psi_ {C} (x) + S _ {C} (x) q _ {1 - \frac {\alpha}{2}} \right]
$$
Proof of Theorem B.6. Recall that  $\Phi$  is the feature map associated with the kernel  $k$  of the RKHS  $\mathcal{H}$ . We apply the influence function formula in [25] (see also [31, 50]) to obtain the infinitesimal jackknife:
$$
I F (z; T, \pi_ {n}) = - S _ {\pi_ {n}} ^ {- 1} (2 \lambda_ {0} g _ {\pi_ {n}, \lambda_ {0}}) + \mathcal {L} ^ {\prime} (z _ {y} - g _ {\pi_ {n}, \lambda_ {0}} (z _ {x})) S _ {\pi_ {n}} ^ {- 1} \Phi (z _ {x})
$$
where  $S_P: \mathcal{H} \to \mathcal{H}$  is defined by
$$
S _ {P} (f) = 2 \lambda_ {0} f + \mathbb {E} _ {P} \left[ \mathcal {L} ^ {\prime \prime} \left(Y, g _ {P, \lambda_ {0}} (X)\right) f (X) \Phi (X) \right].
$$
This can be seen from (14) by setting  $P = \pi_{n}$  and using the closed-form solution of the kernel ridge regression.