SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Finally, we provide additional remarks on why  $\hat{h}_n^* (x)$  cannot be computed easily except using our Algorithm 1. First, the following two candidate approaches encounter computational issues: 1) One may use deep ensemble with sufficiently many networks in the ensemble. However, this approach is time-consuming as  $m$  networks in the ensemble mean  $m$ -fold training times.  $m$  is typically as small as five in practice [73] so it cannot approximate  $\hat{h}_n^* (x)$  well. 2) One may use the closed-
form expression of  $\hat{h}_n^*$  in (3), which requires computing the NTK  $K(x,x')$  and the inversion of the NTK Gram matrix  $\pmb{K}(\pmb{x},\pmb{x})$ .  $K(x,x')$  is recursively defined and does not have a simple form for computation, which might be addressed by approximating it with the empirical NTK  $K_{\theta}(x,x')$  numerically (See Appendix C). However, the inversion of the NTK Gram matrix gives rise to a more serious computational issue: the dimension of the NTK Gram matrix is large on large datasets, making the matrix inversion very time-consuming. Another approach that seems plausible is that: 3) One may initialize one network that is equivalent to  $\bar{s}(x)$  and then train it. However, this approach cannot obtain  $\hat{h}_n^*(x)$  based on Proposition 3.1 because Proposition 3.1 requires random initialization. If one starts from a deterministic network initialization such as a zero-constant network, then the NTK theory underpinning the linearized training behavior of over-parameterized neural networks breaks down and the resulting network cannot be described by Proposition 3.1.
3.3 Constructing Confidence Intervals from PNC Predictors
We construct confidence intervals for  $h^*(x)$  leveraging our PNC predictor in Algorithm 1. To handle data variability, two lines of works borrowed from classical statistics may be considered. First is an analytical approach using the delta method for asymptotic normality, which involves computing the influence function [51, 38] that acts as the functional gradient of the predictor with respect to the data distribution. It was introduced in modern machine learning for understanding a training point's effect on a model's prediction [72, 71, 13]. The second is to use resampling [37, 30, 102], such as the bootstrap or jackknife, to avoid explicit variance computation. The classical resampling method requires sufficiently large resample replications and thus incurs demanding resampling effort. For instance, the jackknife approach requires the number of training times to be the same as the training data size, which is very time-consuming and barely feasible for neural networks. Standard bootstrap requires a sufficient number of resampling and retraining to produce accurate resample quantiles. Given these computational bottlenecks, we consider utilizing light-computation resampling alternatives, including batching [47, 99, 100] and the so-called cheap bootstrap method [75, 76], which allows valid confidence interval construction using as few as two model repetitions.
Large-sample asymptotics of the PNC predictor. To derive our intervals, we first gain understanding on the large-sample properties of the PNC predictor. Proposition 3.1 shows that  $\hat{h}_n^* (x)$  in (3) is the solution to the following empirical risk minimization problem:
$$
\hat {h} _ {n} ^ {*} (\cdot) = \bar {s} (\cdot) + \min  _ {g \in \mathcal {H}} \frac {1}{n} \sum_ {i = 1} ^ {n} \left[ \left(y _ {i} - \bar {s} \left(x _ {i}\right) - g \left(x _ {i}\right)\right) ^ {2} \right] + \lambda_ {n} \| g \| _ {\mathcal {H}} ^ {2}. \tag {5}
$$
Its corresponding population risk minimization problem (i.e., removing the data variability) is:
$$
h ^ {*} (\cdot) = \bar {s} (\cdot) + \min  _ {g \in \mathcal {H}} \mathbb {E} _ {\pi} [ (Y - \bar {s} (X) - g (X)) ^ {2} ] + \lambda_ {0} \| g \| _ {\mathcal {H}} ^ {2} \tag {6}
$$
where  $\lambda_0 = \lim_{n\to \infty}\lambda_n$ . To study the difference between the empirical and population risk minimization problems of kernel ridge regression, we introduce the following established result on the asymptotic normality of kernel ridge regression (See Appendix B for details):
Proposition 3.4 (Asymptotic normality of kernel ridge regression [50]). Let  $\mathcal{H}$  be a generic RKHS. Suppose that Assumptions B.3 and B.4 hold. Let  $g_{P,\lambda}$  be the solution to the following problem:  $g_{P,\lambda} := \min_{g \in \mathcal{H}} \mathbb{E}_P[(Y - g(X))^2] + \lambda \| g\|_{\mathcal{H}}^2$  where  $P = \pi_n$  or  $\pi$ . Then
$$
\sqrt {n} \left(g _ {\pi_ {n}, \lambda_ {n}} - g _ {\pi , \lambda_ {0}}\right) \Rightarrow \mathbb {G} i n \bar {H}
$$
where  $\mathbb{G}$  is a zero-mean Gaussian process and  $\Rightarrow$  represents "converges weakly". Moreover, at point  $x$ ,  $\sqrt{n} (g_{\pi_n,\lambda_n}(x) - g_{\pi ,\lambda_0}(x))\Rightarrow \mathcal{N}(0,\xi^2 (x))$  where  $\xi^2 (x) = \int_{z\in \mathcal{X}\times \mathcal{Y}}IF^2 (z;g_{P,\lambda_0},\pi)(x)d\pi (z)$  and  $IF$  is the influence function of statistical functional  $g_{P,\lambda_0}$ .
Next, we apply the above proposition to our problems about  $\hat{h}_n^*$ . Let  $T_{1}(P)(x)$  be the solution of the following problem  $\min_{g\in \mathcal{H}}\mathbb{E}_{P}[(Y - \bar{s} (X) - g(X))^{2}] + \lambda_{0}\| g\|_{\mathcal{H}}^{2}$  that is evaluated at a point  $x$ . Then we have the following large-sample asymptotic of the PNC predictor, providing the theoretical foundation for our subsequent interval construction approaches.
Theorem 3.5 (Large-sample asymptotics of the PNC predictor). Suppose that Assumption C.2 holds. Suppose that Assumptions B.3 and B.4 hold when  $Y$  is replaced by  $Y - \bar{s}(X)$ . Input the training
data  $\mathcal{D}$  into Algorithm 1 to obtain  $\hat{h}_{n,\theta^b}(x) - \hat{\phi}_{n,\theta^b}(x)$ . We have
$$
\sqrt {n} \left(\hat {h} _ {n, \theta^ {b}} (x) - \hat {\phi} _ {n, \theta^ {b}} (x) - h ^ {*} (x)\right) \Rightarrow \mathcal {N} (0, \sigma^ {2} (x)), \tag {7}
$$
$$
\sigma^ {2} (x) = \int_ {z \in \mathcal {X} \times \mathcal {Y}} I F ^ {2} (z; T _ {1}, \pi) (x) d \pi (z). \tag {8}
$$
Thus, an asymptotically (in the sense of  $n \to \infty$ ) exact  $(1 - \alpha)$ -level confidence interval of  $h^{*}(x)$  is  $[\hat{h}_{n,\theta^b}(x) - \hat{\phi}_{n,\theta^b}(x) - \frac{\sigma(x)}{\sqrt{n}} q_{1 - \frac{\alpha}{2}}, \hat{h}_{n,\theta^b}(x) - \hat{\phi}_{n,\theta^b}(x) + \frac{\sigma(x)}{\sqrt{n}} q_{1 - \frac{\alpha}{2}}]$  where  $q_{\alpha}$  is the  $\alpha$ -quantile of the standard Gaussian distribution  $\mathcal{N}(0,1)$ .
Theorem 3.5 does not indicate the value of  $\sigma^2 (x)$ . In general,  $\sigma^2 (x)$  is unknown and needs to be estimated. It is common to approximate  $IF^{2}(z;T_{1},\pi)$  with  $IF^{2}(z;T_{1},\pi_{n})$  and set  $\hat{\sigma}^2 (x) = \frac{1}{n}\sum_{z_i\in \mathcal{D}}IF^2 (z_i;T_1,\pi_n)(x)$ . This method is known as the infinitesimal jackknife variance estimator [102]. In Appendix B, we derive the exact closed-form expression of the infinitesimal jackknife variance estimation  $\hat{\sigma}^2 (x)$ , and further prove its consistency as well as the statistical coverage guarantee of confidence intervals built upon it. These results could be of theoretical interest. Yet in practice, the computation of  $\hat{\sigma}^2 (x)$  requires the evaluation of the NTK Gram matrix and its inversion, which is computationally demanding for large  $n$  and thus not recommended for practical implementation on large datasets.
In the following, we provide two efficient approaches that avoid explicit estimation of the asymptotic variance as in the infinitesimal jackknife approach, and the computation of the NTK Gram matrix inversion.
PNC-enhanced batching. We propose an approach for constructing a confidence interval that is particularly useful for large datasets, termed PNC-enhanced batching. The pseudo-code is given in Algorithm 2. Originating from simulation analysis [47, 99, 100], the key idea of batching is to construct a self-normalizing  $t$ -statistic that "cancels out" the unknown variance, leading to a valid confidence interval without explicitly needing to compute this variance. It can be used to conduct inference on serially dependent simulation outputs where the standard error is difficult to compute analytically. Previous studies have demonstrated the effectiveness of batching on the use of inference for Markov chain Monte Carlo [46, 40, 66] and also the so-called input uncertainty problem [48]. Its application in deep learning uncertainty quantification was not revealed in previous work, potentially due to the additional procedural variability. Integrating it with the PNC predictor, PNC-enhanced batching is very efficient and meanwhile possesses asymptotically exact coverage of its confidence interval, as stated below.
Algorithm 2 PNC-Enhanced Batching
Input: Training dataset  $\mathcal{D}$  of size  $n$ . The number of batches  $m^{\prime} \geq 2$ .
Procedure: 1. Split the training data  $\mathcal{D}$  into  $m^{\prime}$  batches and input each batch in Algorithm 1 to output  $\hat{h}_{n',\theta^{b}}^{j}(x) - \hat{\phi}_{n',\theta^{b}}^{j}(x)$  for  $j\in [m^{\prime}]$ , where  $n^\prime = \frac{n}{m^\prime}$ .
2. Compute  $\psi_B(x) = \frac{1}{m'}\sum_{j=1}^{m'}\left(\hat{h}_{n',\theta^b}^j(x) - \hat{\phi}_{n',\theta^b}^j(x)\right)$ ,
and  $S_B(x)^2 = \frac{1}{m' - 1}\sum_{j = 1}^{m'}\left(\hat{h}_{n',\theta^b}^j (x) - \hat{\phi}_{n',\theta^b}^j (x) - \psi_B(x)\right)^2$
Output: At point  $x$ , output  $\psi_B(x)$  and  $S_B(x)^2$ .
Theorem 3.6 (Exact coverage of PNC-enhanced batching confidence interval). Suppose that Assumption C.2 holds. Suppose that Assumptions B.3 and B.4 hold when  $Y$  is replaced by  $Y - \bar{s}(X)$ . Choose any  $m' \geq 2$  in Algorithm 2. Then an asymptotically exact  $(1 - \alpha)$ -level confidence interval of  $h^*(x)$  is  $[\psi_B(x) - \frac{S_B(x)}{\sqrt{m'}} q_{1 - \frac{\alpha}{2}}, \psi_B(x) + \frac{S_B(x)}{\sqrt{m'}} q_{1 - \frac{\alpha}{2}}]$ , where  $q_\alpha$  is the  $\alpha$ -quantile of the  $t$  distribution  $t_{m' - 1}$  with degree of freedom  $m' - 1$ .
PNC-enhanced cheap bootstrap. We propose an alternative approach for constructing a confidence interval that works for large datasets but is also suitable for smaller ones, termed PNC-enhanced cheap bootstrap. The pseudo-code is given in Algorithm 3. Cheap bootstrap [75, 76, 74] is a modified bootstrap procedure with substantially less retraining effort than conventional bootstrap methods, via leveraging the asymptotic independence between the original and resample estimators
and asymptotic normality. Note that our proposal is fundamentally different from the naive use of bootstrap or bagging when additional randomness appears [73, 81, 45], which mixes the procedural and data variabilities and does not directly provide confidence intervals. Like PNC-enhanced batching, PNC-enhanced cheap bootstrap also avoids the explicit estimation of the asymptotic variance. The difference between the above two approaches is that PNC-enhanced batching divides data into a small number of batches, and thus is suggested for large datasets while PNC-enhanced cheap bootstrap re-selects samples from the entire dataset, hence suited also for smaller datasets. On the other hand, in terms of running time, when  $R = m' - 1$ , PNC-enhanced cheap bootstrap and PNC-enhanced batching share the same number of network training, but since batching is trained on subsets of the data, the individual network training in PNC-enhanced batching is faster than PNC-enhanced cheap bootstrap. PNC-enhanced cheap bootstrap also enjoys asymptotically exact coverage of its confidence interval, as stated below.
Algorithm 3 PNC-Enhanced Cheap Bootstrap
Input: Training dataset  $\mathcal{D}$  of size  $n$ . The number of replications  $R \geq 1$ .
Procedure: 1. Input  $\mathcal{D}$  in Algorithm 1 to output  $\hat{h}_{n,\theta^b}(x) - \hat{\phi}_{n,\theta^b}(x)$ .
2. For each replication  $j \in [R]$ , resample  $\mathcal{D}$ , i.e., independently and uniformly sample with replacement from  $\{(x_1, y_1), \ldots, (x_n, y_n)\}$ $n$  times to obtain  $\mathcal{D}^{*j} = \{(x_1^{*j}, y_1^{*j}), \ldots, (x_n^{*j}, y_n^{*j})\}$ . Input  $\mathcal{D}^{*j}$  in Algorithm 1 to output  $\hat{h}_{n,\theta^b}^{*j}(x) - \hat{\phi}_{n,\theta^b}^{*j}(x)$ .
3. Compute  $\psi_C(x) = \hat{h}_{n,\theta^b}(x) - \hat{\phi}_{n,\theta^b}(x)$ ,
and  $S_{C}(x)^{2} = \frac{1}{R}\sum_{j = 1}^{R}\left(\hat{h}_{n,\theta^{b}}^{*j}(x) - \hat{\phi}_{n,\theta^{b}}^{*j}(x) - \psi_{C}(x)\right)^{2}$ .
Output: At point  $x$ , output  $\psi_C(x)$  and  $S_{C}(x)^{2}$ .
Theorem 3.7 (Exact coverage of PNC-enhanced cheap bootstrap confidence interval). Suppose that Assumption C.2 holds. Suppose that Assumptions B.3 and B.4 hold when  $Y$  is replaced by  $Y - \overline{s}(X)$ . Choose any  $R \geq 1$  in Algorithm 3. Then an asymptotically exact  $(1 - \alpha)$ -level confidence interval of  $h^*(x)$  is  $[\psi_C(x) - S_C(x)q_{1 - \frac{\alpha}{2}}, \psi_C(x) + S_C(x)q_{1 - \frac{\alpha}{2}}]$  where  $q_{\alpha}$  is the  $\alpha$ -quantile of the  $t$  distribution  $t_R$  with degree of freedom  $R$ .
We conduct numerical experiments to demonstrate the effectiveness of our approaches. Our proposed approaches are evaluated on the following two tasks: 1) construct confidence intervals and 2) reduce procedural variability to improve prediction. With a known ground-truth regression function, training data are regenerated from the underlying synthetic data generative process. According to the NTK parameterization in Section C, our base network is formed with two fully connected layers with  $n \times 32$  neurons in each hidden layer to ensure the network is sufficiently wide and over-parameterized. Detailed optimization specifications are described in Proposition C.3. Our synthetic datasets #1 are generated with the following distributions:  $X \sim \mathrm{Unif}([0,0.2]^d)$  and  $Y \sim \sum_{i=1}^{d} \sin(X^{(i)}) + \mathcal{N}(0,0.001^2)$ . The training set  $\mathcal{D} = \{(x_i, y_i) : i = 1, \dots, n\}$  is formed by drawing i.i.d. samples of  $(X, Y)$  from the above distribution with sample size  $n$ . We consider multiple dimension settings  $d = 2, 4, 8, 16$  and data size settings  $n = 128, 256, 512, 1024$  to study the effects on different dimensionalities and data sizes. Additional experimental results on more datasets are presented in Appendix F. The implementation details of our experiments are also provided in Appendix F.
Constructing confidence intervals. We use  $x_0 = (0.1, \dots, 0.1)$  as the fixed test point for confidence intervals construction. Let  $y_0 = \sum_{i=1}^{d} \sin(0.1)$  be the ground-truth label for  $x_0$  without aleatoric noise. Our goal is to construct a confidence interval at  $x_0$  for  $y_0$ . To evaluate the performance of confidence intervals, we set the number of experiments  $J = 100$ . In each repetition  $j \in [J]$ , we generate a new training dataset from the same synthetic distribution and construct a new confidence interval  $[L_j(x_0), U_j(x_0)]$  with  $95\%$  or  $90\%$  confidence level, and then check the coverage rate (CR):  $\mathrm{CR} = \frac{1}{J} \sum_{j=1}^{J} \mathbf{1}_{L_j(x_0) \leq y_0 \leq U_j(x_0)}$ . The primary evaluation of the confidence interval is based on whether its coverage rate is equal to or larger than the desired confidence level. In addition to CR,
2The source code for experiments is available at https://github.com/HZ0000/QUforNN.
Table 1: Confidence interval construction on synthetic datasets #1 with different data sizes  $n = 128,256,512,1024$  and different data dimensions  $d = 2,4,8,16$ . The CR results that attain the desired confidence level  $95\% / 90\%$  are in bold.
PNC-enhanced batching	PNC-enhanced cheap bootstrap	DropoutUQ
95%CI(CR/IW)	90%CI(CR/IW)	MP	95%CI(CR/IW)	90%CI(CR/IW)	MP	95%CI(CR/IW)	90%CI(CR/IW)	MP
(d=2)
n=128	0.98/0.0437	0.95/0.0323	0.1998	0.98/0.0571	0.95/0.0438	0.1983	0.93/0.0564	0.87/0.0480	0.2119
n=256	0.99/0.0301	0.96/0.0222	0.2004	1.00/0.0376	0.95/0.0289	0.1997	0.99/0.0390	0.96/0.0327	0.2045
n=512	0.97/0.0211	0.95/0.0156	0.2004	0.98/0.0294	0.96/0.0226	0.1992	0.95/0.0313	0.93/0.0267	0.2049
n=1024	0.96/0.0152	0.92/0.0112	0.2003	0.97/0.0205	0.94/0.0157	0.2011	0.96/0.0287	0.89/0.0244	0.2052
(d=4)
n=128	0.98/0.0622	0.94/0.0460	0.4012	1.00/0.0820	0.95/0.0629	0.4013	0.96/0.0868	0.90/0.0742	0.4157
n=256	0.98/0.0411	0.95/0.0304	0.3991	0.99/0.0569	0.96/0.0437	0.3988	0.97/0.0566	0.92/0.0481	0.4071
n=512	0.97/0.0295	0.94/0.0218	0.3988	0.98/0.0396	0.96/0.0304	0.3989	0.96/0.0375	0.90/0.0316	0.4045
n=1024	0.97/0.0213	0.95/0.0158	0.3983	0.98/0.0296	0.98/0.0227	0.3988	0.93/0.0340	0.84/0.0290	0.4055
(d=8)
n=128	0.98/0.0865	0.95/0.0639	0.7999	0.98/0.1084	0.98/0.0832	0.7980	0.91/0.1348	0.87/0.1142	0.8245
n=256	0.99/0.0594	0.96/0.0439	0.8014	1.00/0.0796	0.99/0.0611	0.7957	0.88/0.0791	0.81/0.0656	0.8152
n=512	0.98/0.0393	0.94/0.0290	0.7983	0.99/0.0588	0.99/0.0452	0.8012	0.99/0.0632	0.97/0.0538	0.7998
n=1024	0.95/0.0270	0.92/0.0200	0.7981	0.99/0.0393	0.96/0.0302	0.7997	0.88/0.0421	0.82/0.0356	0.8040
(d=16)
n=128	0.95/0.1068	0.94/0.0790	1.5946	0.99/0.1568	0.97/0.1204	1.6057	0.99/0.1565	0.96/0.1313	1.6093
n=256	0.98/0.0730	0.94/0.0540	1.5966	1.00/0.1137	0.98/0.0873	1.5954	0.90/0.1072	0.86/0.0909	1.6177
n=512	0.98/0.0543	0.93/0.0401	1.5966	0.99/0.0788	0.98/0.0605	1.5972	0.86/0.0920	0.80/0.0790	1.6132
n=1024	0.95/0.0388	0.92/0.0287	1.5976	0.97/0.0550	0.95/0.0422	1.5980	0.87/0.0760	0.83/0.0647	1.6079
we also report the median point of the interval (MP):  $\mathrm{MP} = \frac{1}{J}\sum_{j=1}^{J}\frac{1}{2}(U_j(x_0) + L_j(x_0))$  and the interval width (IW):  $\mathrm{IW} = \frac{1}{J}\sum_{j=1}^{J}(U_j(x_0) - L_j(x_0))$ . MP reflects the most likely point prediction for  $x_0$ , while IW reflects the conservativeness of the confidence interval, as a wide interval can easily achieve  $95\%$  coverage rate but is not practically preferable. In the confidence interval tasks, we use the dropout-based Bayesian approximate inference (DropoutUQ) [43] as the baseline for comparison since it is one of the most widely-used representatives in Bayesian uncertainty quantification.
Table 1 reports the CR, IW, and MP of  $95\%$  and  $90\%$  confidence intervals from our proposals and baselines. We have the following observations:
1) The CR values: For our two proposed approaches, in the majority of experiments,  $\mathrm{CR} \geq 95\%$  for  $95\%$  confidence intervals, and  $\mathrm{CR} \geq 90\%$  for  $90\%$  confidence intervals, thus satisfying the coverage requirements, while in very few experiments (in Appendix F), the performances are degraded. Nonetheless, this occasional degeneration appears insignificant and can be potentially explained by the fact that the statistical guarantee of confidence intervals generated by PNC-enhanced batching and cheap bootstrap is in the asymptotic sense. In contrast, DropoutUQ does not have such statistical guarantees. The intervals from DropoutUQ cannot normally maintain a satisfactory coverage rate when they are narrow, although they have a similar or larger interval width as PNC-enhanced batching or cheap bootstrap. Moreover, we notice that the dropout rate has a significant impact on the interval width of DropoutUQ, and thus, additional tuning efforts are demanded for this baseline, while our approaches do not need this level of tuning procedure. These observations demonstrate the robustness and effectiveness of our proposals.
2) The IW values: Overall, the IW values are relatively small for both of our approaches, indicating that our framework is able to successfully generate narrow confidence intervals with high coverage rates. Moreover, as shown, the IW values from both of our approaches decrease roughly at the rate  $n^{-\frac{1}{2}}$  along with an increased training data size  $n$ , which corroborates with our theoretical results well (Section 3.3). In comparison with PNC-enhanced cheap bootstrap, we observe that PNC-enhanced batching tends to generate narrower intervals and has less training time per network, as only a subset of training data is involved in one training trial. In general, PNC-enhanced batching requires data splits, making its accuracy vulnerable to small sample sizes, and thus leading to less stable performance compared with PNC-enhanced cheap bootstrap. Therefore, we recommend employing PNC-enhanced cheap bootstrap on small datasets and applying PNC-enhanced batching on datasets with sufficient training samples.
3) The MP values: MP appears consistent with the ground-truth label  $y_{0}$  in all experiments, showing that our confidence intervals can accurately capture the ground-truth label on average under multiple problem settings.
Table 2: Reducing procedural variability to improve prediction on synthetic datasets #1 with different data sizes  $n = {128},{256},{512},{1024}$  and different data dimensions  $d = 2,4,8,{16}$  . The best MSE results are in bold.
MSE	One base network	PNC predictor	Deep ensemble (5 networks)	Deep ensemble (2 networks)
(d=2)
n=128	(6.68±2.74) × 10-4	(3.68±1.84) × 10-4	(4.22±1.74) × 10-4	(6.28±2.74) × 10-4
n=256	(2.38±1.06) × 10-4	(6.22±2.01) × 10-5	(8.86±4.94) × 10-5	(1.25±0.67) × 10-4
n=512	(1.03±0.98) × 10-4	(2.06±0.72) × 10-5	(3.32±1.10) × 10-5	(5.11±3.33) × 10-5
n=1024	(6.98±4.77) × 10-5	(8.92±5.69) × 10-6	(1.72±0.77) × 10-5	(5.75±2.28) × 10-5
(d=4)
n=128	(2.11±1.49) × 10-3	(1.18±0.25) × 10-3	(1.53±0.60) × 10-3	(1.83±0.80) × 10-3
n=256	(8.82±3.26) × 10-4	(4.09±0.85) × 10-4	(4.22±2.01) × 10-4	(5.47±1.91) × 10-4
n=512	(5.35±1.91) × 10-4	(1.92±0.53) × 10-4	(2.88±1.80) × 10-4	(3.99±1.87) × 10-4
n=1024	(2.23±0.83) × 10-4	(4.22±0.43) × 10-5	(8.50±2.39) × 10-5	(1.65±0.57) × 10-4
(d=8)
n=128	(4.07±1.04) × 10-3	(2.54±0.29) × 10-3	(2.73±0.86) × 10-3	(3.27±0.96) × 10-3
n=256	(2.13±0.70) × 10-3	(1.05±0.18) × 10-3	(1.34±0.33) × 10-3	(1.48±0.38) × 10-3
n=512	(1.36±0.34) × 10-3	(5.04±0.70) × 10-4	(7.40±1.35) × 10-4	(1.08±0.40) × 10-3
n=1024	(8.54±2.23) × 10-4	(2.02±0.24) × 10-4	(3.79±1.01) × 10-4	(5.91±1.87) × 10-4
(d=16)
n=128	(9.03±1.64) × 10-3	(6.00±0.83) × 10-3	(6.37±0.67) × 10-3	(7.47±1.29) × 10-3
n=256	(6.76±1.79) × 10-3	(4.19±0.94) × 10-3	(4.83±1.14) × 10-3	(5.46±1.49) × 10-3
n=512	(4.19±0.51) × 10-3	(1.60±0.35) × 10-3	(2.05±0.30) × 10-3	(2.87±0.34) × 10-3
n=1024	(3.16±0.35) × 10-3	(7.44±0.99) × 10-4	(1.20±0.13) × 10-3	(1.91±0.20) × 10-3
Reduce procedural variability to improve prediction. In this part, we illustrate that our proposed PNC predator (Algorithm 1) can achieve better prediction by reducing procedural variability. We use empirical mean square error (MSE) to evaluate the prediction performance. Specifically, the empirical MSE is computed as  $\mathrm{MSE}(h)\coloneqq \frac{1}{N_{te}}\sum_{i = 1}^{N_{te}}(h(x_i') - y_i'))^2$ , where  $(x_i',y_i')$ ,  $i = 1,\dots,N_{te}$  are i.i.d. test data independent of the training. We set  $N_{te} = 2048$  for test performance evaluation in all experiments. We compare the prediction performance of our proposed PNC predictor with the following approaches: a base network, i.e., a standard network training with a single random initialization, and the deep ensemble approach with different numbers ( $m$ ) of networks in the ensemble [73]. In deep ensemble, we consider  $m = 2$  as it has a similar running time as our PNC (with one additional network training), and  $m = 5$  as it is widely used in previous work [73, 94]. All networks share the same hyperparameters and specifications.