SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 4: Reducing procedural variability to improve prediction on synthetic datasets #2 with different data sizes  $n = 128,256,512,1024$  and different data dimensions  $d = 2,4,8,16$ . The best MSE results are in bold.
MSE	One base network	PNC predictor	Deep ensemble (5 networks)	Deep ensemble (2 networks)
(d=2)
n=128	(1.02±1.16) × 10-4	(1.38±0.18) × 10-5	(2.90±1.72) × 10-5	(5.39±3.01) × 10-5
n=256	(7.36±4.84) × 10-5	(1.28±0.10) × 10-5	(2.39±1.17) × 10-5	(3.67±1.61) × 10-5
n=512	(4.41±2.34) × 10-5	(1.11±0.10) × 10-5	(1.79±5.19) × 10-5	(2.52±1.55) × 10-5
n=1024	(3.46±1.91) × 10-5	(9.78±0.94) × 10-6	(1.03±0.39) × 10-5	(1.82±1.27) × 10-5
(d=4)
n=128	(1.89±1.23) × 10-4	(1.98±0.25) × 10-5	(4.64±1.49) × 10-5	(1.02±0.31) × 10-4
n=256	(1.59±0.47) × 10-4	(1.49±0.08) × 10-5	(4.55±1.58) × 10-5	(1.02±0.37) × 10-4
n=512	(1.72±0.65) × 10-4	(1.29±0.05) × 10-5	(3.73±0.88) × 10-5	(7.78±1.72) × 10-5
n=1024	(9.75±3.49) × 10-5	(1.05±0.03) × 10-5	(3.70±0.79) × 10-5	(8.92±2.09) × 10-5
(d=8)
n=128	(8.00±1.36) × 10-4	(5.07±0.65) × 10-5	(2.18±0.59) × 10-4	(4.77±1.32) × 10-4
n=256	(4.50±0.30) × 10-4	(2.30±0.24) × 10-5	(9.44±1.44) × 10-5	(2.08±0.19) × 10-4
n=512	(3.80±0.35) × 10-4	(1.47±0.07) × 10-5	(8.44±1.11) × 10-5	(2.02±0.23) × 10-4
n=1024	(3.46±0.50) × 10-4	(1.12±0.04) × 10-5	(8.11±1.27) × 10-5	(1.71±0.23) × 10-4
(d=16)
n=128	(2.32±0.24) × 10-3	(1.39±0.14) × 10-4	(5.71±0.74) × 10-4	(1.36±0.30) × 10-3
n=256	(1.86±0.21) × 10-3	(6.55±0.75) × 10-5	(3.96±0.34) × 10-4	(9.49±0.67) × 10-4
n=512	(1.42±0.08) × 10-3	(3.32±0.29) × 10-5	(3.00±0.16) × 10-4	(7.14±0.42) × 10-4
n=1024	(1.18±0.06) × 10-3	(2.58±0.24) × 10-5	(2.56±0.18) × 10-4	(6.30±0.39) × 10-4
Gaussian random variables. Construct a confidence interval based on this "regenerating" training data  $(x_{i},y_{i} + \epsilon_{i,j})$ $(i\in \mathcal{I})$  and evaluate it on  $(x_0,y_0)$
In the above setting,  $y_{0}$  is treated as the true mean response of  $x_{0}$  without aleatoric noise, and  $\epsilon_{i,j}$  represents the only data variability. As discussed above, precise evaluation is impossible for real-world data; the above procedure, although not precise, is the best we can do to provide a resembling evaluation. Using this setting, we conduct experiments on real-world benchmark regression datasets from UCI datasets: Boston, Concrete, and Energy. These results are shown in Table 5.
From the results, we observe that our approaches also work well for these "simulated" real-world datasets. Under various problem settings, our proposed approach can robustly provide accurate confidence intervals that satisfy the coverage requirement of the confidence level. In contrast, DropoutUQ does not have such statistical guarantees. Overall, our approaches not only work for synthetic datasets but also is scalable to be applied potentially in benchmark real-world datasets.
Finally, we conduct experiments of reducing procedural variability to show the performance improvement capability of proposed models and report MSE results on the "simulated" real-world datasets: Boston, Concrete, and Energy. In each experiment, we use an  $80\% / 20\%$  random split for training data and test data in the original dataset, and then report MSE on its corresponding "simulated" dataset from our proposed PNC predictor and other baseline approaches with 10 experimental repetition times. These results are shown in Table 6.
Table 5: Confidence interval construction on "simulated" real-world benchmark datasets: Boston, Concrete, and Energy.  $J = {40}$  . The CR results that attain the desired confidence level  ${95}\% /{90}\%$  are in bold.
PNC-enhanced batching	PNC-enhanced cheap bootstrap	DropoutUQ
95%CI(CR/IW)	90%CI(CR/IW)	MP	95%CI(CR/IW)	90%CI(CR/IW)	MP	95%CI(CR/IW)	90%CI(CR/IW)	MP
Boston
Test 1	1.0/1.172	0.975/0.866	0.486	0.925/1.024	0.875/0.786	0.333	0.85/0.997	0.7/0.832	0.109
Test 2	1.0/0.873	0.95/0.645	0.100	0.975/0.869	0.925/0.667	0.042	0.75/0.981	0.625/0.842	0.089
Test 3	1.0/0.896	1.0/0.662	-0.047	0.975/1.473	0.85/1.131	0.0875	0.725/1.005	0.575/0.866	-0.119
Test 4	1.0/0.915	1.0/0.676	0.219	0.925/0.829	0.875/0.637	0.372	0.90/1.164	0.825/0.981	0.131
Test 5	1.0/0.587	1.0/0.434	0.145	1.0/0.531	0.975/0.407	0.272	0.75/1.093	0.725/0.920	0.454
Concrete
Test 1	1.0/1.378	1.0/1.019	0.416	0.975/1.013	0.975/0.778	0.379	0.725/1.176	0.675/1.016	0.088
Test 2	1.0/1.583	1.0/1.171	0.161	1.0/0.874	1.0/0.671	0.362	0.675/1.112	0.65/0.952	0.780
Test 3	1.0/1.673	1.0/1.237	0.0455	1.0/0.970	0.975/0.745	0.516	0.875/1.061	0.75/0.887	0.597
Test 4	1.0/1.156	1.0/0.855	0.484	1.0/0.960	0.95/0.737	0.466	0.875/1.101	0.825/0.949	0.552
Test 5	1.0/1.208	1.0/0.893	0.216	1.0/0.601	1.0/0.462	0.309	0.825/1.027	0.8/0.880	0.684
Energy
Test 1	0.975/0.566	0.975/0.418	0.331	1.0/0.596	1.0/0.458	0.152	0.75/0.854	0.725/0.733	0.037
Test 2	1.0/0.422	1.0/0.312	0.294	0.95/0.570	0.875/0.437	0.191	0.80/0.611	0.725/0.504	0.231
Test 3	1.0/0.435	1.0/0.321	-0.734	1.0/0.724	1.0/0.556	-0.766	0.75/0.749	0.675/0.630	-0.908
Test 4	1.0/0.572	1.0/0.423	-0.723	1.0/0.613	1.0/0.470	-0.792	0.825/0.651	0.775/0.549	-0.847
Test 5	0.975/0.512	0.975/0.379	-0.679	0.975/0.550	0.975/0.422	-0.702	0.825/0.644	0.80/0.550	-0.807
Table 6: Reducing procedural variability to improve prediction on "simulated" real-world benchmark datasets: Boston, Concrete, and Energy. The best MSE results are in bold.
MSE	One base network	PNC predictor	Deep ensemble (5 networks)	Deep ensemble (2 networks)
Boston
Test 1	(3.52 ± 0.78) × 10-1	(1.04 ± 0.03) × 10-1	(1.56 ± 0.28) × 10-1	(2.24 ± 0.51) × 10-1
Test 2	(3.15 ± 0.59) × 10-1	(1.01 ± 0.04) × 10-1	(1.38 ± 0.26) × 10-1	(2.03 ± 0.47) × 10-1
Test 3	(4.18 ± 0.69) × 10-1	(1.93 ± 0.05) × 10-1	(2.28 ± 0.33) × 10-1	(2.68 ± 0.37) × 10-1
Test 4	(3.33 ± 0.81) × 10-1	(1.17 ± 0.06) × 10-1	(1.48 ± 0.17) × 10-1	(1.84 ± 0.24) × 10-1
Test 5	(3.46 ± 0.45) × 10-1	(1.52 ± 0.04) × 10-1	(1.99 ± 0.31) × 10-1	(2.45 ± 0.32) × 10-1
Concrete
Test 1	(2.47 ± 0.24) × 10-1	(1.90 ± 0.04) × 10-1	(2.02 ± 0.07) × 10-1	(2.21 ± 0.14) × 10-1
Test 2	(3.00 ± 0.29) × 10-1	(2.18 ± 0.04) × 10-1	(2.28 ± 0.15) × 10-1	(2.43 ± 0.16) × 10-1
Test 3	(2.39 ± 0.16) × 10-1	(1.85 ± 0.04) × 10-1	(2.01 ± 0.11) × 10-1	(2.18 ± 0.08) × 10-1
Test 4	(2.85 ± 0.55) × 10-1	(1.84 ± 0.02) × 10-1	(1.92 ± 0.14) × 10-1	(2.17 ± 0.38) × 10-1
Test 5	(2.51 ± 0.19) × 10-1	(1.75 ± 0.04) × 10-1	(1.86 ± 0.06) × 10-1	(2.16 ± 0.16) × 10-1
Energy
Test 1	(1.09 ± 0.14) × 10-1	(7.73 ± 0.08) × 10-2	(8.56 ± 0.64) × 10-2	(9.91 ± 0.86) × 10-2
Test 2	(1.08 ± 0.10) × 10-1	(7.46 ± 0.17) × 10-2	(8.29 ± 0.44) × 10-2	(9.57 ± 0.84) × 10-2
Test 3	(1.23 ± 0.10) × 10-1	(7.51 ± 0.12) × 10-2	(8.45 ± 0.43) × 10-2	(1.01 ± 0.12) × 10-1
Test 4	(1.25 ± 0.12) × 10-1	(7.24 ± 0.16) × 10-2	(8.31 ± 0.91) × 10-2	(1.07 ± 0.19) × 10-1
Test 5	(1.12 ± 0.09) × 10-1	(6.81 ± 0.23) × 10-2	(7.97 ± 0.44) × 10-2	(9.69 ± 0.57) × 10-2
F.3 Confidence Intervals for Coverage Results
When evaluating the performance of confidence intervals, the CR value is computed based on a finite number of repetitions, which will incur a binomial error on the (population) coverage estimate. Therefore, in addition to reporting CR as a point estimate, we also report a confidence interval for the (population) coverage. Table 7 reports the Clopper-Pearson exact binomial confidence interval. Note that this interval can be computed straightforwardly by the point estimate (CR) and the number of repetitions  $(J)$ . Therefore, we can refer to Table 7 to additionally obtain a confidence interval of coverage for our results, e.g., in Tables 1 and 3.
Table 7: Clopper-Pearson exact binomial confidence interval for coverage estimates at confidence level  $95\%$ .  $J = 100$
CR value	Confidence interval of coverage	CR value	Confidence interval of coverage	CR value	Confidence interval of coverage
1.00	[0.964, 1.000]	0.99	[0.946, 1.000]	0.98	[0.930, 0.998]
0.97	[0.915, 0.994]	0.96	[0.901, 0.989]	0.95	[0.887, 0.984]
0.94	[0.874, 0.978]	0.93	[0.861, 0.971]	0.92	[0.848, 0.965]
0.91	[0.836, 0.958]	0.90	[0.824, 0.951]	0.89	[0.812, 0.944]
0.88	[0.800, 0.936]	0.87	[0.788, 0.929]	0.86	[0.776, 0.921]
0.85	[0.765, 0.914]	0.84	[0.753, 0.906]	0.83	[0.742, 0.898]
0.82	[0.731, 0.890]	0.81	[0.719, 0.882]	0.80	[0.708, 0.873]