SlowGuess/NeurIPS_2023_OCR huggingface dataset:
DCSVGP From Table 12, we observe that, DCSVGP favors small  $\beta_{2}$  values and  $\beta_{2} = 0.001$  or even  $\beta_{2} = 0$  are good choices. The general trend is that DCSVGP performs better as  $\beta_{2}$  decreases. To clearly see this trend, in Figure 8 we provide detailed performance study on 3 datasets: the 1D synthetic dataset, the Pol dataset, and the Song dataset. We observe that both RMSE and NLL decrease as  $\beta_{2}$  decreases, and the learned lengthscales differ more and more as  $\beta_{2}$  decreases to better model the mean and covariance differently.
DCPPGPR From Table 13, we observe similar trend that DCPPGPR also favors small  $\beta_{2}$  values and  $\beta_{2} = 0.001$  or  $\beta_{2} = 0$  are still good choices. In the main paper, we observe that DCPPGPR overfits on dataset Elevators and Protein, and on Protein we show how varying  $\beta_{2}$  improves test NLL in detail (Figure 2 in the main paper). Here we also observe from Table 13 that regularization with  $\beta_{2} = 1.0$  would fix the issue for both dataset Elevators and Protein. Figure 13 shows how the test RMSE and NLL change with  $\beta_{2}$  on the Elevators dataset - we observe that RMSE does not change much as  $\beta_{2}$  varies, but NLL decreases as  $\beta_{2}$  increases, showing how regularization improves test NLL.
(a) Test NLL:0.63,  $l$ : 1.13
(b) Test NLL:0.54,  $l_{\mathrm{mean}}$  : 0.66,  $l_{\mathrm{covar}}$  : 0.68
(c) Test NLL:0.17,  $l_{\mathrm{mean}}$  : 0.34,  $l_{\mathrm{covar}}$  : 0.52
Figure 7: We compare model fit on a 1D latent function using 100 training samples and 10 inducing points. Solid curves with shading area depict the predictive mean and  $95\%$  confidence interval. In **subplot** (a), SVGP underfits the latent function with large lengthscale  $l = 1.13$ . From **subplot** (b) to (d), DCSVGP fits better and better with decreasing  $\beta_{2} = \{1.0, 0.1, 0.01\}$ . We also observe that the decoupled lengthscales  $l_{\mathrm{mean}}$  and  $l_{\mathrm{covar}}$  differ more and more as  $\beta_{2}$  decreases and thus allowing the model to better fit mean and covariance differently. See App. B.2 for more details.
(d) Test NLL:0.55,  $l_{\mathrm{mean}}$  .. 0.28,  $l_{\mathrm{covar}}$  .. 0.71
Table 13: Test RMSE and NLL on 10 regression datasets (lower is better). We compare DCPPGPR with different  $\beta_{2}$  values. Results are averaged over 10 random train/validation/test splits. Best ones with statistical significance are bold. We observe that DCPPGPR generally favors small  $\beta_{2}$  values, and regularization would fix rare overfitting (on Elevators and Protein). See App. B.2 for more details.
Test RMSE	Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
β2=1.0	0.258	0.393	0.366	0.233	0.645	0.090	0.044	0.124	0.546	0.778
β2=0.001	0.178	0.395	0.348	0.226	0.632	0.089	0.042	0.124	0.543	0.779
β2=0	0.177	0.393	0.345	0.227	0.634	0.090	0.043	0.123	0.544	0.779
Test NLL	Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
β2=1.0	-0.287	0.377	-0.815	-0.987	0.790	-1.616	-1.524	-1.859	0.237	1.108
β2=0.001	-0.588	0.403	-0.901	-1.067	0.854	-1.648	-1.574	-1.904	0.227	1.109
β2=0	-0.616	0.399	-0.899	-1.065	0.855	-1.635	-1.538	-1.906	0.225	1.110
Table 14: Test RMSE on 10 regression datasets (lower is better). We compare NNSVGP to DCSVGP and SVGP and compare NNPPGPR to DCPPGPR and PPGPR. Results are averaged over 10 random train/validation/test splits. Best ones with statistical significance are bold. NNSVGP and NNPPGPR both yield the best test RMSE on 9 out of 10 datasets.
Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
SVGP	0.313	0.380	0.294	0.186	0.662	0.089	0.131	0.122	0.511	0.797
DCSVGP	0.156	0.379	0.286	0.150	0.604	0.086	0.039	0.121	0.434	0.777
NNSVGP	0.069	0.353	0.057	0.103	0.590	0.084	0.019	0.117	0.523	0.778
PPGPR	0.306	0.392	0.377	0.282	0.659	0.091	0.205	0.125	0.552	0.780
DCPPGPR	0.178	0.395	0.348	0.226	0.632	0.089	0.042	0.124	0.543	0.779
NNPPGPR	0.069	0.352	0.057	0.118	0.604	0.085	0.021	0.118	0.623	0.776
B.3 Plausible Extension
RMSE results of NNSVGP We provide the RMSE results of NNSVGP and NNPPGPR here in Table 14 to support the observation that they yield the best test RMSE on 9 out of 10 datasets.
Figure 8: We evaluate DCSVGP with varying  $\beta_{2}$  on the 1D dataset (first row), dataset Pol (second row) and Song (third row). Solid blue lines show DCSVGP with nonzero  $\beta_{2}$  and green dashed lines show DCSVGP with  $\beta_{2} = 0$ . First two columns contain test metrics, lower the better. Last two columns show the learned decoupled lengthscales  $l_{\mathrm{mean}}$  and  $l_{\mathrm{covar}}$ . We observe that both RMSE and NLL decrease as  $\beta_{2}$  decreases. See App. B.2 for more details.
Figure 9: We evaluate DCPPGPR with varying  $\beta_{2}$  on dataset Elevators, similar to Figure 8. We observe that test RMSE does not change much as  $\beta_{2}$  varies, but NLL decreases as  $\beta_{2}$  increases, showing how regularization improves test NLL. See App. B.2 for more details.