SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Analytic expression of  $\Omega$  We could compute explicitly the analytical expression for the  $\Omega$  term. For notational simplicity, let decoupled and exact training conditionals be
$$
\psi (\mathbf {f} | \mathbf {f} _ {m}) = \mathcal {N} (\mathbf {Q} _ {n m} \mathbf {Q} _ {m m} ^ {- 1} \mathbf {f} _ {m}, \tilde {\mathbf {K}} _ {n n}) := \mathcal {N} (\boldsymbol {\mu} _ {1}, \boldsymbol {\Sigma}),
$$
$$
p (\mathbf {f} | \mathbf {f} _ {m}) = \mathcal {N} \left(\mathbf {K} _ {n m} \mathbf {K} _ {m m} ^ {- 1} \mathbf {f} _ {m}, \tilde {\mathbf {K}} _ {n n}\right) := \mathcal {N} \left(\boldsymbol {\mu} _ {2}, \boldsymbol {\Sigma}\right).
$$
The  $\Omega$  term is an expectation of a quadratic form:
$$
\begin{array}{l} \Omega = \int \phi (\mathbf {f} _ {m}) \mathrm {K L} (\psi (\mathbf {f} | \mathbf {f} _ {m}) | | p (\mathbf {f} | \mathbf {f} _ {m})) d \mathbf {f} _ {m} \\ = \frac {1}{2} \int \phi \left(\mathbf {f} _ {m}\right) \left(\left(\boldsymbol {\mu} _ {1} - \boldsymbol {\mu} _ {2}\right) ^ {T} \boldsymbol {\Sigma} ^ {- 1} \left(\boldsymbol {\mu} _ {1} - \boldsymbol {\mu} _ {2}\right)\right) d \mathbf {f} _ {m} \tag {8} \\ = \frac {1}{2} \mathbb {E} [ \mathbf {f} _ {m} ^ {T} \mathbf {T} \mathbf {f} _ {m} ] = \frac {1}{2} \left(\operatorname {T r} (\mathbf {T S}) + \mathbf {m} ^ {T} \mathbf {T m}\right), \\ \end{array}
$$
where  $\mathbf{T} = \mathbf{A}^T\pmb{\Sigma}^{-1}\mathbf{A}, \mathbf{A} = \mathbf{Q}_{nm}\mathbf{Q}_{mm}^{-1} - \mathbf{K}_{nm}\mathbf{K}_{mm}^{-1}$ , and  $\phi(\mathbf{f}_m) = \mathcal{N}(\mathbf{m},\mathbf{S})$ .
A.2 Whitening
In this section, we provide detailed discussion of the whitening scheme used in the decoupled models. We first present the standard whitening scheme used in variational GP models, then derive two plausible whitening schemes for decoupled models - the K-whitening and the Q-whitening. We implemented both ways of whitening for decoupled models, and empirically found that the latter (Q-whitening) performs better (See Section B.1.5).
A.2.1 The standard whitening
For standard models like SVGP, using the square root of  $\mathbf{K}_{mm}$ , the variational distribution is "whitened" (reparameterized) in the following way:
$$
\bar {q} (\mathbf {f} _ {m}) = \mathcal {N} (\bar {\mathbf {m}}, \bar {\mathbf {S}}) = \mathcal {N} (\mathbf {K} _ {m m} ^ {- 1 / 2} \mathbf {m}, \mathbf {K} _ {m m} ^ {- 1 / 2} \mathbf {S} \mathbf {K} _ {m m} ^ {- 1 / 2}).
$$
Under such reparameterization, the KL divergence is equivalent to a simpler form:
$$
\begin{array}{l} \mathrm {K L} (q (\mathbf {f} _ {m}) | | p (\mathbf {f} _ {m})) = \frac {1}{2} \left[ - \log | \mathbf {S} | + \log | \mathbf {K} _ {m m} | + \operatorname {T r} (\mathbf {K} _ {m m} ^ {- 1} \mathbf {S}) + \mathbf {m} ^ {T} \mathbf {K} _ {m m} ^ {- 1} \mathbf {m} - M \right] \\ = \frac {1}{2} \left[ - \log | \bar {\mathbf {S}} | + \operatorname {T r} \left(\mathbf {K} _ {m m} ^ {- 1 / 2} \mathbf {K} _ {m m} ^ {- 1 / 2} \mathbf {K} _ {m m} ^ {1 / 2} \bar {\mathbf {S}} \mathbf {K} _ {m m} ^ {1 / 2}\right) + \bar {\mathbf {m}} ^ {T} \mathbf {K} _ {m m} ^ {1 / 2} \mathbf {K} _ {m m} ^ {- 1 / 2} \mathbf {K} _ {m m} ^ {- 1 / 2} \mathbf {K} _ {m m} ^ {1 / 2} \bar {\mathbf {m}} - M \right] \\ = \frac {1}{2} \left[ - \log | \bar {\mathbf {S}} | + \operatorname {T r} (\bar {\mathbf {S}}) + \bar {\mathbf {m}} ^ {T} \bar {\mathbf {m}} - M \right] \\ = \operatorname {K L} (\bar {q} (\mathbf {f} _ {m}) | | p _ {0} (\mathbf {f} _ {m})), \\ \end{array}
$$
where  $p_0(\mathbf{f}_m) = \mathcal{N}(\mathbf{0},\mathbf{I})$  is the standard Normal, and  $M$  is the number of inducing points. Now plug  $\bar{\mathbf{m}}$  and  $\bar{\mathbf{S}}$  in the predictive distribution  $p(\mathbf{f})$ , we have the predictive mean  $\mu_{\mathrm{f}}(\mathbf{x}_i)$  and latent function variance  $\sigma_{\mathrm{f}}(\mathbf{x}_i)^2$  in simplified forms as well:
$$
\begin{array}{l} \mu_ {\mathbf {f}} (\mathbf {x} _ {i}) = \mathbf {K} _ {i m} \mathbf {K} _ {m m} ^ {- 1 / 2} \bar {\mathbf {m}}, \\ \sigma_ {\mathbf {f}} (\mathbf {x} _ {i}) ^ {2} = \mathbf {K} _ {i i} - \mathbf {K} _ {i m} \mathbf {K} _ {m m} ^ {- 1} \mathbf {K} _ {m i} + \mathbf {K} _ {i m} \mathbf {K} _ {m m} ^ {- 1 / 2} \bar {\mathbf {S}} \mathbf {K} _ {m m} ^ {- 1 / 2} \mathbf {K} _ {m i}. \\ \end{array}
$$
A.2.2 The K-whitening
For decoupled models like DCSVGP, we could similarly use the square root of  $\mathbf{K}_{mm}$  to whiten the variational distribution:
Same as the standard whitening, the KL divergence is simplified:
$$
\begin{array}{l} \operatorname {K L} (q (\mathbf {f} _ {m}) | | p (\mathbf {f} _ {m})) = \frac {1}{2} \left[ - \log | \mathbf {S} | + \log | \mathbf {K} _ {m m} | + \operatorname {T r} (\mathbf {K} _ {m m} ^ {- 1} \mathbf {S}) + \mathbf {m} ^ {T} \mathbf {K} _ {m m} ^ {- 1} \mathbf {m} - M \right] \\ = \frac {1}{2} \left[ - \log | \bar {\mathbf {S}} | + \mathrm {T r} (\bar {\mathbf {S}}) + \bar {\mathbf {m}} ^ {T} \bar {\mathbf {m}} - M \right] \\ = \operatorname {K L} (\bar {q} (\mathbf {f} _ {m}) | | p _ {0} (\mathbf {f} _ {m})), \\ \end{array}
$$
where  $p_0(\mathbf{f}_m) = \mathcal{N}(\mathbf{0},\mathbf{I})$  is the standard Normal, and  $M$  is the number of inducing points. However, if we plug  $\tilde{\mathbf{m}}$  and  $\tilde{\mathbf{S}}$  in the predictive distribution  $p(\mathbf{f})$ , the predictive mean  $\tilde{\mu}_{\mathbf{f}}(\mathbf{x}_i)$  and latent function variance  $\tilde{\sigma}_{\mathbf{f}}(\mathbf{x}_i)^2$  are not simplified in the same way as the standard case:
$$
\begin{array}{l} \tilde {\mu} _ {\mathbf {f}} \left(\mathbf {x} _ {i}\right) = \mathbf {Q} _ {i m} \mathbf {Q} _ {m m} ^ {- 1} \mathbf {K} _ {m m} ^ {1 / 2} \bar {\mathbf {m}}, \\ \tilde {\sigma} _ {\mathbf {f}} (\mathbf {x} _ {i}) ^ {2} = \mathbf {K} _ {i i} - \mathbf {K} _ {i m} \mathbf {K} _ {m m} ^ {- 1} \mathbf {K} _ {m i} + \mathbf {Q} _ {i m} \mathbf {Q} _ {m m} ^ {- 1} \mathbf {K} _ {m m} ^ {1 / 2} \bar {\mathbf {S}} \mathbf {K} _ {m m} ^ {1 / 2} \mathbf {Q} _ {m m} ^ {- 1} \mathbf {Q} _ {m i}. \\ \end{array}
$$
A.2.3 The Q-whitening
For decoupled models like DCSVGP, another plausible way of whitening for decoupled models is to use the square root of  $\mathbf{Q}_{mm}$ :
$$
\bar {q} (\mathbf {f} _ {m}) = \mathcal {N} (\bar {\mathbf {m}}, \bar {\mathbf {S}}) = \mathcal {N} (\mathbf {Q} _ {m m} ^ {- 1 / 2} \mathbf {m}, \mathbf {Q} _ {m m} ^ {- 1 / 2} \mathbf {S} \mathbf {Q} _ {m m} ^ {- 1 / 2}).
$$
In this way, the KL divergence term is equivalent to a different form:
$$
\begin{array}{l} \operatorname {K L} (q (\mathbf {f} _ {m}) | | p (\mathbf {f} _ {m})) = \frac {1}{2} \left[ - \log | \mathbf {S} | + \log | \mathbf {K} _ {m m} | + \operatorname {T r} (\mathbf {K} _ {m m} ^ {- 1} \mathbf {S}) + \mathbf {m} ^ {T} \mathbf {K} _ {m m} ^ {- 1} \mathbf {m} - M \right] \\ = \frac {1}{2} \left[ - \log | \mathbf {Q} _ {m m} ^ {1 / 2} \bar {\mathbf {S}} \mathbf {Q} _ {m m} ^ {1 / 2} | + \log | \mathbf {K} _ {m m} | + \operatorname {T r} \left(\mathbf {K} _ {m m} ^ {- 1} \mathbf {Q} _ {m m} ^ {1 / 2} \bar {\mathbf {S}} \mathbf {Q} _ {m m} ^ {1 / 2}\right) + \bar {\mathbf {m}} ^ {T} \mathbf {Q} _ {m m} ^ {1 / 2} \mathbf {K} _ {m m} ^ {- 1} \mathbf {Q} _ {m m} ^ {1 / 2} \bar {\mathbf {m}} - M \right] \\ = \frac {1}{2} \left[ - \log | \mathbf {Q} _ {m m} ^ {1 / 2} \bar {\mathbf {S}} \mathbf {Q} _ {m m} ^ {1 / 2} | + \log | \mathbf {K} _ {m m} ^ {1 / 2} \mathbf {K} _ {m m} ^ {1 / 2} | + \operatorname {T r} (\mathbf {K} _ {m m} ^ {- 1 / 2} \mathbf {K} _ {m m} ^ {- 1 / 2} \mathbf {Q} _ {m m} ^ {1 / 2} \bar {\mathbf {S}} \mathbf {Q} _ {m m} ^ {1 / 2}) + \bar {\mathbf {m}} ^ {T} \mathbf {Q} _ {m m} ^ {1 / 2} \mathbf {K} _ {m m} ^ {- 1 / 2} \mathbf {K} _ {m m} ^ {- 1 / 2} \mathbf {Q} _ {m m} ^ {1 / 2} \bar {\mathbf {m}} - M \right] \\ = \frac {1}{2} \left[ - \log | \bar {\mathbf {S}} | + \log | \mathbf {Q} _ {m m} ^ {- 1 / 2} \mathbf {K} _ {m m} ^ {1 / 2} \mathbf {K} _ {m m} ^ {1 / 2} \mathbf {Q} _ {m m} ^ {- 1 / 2} | + \operatorname {T r} (\mathbf {K} _ {m m} ^ {- 1 / 2} \mathbf {Q} _ {m m} ^ {1 / 2} \bar {\mathbf {S}} \mathbf {Q} _ {m m} ^ {1 / 2} \mathbf {K} _ {m m} ^ {- 1 / 2}) + \bar {\mathbf {m}} ^ {T} \mathbf {Q} _ {m m} ^ {1 / 2} \mathbf {K} _ {m m} ^ {- 1 / 2} \mathbf {K} _ {m m} ^ {- 1 / 2} \mathbf {Q} _ {m m} ^ {1 / 2} \bar {\mathbf {m}} - M \right] \\ = \frac {1}{2} \left[ - \log | \bar {\mathbf {S}} | + \log | \bar {\mathbf {L}} \bar {\mathbf {L}} ^ {T} | + \operatorname {T r} \left(\bar {\mathbf {L}} ^ {- 1} \bar {\mathbf {S}} \bar {\mathbf {L}} ^ {- T}\right) + \bar {\mathbf {m}} ^ {T} \bar {\mathbf {L}} ^ {- T} \bar {\mathbf {L}} ^ {- 1} \bar {\mathbf {m}} - M \right] \\ = \frac {1}{2} \left[ - \log | \bar {\mathbf {S}} | + \log | \bar {\mathbf {K}} | + \operatorname {T r} \left(\bar {\mathbf {K}} ^ {- 1} \bar {\mathbf {S}}\right) + \bar {\mathbf {m}} ^ {T} \bar {\mathbf {K}} ^ {- 1} \bar {\mathbf {m}} - M \right] \\ = \operatorname {K L} (\bar {q} (\mathbf {f} _ {m}) | | \bar {p} (\mathbf {f} _ {m})), \\ \end{array}
$$
where  $\bar{p} (\mathbf{f}_m) = \mathcal{N}(\mathbf{0},\bar{\mathbf{K}})$  and  $\bar{\mathbf{K}} = \bar{\mathbf{L}}\bar{\mathbf{L}}^T$ $\bar{\mathbf{L}} = \mathbf{Q}_{mm}^{-1 / 2}\mathbf{K}_{mm}^{1 / 2}$  and  $M$  is the number of inducing points. Now plug  $\bar{\mathbf{m}}$  and  $\bar{\mathbf{S}}$  in the predictive distribution  $p(\mathbf{f})$ , we have the predictive mean  $\tilde{\mu}_{\mathrm{f}}(\mathbf{x}_i)$  and latent function variance  $\tilde{\sigma}_{\mathrm{f}}(\mathbf{x}_i)^2$  simplified in a similar way as the standard case:
$$
\begin{array}{l} \tilde {\mu} _ {\mathbf {f}} \left(\mathbf {x} _ {i}\right) = \mathbf {Q} _ {i m} \mathbf {Q} _ {m m} ^ {- 1 / 2} \bar {\mathbf {m}}, \\ \tilde {\sigma} _ {\mathbf {f}} \left(\mathbf {x} _ {i}\right) ^ {2} = \mathbf {K} _ {i i} - \mathbf {K} _ {i m} \mathbf {K} _ {m m} ^ {- 1} \mathbf {K} _ {m i} + \mathbf {Q} _ {i m} \mathbf {Q} _ {m m} ^ {- 1 / 2} \bar {\mathbf {S}} \mathbf {Q} _ {m m} ^ {- 1 / 2} \mathbf {Q} _ {m i}. \\ \end{array}
$$
As discussed in the main paper, it is not surprising that the  $\mathbf{Q}$ -whitening performs better in practice. This is because the  $\mathbf{K}$ -whitening simplifies the KL divergence but leaves the predictive distribution complicated, while the  $\mathbf{Q}$ -whitening does the opposite, and a complicated predictive distribution makes the optimization much harder than a complicated KL divergence term.
B.1 Regression Tasks
Experiment setup Table 6 provides details on the 10 UCI regression datasets [10] and the number of inducing points used for each dataset. For ODSVGP, since the goal is to use more inducing points for mean and less for covariance under the same budget, we use the setup in the original paper [41]: if SVGP uses 4X inducing points, then ODSVGP uses 7X inducing points for mean and 3X inducing points for covariance. We use the Adam [25] optimizer with a multistep scheduler to train all models on all datasets, and we train for 300 epochs using training batch size 1024. We selected the best training hyperparameters for SVGP and use the same ones for all models - learning rate  $\mathrm{lr} = 5\mathrm{e} - 3$  and a multistep learning rate scheduler (multiplicative factor  $\gamma = 0.2$ ).
B.1.1 Learned lengthscales
Table 6 also provides learned lengthscales of all models, averaged over 10 random dataset splits, as supplementary results for Table 2. As is shown in Table 6, we observe how lengthscales are decoupled in DCSVGP and DCPPGPR – the lengthscale for mean function  $l_{\mathrm{mean}}$  is typically smaller than the shared lengthscale  $l$ , while the lengthscale for covariance function  $l_{\mathrm{covar}}$  is typically larger than the shared lengthscale  $l$ . This is indeed consistent with the motivation of decoupling lengthscales since a smaller lengthscale for the mean function would typically lead to a better mean fit when the number of inducing points is limited compared to total training data; when the mean function is well fit, the lengthscale for the covariance can be larger to further reduce the predictive covariance.
B.1.2 Error bars of main results
Here we provide standard errors of main results Table 2 and Table 3 - see Figure 4 and Figure 5 respectively. Error bars for other similar results, i.e. Tables in the Appendix, are similarly small and not informative and therefore we avoid redundant results.
Table 6: We present details on UCI datasets and number of inducing points used in SVGP, DCSVGP, PPGPR and DCPPGPR here. We also compare the learned lengthscale values (averaged over 10 random dataset splits): lengthscale  $l$  of baseline models SVGP, ODSVGP, PPGPR and ODPPGPR, lengthscales  $l_{\mathrm{mean}}$  and  $l_{\mathrm{covar}}$  of decoupled models DCSVGP and DCPPGPR. We observe that the lengthscale for mean  $l_{\mathrm{mean}}$  is typically smaller than the shared lengthscale  $l$ , while the lengthscale for covariance  $l_{\mathrm{covar}}$  is typically larger than the shared lengthscale  $l$ . See App. B.1 and App. B.1.1 for more details.
Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
Total number of data	11250	12449	13034	30000	34297	36620	40125	47706	326155	386508
Input dimension	26	18	17	8	9	20	380	26	3	90
#inducing points	500	500	500	800	800	800	800	800	1000	1000
SVGP l	0.976	1.370	2.330	1.220	0.260	1.150	19.600	2.270	0.120	1.110
ODSVGP l	1.418	1.561	3.162	1.710	0.412	1.726	20.694	2.665	0.158	1.664
DCSVGP lmean	0.291	0.761	1.174	0.673	0.104	0.791	4.956	1.273	0.038	0.346
DCSVGP lcovar	3.304	2.257	3.459	1.721	0.555	3.190	28.112	5.876	0.170	2.124
PPGPR l	0.789	1.060	2.890	1.300	0.200	1.030	21.000	2.510	0.110	0.490
ODPPGPR l	1.079	1.171	3.387	1.825	0.261	1.298	20.536	2.583	0.137	0.591
DCPPGPR lmean	0.290	0.790	1.423	0.797	0.122	0.923	5.053	2.161	0.100	0.349
DCPPGPR lcovar	3.811	1.361	4.556	1.793	0.298	2.122	26.734	4.814	0.138	0.640
Figure 4: Test RMSE and NLL results with standard errors on UCI datasets, same results as Table 2 in main paper. We compare DCSVGP to SVGP and ODSVGP and compare DCPPGPR to PPGPR and ODPPGPR. Row 1 and 2 include RMSE results and row 3 and 4 include NLL results (same row order as Table 2 in the main paper). See App. B.1.2 for more details.
Figure 5: Test RMSE and NLL results with standard errors on UCI datasets, same results as Table 3 in main paper. We compare SVGP-DCDKL to SVGP-DKL and compare PPGPR-DCDKL to PPGPR-DKL. Row 1 and 2 include RMSE results and row 3 and 4 include NLL results (same row order as Table 3 in the main paper). See App. B.1.2 for more details.
Figure 6: On dataset Kin40K, we plot z-scores distributions of model prediction (orange curves), and compare with the standard Normal (black dashed curves). The closer the model predictive distribution is to the standard Normal, the better the model is calibrated. Clearly, decoupled models DCSVGP and DCPPGPR are better calibrated compare to their counterparts SVGP and PPGPR, respectively. Quantitatively, in the titles we show the Wasserstein distance  $D$  between the two distributions, the smaller the better. See App. B.1.3 for more details.
Table 7: Test RMSE and NLL on 10 univariate regression datasets (lower is better) with ARD kernels. We compare decoupled models DCSVGP and DCPPGPR with their coupled counterparts SVGP and PPGPR. Results are averaged over 6 random train/validation/test splits. Best ones with statistical significance are bold. See App. B.1.4 for more details.
Test RMSE	Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
SVGP	0.167	0.370	0.089	0.166	0.646	0.089	0.111	0.122	0.376	0.794
DCSVGP	0.078	0.366	0.058	0.116	0.590	0.087	0.041	0.120	0.323	0.772
PPGPR	0.166	0.376	0.127	0.240	0.643	0.092	0.150	0.124	0.450	0.779
DCPPGPR	0.084	0.372	0.064	0.163	0.614	0.090	0.047	0.123	0.458	0.776
Test NLL	Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
SVGP	-0.286	0.426	-0.873	-0.315	0.991	-1.027	-0.482	-0.689	0.457	1.188
DCSVGP	-1.102	0.414	-1.453	-0.712	0.901	-1.038	-1.279	-0.710	0.289	1.161
PPGPR	-0.871	0.359	-1.818	-0.844	0.783	-1.603	-0.919	-1.900	-0.084	1.108
DCPPGPR	-1.494	0.392	-2.223	-1.177	0.885	-1.622	-1.505	-1.989	-0.083	1.121
B.1.3 Model Calibration
To study how decoupled conditionals affect model calibration, we evaluate the z-score distribution from model predictions on all UCI datasets and compare it to the standard Normal. The closer the z-score distribution from model predictions is to the standard Normal, the better the model is calibrated. As an example, Figure 6 shows the z-score distributions of SVGP, DCSVGP, PPGPR and DCPPGPR on the Kin40k dataset. As Figure 6 shows, on the Kin40k dataset, decoupled models DCSVGP and DCPPGPR are better calibrated compare to their counterparts SVGP and PPGPR, respectively. Quantitatively, we compute the Wasserstein distance  $D$  [37] between the model z-score distribution and standard Normal (see subtitles in Figure 6 for example), the smaller the better. Averaged over all 10 UCI datasets, SVGP and DCSVGP both have  $D = 0.27$ , PPGPR has  $D = 0.22$  and DCPPGPR has the best  $D = 0.17$ . Therefore, we conclude that decoupling conditionals does not lead to a poorly calibrated model, but can even improve model calibration on average.
B.1.4 ARD kernels
Here we provide additional results on regression tasks using ARD kernels. In Table 2, we show that decoupled models outperform the baseline models using non-ARD kernels (kernels with one-dimensional lengthscale). Here, we use ARD kernels (kernels with multi-dimensional lengthscales, one for each input dimension) for all models and show consistent advantages of using decoupled models. Table 7 summarizes results, similar to the main results using non-ARD kernels (Table 2 in main paper), we observe that all decoupled models outperform baseline models.
B.1.5 Whitening of decoupled models
In Section A.2, we discussed two plausible whitening schemes for the decoupled models and claimed that the  $\mathbf{Q}$ -whitening is more favorable than the  $\mathbf{K}$ -whitening due to the simplified predictive
Table 8: Test RMSE and NLL on 10 univariate regression datasets (lower is better). Using the DCSVGP model, we compare the K-whitening and Q-whitening discussed in App. A.2. Results are averaged over 10 random train/validation/test splits. Best ones with statistical significance are bold. See App. B.1.5 for more details.
Test RMSE	Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
K-whitening	0.163	0.380	0.260	0.158	0.635	0.087	0.041	0.119	0.437	0.774
Q-whitening	0.156	0.379	0.286	0.150	0.604	0.086	0.039	0.121	0.434	0.777
Test NLL	Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
K-whitening	-0.099	0.457	0.083	-0.453	0.966	-1.045	-1.243	-0.703	0.591	1.163
Q-whitening	-0.373	0.450	0.165	-0.502	0.919	-1.047	-1.293	-0.697	0.586	1.166
Table 9: Test RMSE and NLL on 10 univariate regression datasets (lower is better). Using the DCPPGPR model, we compare the K-whitening and Q-whitening discussed in App. A.2. Results are averaged over 10 random train/validation/test splits. Best ones with statistical significance are bold. See App. B.1.5 for more details.
Test RMSE	Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
K-whitening	0.178	0.387	0.296	0.214	0.659	0.089	0.046	0.123	0.567	0.781
Q-whitening	0.178	0.395	0.348	0.226	0.632	0.089	0.042	0.124	0.543	0.779
Test NLL	Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
K-whitening	0.434	0.475	-0.786	-0.941	0.813	-1.638	-1.472	-1.861	0.277	1.112
Q-whitening	-0.588	0.403	-0.901	-1.067	0.854	-1.648	-1.574	-1.904	0.227	1.109
Table 10: Test RMSE and NLL on 10 univariate regression datasets (lower is better) using SVGP-DKL. We compare learning inducing points in the feature space with learning inducing points in the input space. Results are averaged over 10 random train/validation/test splits. Best ones with statistical significance are bold. See App. B.1.6 for more details.
Test RMSE	Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
feature space	0.0867	0.347	0.067	0.086	0.614	0.0905	0.0237	0.122	0.382	0.772
input space	0.0614	0.343	0.013	0.067	0.598	0.0864	0.0189	0.120	0.329	0.773
Test NLL	Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
feature space	-1.014	0.361	-1.292	-1.070	0.931	-1.007	-2.214	-0.691	0.464	1.161
input space	-1.388	0.347	-2.590	-1.227	0.905	-1.045	-2.550	-0.712	0.320	1.161
distribution. Here, we empirically compare the two whitening schemes and verify such claim. Table 8 summarizes a comparison of the K-whitening and Q-whitening in the DCSVGP model, and we observe that Q-whitening outperforms K-whitening on 8 out of 10 datasets. Table 9 shows similar results on the DCPPGPR model.
B.1.6 DKL discussion
In the main paper, we evaluate SVGP-DKL, PPGPR-DKL and their decoupled counterparts SVGP-DCDKL and PPGPR-DCDKL on 10 UCI datasets, and we choose to learn inducing points in the input space for all models. Despite the convention of learning inducing points in the feature space, we found that for baseline models SVGP-DKL and PPGPR-DKL, learning inducing points in the input space performs better on the UCI datasets, see Table 10 and Table 11.
B.2 Ablation Study on  $\beta_{2}$
In addition to the ablation study results on DCPPGPR in the main paper, here we provide more ablation study results on DCSVGP and DCPPGPR with varying  $\beta_{2}$  values on all datasets. Together with the representative results shown in the main paper, We obtain the same conclusion that
Table 11: Test RMSE and NLL on 10 univariate regression datasets (lower is better) using PPGPR-DKL. We compare learning inducing points in the feature space with learning inducing points in the input space. Results are averaged over 10 random train/validation/test splits. Best ones with statistical significance are bold. See App. B.1.6 for more details.
Test RMSE	Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
feature space	0.1614	0.346	0.096	0.099	0.620	0.0953	0.0213	0.124	0.481	0.769
input space	0.0847	0.343	0.030	0.082	0.597	0.0871	0.0176	0.121	0.375	0.771
Test NLL	Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
feature space	-2.169	0.303	-3.054	-1.754	0.764	-1.574	-2.118	-1.956	0.106	1.100
input space	-2.324	0.303	-3.096	-1.739	0.712	-1.627	-2.625	-1.951	-0.232	1.098
Table 12: Test RMSE and NLL on 10 regression datasets (lower is better). We compare DCSVGP with different  $\beta_{2}$  values. Results are averaged over 10 random train/validation/test splits. Best ones with statistical significance are bold. We observe that  $\beta_{2} = 0.001$  or even  $\beta_{2} = 0$  are good choices. See App. B.2 for more details.
Test RMSE	Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
β2=1.0	0.255	0.379	0.290	0.158	0.640	0.087	0.039	0.121	0.452	0.780
β2=0.001	0.156	0.379	0.286	0.150	0.604	0.086	0.039	0.121	0.434	0.777
β2=0	0.155	0.379	0.286	0.151	0.605	0.087	0.039	0.121	0.433	0.778
Test NLL	Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
β2=1.0	0.120	0.450	0.184	-0.421	0.974	-1.043	-1.274	-0.695	0.632	1.170
β2=0.001	-0.373	0.450	0.165	-0.502	0.919	-1.047	-1.293	-0.697	0.586	1.166
β2=0	-0.370	0.450	0.165	-0.497	0.920	-1.039	-1.293	-0.698	0.582	1.167
1)  $\beta_{2} = 0.001$  or even  $\beta_{2} = 0$  are good choices for both DCSVGP and DCPPGPR (see Table 12 and Table 13);
2) on rare cases (only on the Pol and Elevators dataset, see Table 13), DCPPGPR could overfit and  $\beta_{2} = 0.1$  or  $\beta_{2} = 1.0$  would fix the issue and improve the prediction performance.
More detailed results and discussions follow.
1D Example Using the same 1D toy example from the main paper, here in Figure 7 we show ablation study on this 1D example. Consistently, DCSVGP outperforms SVGP with 10 inducing points, and DCSVGP keeps improving as  $\beta_{2}$  decreases. We also observe that the decoupled lengthscales  $l_{\mathrm{mean}}$  and  $l_{\mathrm{covar}}$  differ more and more as  $\beta_{2}$  decreases and thus allowing the model to better fit mean and covariance differently.