SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The regularization parameter. In Eq. 5,  $\mathrm{ELBO}_{\mathrm{DCSVGP}}$  contains two KL divergence terms with regularization parameters  $\beta_{1}$  and  $\beta_{2}$ , respectively. Varying  $\beta_{1}$  controls the regularization on prior similarities, same as SVGP, while varying  $\beta_{2}$  controls the regularization on the difference between the decoupled conditional and the exact one. In the limit where  $\beta_{2} \to +\infty$ , decoupling is disallowed and DCSVGP degenerates to SVGP. See Sec. 5.2 for more discussion and empirical study.
Whitening. Decoupling the mean and covariance hyperparameters introduces a challenge where one can whiten with either  $\mathbf{K}_{mm}^{1/2}$  or  $\mathbf{Q}_{mm}^{1/2}$ , with the drawback of only simplifying one term.  $\mathbf{K}$ -whitening simplifies the KL divergence but leaves the predictive distribution significantly "less linear", while  $\mathbf{Q}$ -whitening does the opposite. For example, the predictive distribution from the two whitening choices are:
$$
\mathbf {Q} \text {- w h i t e n g h e d t h e r m a l {\tilde {\mu}} _ {f} (\mathbf {x} _ {i}) = \mathbf {Q} _ {i m} \mathbf {Q} _ {m m} ^ {- 1 / 2} \bar {\mathbf {m}} , \tilde {\sigma} _ {f} (\mathbf {x} _ {i}) ^ {2} = \tilde {\mathbf {K}} _ {i i} + \mathbf {Q} _ {i m} \mathbf {Q} _ {m m} ^ {- 1 / 2} \bar {\mathbf {S}} \mathbf {Q} _ {m m} ^ {- 1 / 2} \mathbf {Q} _ {m i} ,}}
$$
$$
\mathbf {K} \mathrm {- w h i t e n i n g} \tilde {\mu} _ {\mathbf {f}} (\mathbf {x} _ {i}) = \mathbf {Q} _ {i m} \mathbf {Q} _ {m m} ^ {- 1} \mathbf {K} _ {m m} ^ {1 / 2} \bar {\mathbf {m}}, \tilde {\sigma} _ {\mathbf {f}} (\mathbf {x} _ {i}) ^ {2} = \tilde {\mathbf {K}} _ {i i} + \mathbf {Q} _ {i m} \mathbf {Q} _ {m m} ^ {- 1} \mathbf {K} _ {m m} ^ {1 / 2} \bar {\mathbf {S}} \mathbf {K} _ {m m} ^ {1 / 2} \mathbf {Q} _ {m m} ^ {- 1} \mathbf {Q} _ {m i}.
$$
Empirically  $\mathbf{Q}$ -whitening performs better because a simplified predictive distribution is more favorable for model fitting. See App. A.2 for KL divergence terms and derivation details.
Variational GPs have been successfully extended to various settings [1, 3, 4, 11, 16, 19, 20, 31, 51, 55, 54]. Among these vast enhancements, much attention has been devoted to the study of inducing points, the core part that leads to an expressiveness-and-complexity trade-off. Many works have studied better placement of inducing points, including selecting from training data [9, 24, 42, 44, 47, 7, 34] and from the input domain [40, 50], but typically inducing points are learned as model parameters [17, 48]. On the other hand, several schemes have been developed to reduce complexity and allow more inducing points, e.g. Fourier methods [19, 29] and methods that decouple inducing points used in variational mean and covariance [8, 15, 41]. However, more inducing points could be less informative under certain data characteristics [6, 16, 50] or result in suboptimal model fitting [15, 57]. Therefore, our work takes a different direction to improve variational GPs via more flexible mean and covariance modeling rather than replying on more inducing points. The work that is most closely related to ours is the ODSVGP model [41] that decouples the inducing points for mean and variance to allow more inducing points used in mean modeling. With similar decoupling idea, we are motivated by increasing model flexibility rather than reducing model complexity and we simply decouple the kernel hyperparameters with negligible additional costs.
We evaluate the performance of decoupled models proposed in Sec. 3.2: DCSVGP (variational GPs using decoupled lengthscales) and SVGP-DCDKL (variational GPs with deep kernel learning using decoupled deep feature extractors). Because PPGPR [22] is orthogonal to our decoupling method, similarly we also evaluate DCPPGPR (decoupled lengthscales) and PPGPR-DCDKL (decoupled feature extractors). All experiments use an RBF kernel and a zero prior mean and are accelerated through GPyTorch [14] on a single GPU. Code is available at https://github.com/xinranzhu/Variational-GP-Decoupled-Conditionals.
5.1 Regression Tasks
We consider 10 UCI regression datasets [10] with up to 386508 training examples and up to 380 dimensions. We present main results with non-ARD RBF kernels, the Q-whitening scheme described in Sec. 3.3, and set  $\beta_{2} = 1\mathrm{e - }3$ . Results are averaged over 10 random dataset splits. For additional results with ARD kernels and K-whitening, see App. B.1.4 and B.1.5. For more experiment setup and training details such as the number of inducing points, we refer to App. B.1.
DCSVGP and DCPPGPR. We first compare DCSVGP to baseline SVGP [17] and ODSVGP [41] and similarly compare DCPPGPR to PPGPR [22] and ODPPGPR. Table 2 shows the test RMSE and NLL results. We observe that DCSVGP (and similarly DCPPGPR) yields the lowest RMSE and NLL on 8 datasets, demonstrating the improved flexibility of our method. We report all learned lengthscales in App. B.1.1. On the other hand, we also note that on Protein and Elevators, DCPPGPR slightly overfits with worse test NLL. We show in Sec. 5.2 that increasing  $\beta_{2}$  resolves this issue. We also find that decoupling lengthscales generally improves model calibration (see App. B.1.3 for results and discussions).
Table 2: Test RMSE and NLL on 10 regression datasets (lower is better). Results are averaged over 10 random train/validation/test splits. Statistical significance is indicated by bold. See the supplement for standard errors.
Test RMSE	Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
SVGP	0.313	0.380	0.294	0.186	0.662	0.089	0.131	0.122	0.511	0.797
ODSVGP	0.321	0.373	0.222	0.175	0.667	0.093	0.087	0.121	0.534	0.794
DCSVGP	0.156	0.379	0.286	0.150	0.604	0.086	0.039	0.121	0.434	0.777
PPGPR	0.306	0.392	0.377	0.282	0.659	0.091	0.205	0.125	0.552	0.780
ODPPGPR	0.333	0.376	0.277	0.394	0.647	0.090	0.092	0.123	0.565	0.778
DCPPGPR	0.178	0.395	0.348	0.226	0.632	0.089	0.042	0.124	0.543	0.779
Test NLL	Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
SVGP	0.331	0.452	0.207	-0.188	1.013	-1.018	-0.409	-0.683	0.752	1.192
ODSVGP	0.278	0.433	-0.094	-0.354	1.011	-1.029	-0.578	-0.694	0.797	1.187
DCSVGP	-0.373	0.450	0.165	-0.502	0.919	-1.047	-1.293	-0.697	0.586	1.166
PPGPR	-0.056	0.377	-0.715	-0.772	0.804	-1.606	-0.906	-1.801	0.260	1.112
ODPPGPR	-0.064	0.382	-0.842	-0.972	0.812	-1.603	-0.937	-1.791	0.316	1.111
DCPPGPR	-0.588	0.403	-0.901	-1.067	0.854	-1.648	-1.574	-1.904	0.227	1.109
Table 3: Test RMSE and NLL on 10 regression datasets (lower is better). Results are averaged over 10 random train/validation/test splits. Statistical significance is indicated by bold. See the supplement for standard errors.
Test RMSE	Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
SVGP-DKL	0.0614	0.343	0.013	0.067	0.598	0.0864	0.0189	0.120	0.329	0.773
SVGP-DCDKL	0.0513	0.343	0.011	0.047	0.577	0.0855	0.0223	0.119	0.288	0.774
PPGPR-DKL	0.0847	0.343	0.030	0.082	0.597	0.0871	0.0176	0.121	0.375	0.771
PPGPR-DCDKL	0.0593	0.346	0.027	0.061	0.594	0.0870	0.0171	0.120	0.350	0.771
Test NLL	Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
SVGP-DKL	-1.388	0.347	-2.590	-1.227	0.905	-1.045	-2.550	-0.712	0.320	1.161
SVGP-DCDKL	-1.600	0.348	-2.786	-1.637	0.888	-1.052	-2.379	-0.713	0.173	1.164
PPGPR-DKL	-2.324	0.303	-3.096	-1.739	0.712	-1.627	-2.625	-1.951	-0.232	1.098
PPGPR-DCDKL	-2.584	0.358	-3.192	-2.186	1.194	-1.650	-2.676	-1.911	-0.340	1.117
SVGP-DCDKL and PPGPR-DCDKL. We then compare SVGP-DCDKL with baseline model SVGP-DKL [53] and compare PPGPR-DCDKL with baseline model PPGPR-DKL [22]. Here, all models learn inducing points in the input space rather than in the feature space and we refer to App. B.1.6 for more discussion on this choice and supporting results. Table 3 shows the test RMSE and NLL and we observe that SVGP-DCDKL yields better (or equivalent) RMSE and NLL on all datasets but the Slice dataset; PPGPR-DCDKL always yields better (or equivalent) RMSE and only obtains worse NLL on 3 datasets due to an overfitting issue similar to DCPPGPR.
5.2 Ablation Study On  $\beta_{2}$
In Sec. 5.1, we evaluate decoupled models with fixed  $\beta_{2} = 1\mathrm{e - }3$  and observe mostly superior performance except for occasional overfitting with DCPPGPR and PPGPR-DCDKL. Here, we study how the regularization parameter  $\beta_{2}$  affects model performance. We evaluate DCPPGPR with  $\beta_{2}\in \{0,0.001,0.005,0.01,0.05,0.1,0.5,1.0\}$  on two datasets-Pol and Protein- that represent our general findings.
The first row of Fig. 2 shows performance on Pol: DCPPGPR gets lower RMSE and NLL with decreasing  $\beta_{2}$ , and  $\beta_{2} \rightarrow 0$  is desirable. We also see how the decoupled lengthscales  $l_{\mathrm{mean}}$  and  $l_{\mathrm{covar}}$  diverge from PPGPR's lengthscale as  $\beta_{2}$  decreases. The second row of Fig. 2 shows results on Protein: DCPPGPR gets better RMSE but worse NLL as  $\beta_{2}$  decreases and thus overfits when  $\beta_{2} < 0.1$ . In this case,  $\beta_{2} = 0.1$  is the best. We refer to App. B.2 for more supporting results, and conclude that 1) small  $\beta_{2}$  or even 0 is usually ideal and 2) adding back some small regularization  $\beta_{2} = 0.1$  appears to resolve the instances of overfitting we noticed.
Figure 2: We evaluate DCPPGPR with varying  $\beta_{2}$  on dataset Pol (first row) and Protein (second row). Results are averaged over 10 random dataset splits with standard errors included. Solid blue lines show DCPPGPR with nonzero  $\beta_{2}$  and green dashed lines show DCPPGPR with  $\beta_{2} = 0$ . Baseline PPGPR is shown in orange dotted line. First two columns contain test RMSE and NLL (lower is better). Last two columns show the learned decoupled lengthscales  $l_{\mathrm{mean}}$  and  $l_{\mathrm{covar}}$ . PPGPR has equal  $l_{\mathrm{mean}}$  and  $l_{\mathrm{covar}}$  due to no decoupling. See Sec. 5.2 for details.
Figure 3: Optimization performance in terms of objective values on tasks Rover (dim=60), Lunar (dim=12) and Ranolazine (dim=256) is shown (higher is better). We compare baseline GP models (SVGP or PPGPR-DKL) with their decoupled counterparts (DCSVGP or PPGPR-DCDKL). Decoupling conditionals results in significant improvement on all tasks. See Sec. 5.3 for details.
5.3 Applications to Bayesian Optimization (BO)
In this section, we apply our decoupled variational GPs to BO tasks [13]. We consider 3 real BO tasks: the rover trajectory planning problem (Rover) [12, 52], the lunar landing reinforcement learning (Lunar) [12], and a challenging molecule design task Ranolazine MPO (Ranolazine) [5, 33]. We take TuRBO [12] as the BO algorithm and use different variational GPs in TuRBO. On Rover and Lunar, we use SVGP with TuRBO and compare SVGP with DCSVGP. On Ranolazine we first reduce the problem dimension to 256 using a Variational Autoencoder (VAE) [26], then perform BO in the latent space using PPGPR-DKL with TuRBO [33]. We compare PPGPR-DKL with PPGPR-DCDKL on Ranolazine. Consistent with our ablation study in Sec. 5.2, we find that  $\beta_{2} = 0$  performs the best on Rover and Ranolazine while Lunar requires more regularization with  $\beta_{2} = 0.1$ . Fig. 3 summarizes optimization performance averaged on at least 10 runs - all decoupled models outperform their coupled counterparts, showing the advantage of decoupling conditionals in BO applications.
5.4 Plausible Extensions
We further explore extensions of decoupled conditionals to achieve more flexible predictive mean and variance of variational GPs. Beyond decoupling the same parametric form, we could model the predictive mean and covariance differently for a better fit. Despite lack of theoretical grounding, we evaluate two possibilities, both targeting a more complex predictive mean than the covariance.
Replace the predictive mean of SVGP by a neural network. In fact, the importance of the parametric form of the covariance field is mentioned in Jankowiak et al. [22]: "a good ansatz for the predictive variance is important for good uncertainty prediction". They suggest, but never implement
Table 4: Test NLL on 10 regression datasets (lower is better). Results are averaged over 10 random train/validation/test splits. Best ones with statistical significance are bold. See Sec. 5.4 for details.
Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
SVGP	0.331	0.452	0.207	-0.188	1.013	-1.018	-0.409	-0.683	0.752	1.192
DCSVGP	-0.373	0.450	0.165	-0.502	0.919	-1.047	-1.293	-0.697	0.586	1.166
NNSVGP	-1.050	0.419	-1.231	-0.827	0.895	-1.060	-0.948	-0.725	0.776	1.171
PPGPR	-0.056	0.377	-0.715	-0.772	0.804	-1.606	-0.906	-1.801	0.260	1.112
DCPPGPR	-0.588	0.403	-0.901	-1.067	0.854	-1.648	-1.574	-1.904	0.227	1.109
NNPPGPR	-1.169	0.552	-1.560	-1.074	0.766	-1.514	-1.369	-1.632	0.412	1.113
Table 5: Test RMSE and NLL on 10 regression datasets (lower is better). Results are averaged over 10 random train/validation/test splits. Best ones with statistical significance are bold. See Sec. 5.4 for details.
Test RMSE	Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
SVGP-DKL	0.0614	0.343	0.013	0.067	0.598	0.0864	0.0189	0.120	0.329	0.773
SVGP-DCDKL	0.0513	0.343	0.011	0.047	0.577	0.0855	0.0223	0.119	0.288	0.774
SVGP-MeanDKL	0.0576	0.344	0.027	0.048	0.576	0.0859	0.0259	0.119	0.285	0.771
PPGPR-DKL	0.0847	0.343	0.030	0.082	0.597	0.0871	0.0176	0.121	0.375	0.771
PPGPR-DCDKL	0.0593	0.346	0.027	0.061	0.594	0.0870	0.0171	0.120	0.350	0.771
PPGPR-MeanDKL	0.0736	0.344	0.055	0.062	0.594	0.0868	0.0223	0.121	0.349	0.771
Test NLL	Pol	Elevators	Bike	Kin40k	Protein	Keggdir	Slice	Keggundir	3Droad	Song
SVGP-DKL	-1.388	0.347	-2.590	-1.227	0.905	-1.045	-2.550	-0.712	0.320	1.161
SVGP-DCDKL	-1.600	0.348	-2.786	-1.637	0.888	-1.052	-2.379	-0.713	0.173	1.164
SVGP-MeanDKL	-1.450	0.349	-1.989	-1.645	0.883	-1.053	-1.777	-0.715	0.163	1.160
PPGPR-DKL	-2.324	0.303	-3.096	-1.739	0.712	-1.627	-2.625	-1.951	-0.232	1.098
PPGPR-DCDKL	-2.584	0.358	-3.192	-2.186	1.194	-1.650	-2.676	-1.911	-0.340	1.117
PPGPR-MeanDKL	-2.362	0.312	-2.831	-2.150	1.048	-1.666	-2.175	-1.906	-0.344	1.099
or evaluate, that one could replace the mean function with a neural network so that inducing points are only used for variance prediction. We empirically evaluate this idea, denoted as NNSVGP (and NNPPGPR). With a totally flexible predictive mean, NNSVGP and NNPPGPR yield best test RMSE over 9 out of 10 datasets (see supplement for the RMSE table). However, with the predictive mean fully independent of the GP framework, NNSVGP and NNPPGPR yield worse NLL typically. Table 4 reports test NLL, where results are generally mixed.
Simplified deep kernels for covariances. In the DKL setting, instead of using decoupled feature maps, one could use entirely different neural network architectures for the predictive mean and covariance. We empirically find that a simpler feature mapping, or even no feature mapping, for the covariance still make a plausible model with good performance but smaller training cost. We denote the method with no feature mapping for the covariance as SVGP-MeanDKL and PPGPR-MeanDKL.
We present results in Table 5. We observe SVGP-MeanDKL does not significantly outperform the decoupled models SVGP-DCDKL, but it improves baseline SVGP-DKL on 8 out of 10 datasets. PPGPR-MeanDKL performs similarly in terms of the RMSE metric (better than baseline PPGPR-DKL but worse than PPGPR-DCDKL). However, in terms of NLL, it gives poor uncertainty prediction (worse NLL). This suggests that it is more beneficial to have different models (decoupled feature extractors in SVGP-DCDKL) for conditional mean and covariance rather than only modeling the conditional mean (SVGP-MeanDKL) or having same models (SVGP-DKL), and SVGP-based model is more robust than the PPGPR-based model in uncertainty prediction.
Variational GPs scale approximate GP inference to millions of training examples but may yield suboptimal accuracy due to lack of model expressiveness or poor model fitting. We propose a simple idea to improve model expressiveness by decoupling the parametric form of the mean and variance in the conditionals of variational GPs. We derive an ELBO for our model with decoupled conditionals, which end up being similar to the ELBO of standard variational GP with an additional a regularization
term. Our method is simple yet effective, and it applies to various variational GPs. We provide two concrete examples, one decoupling kernel lengthscales in the basic variational GP setting and one decoupling the feature mapping in the deep kernel learning setting. Through extensive empirical study, we show that the decoupled conditionals effectively improve the accuracy of variational GPs in terms of both mean and uncertainty prediction. We also empirically study two plausible extensions of our method, motivated by the idea of modeling the mean and variance differently, but we conclude that they are not as effective as our decoupled models. Current limitations of our work and therefore future directions include but not limit to: 1) the application of decoupled conditionals to more SVGP-based models; 2) further generalization of flexible forms of training and testing conditionals that improve model performance; 3) the application of decoupled variational GP framework to tasks other than regression or BO, such as classification tasks.
We acknowledge support from Simons Foundation. This work was supported by a grant from the Simons Foundation (601956, DSB). JRG is supported by NSF award IIS-2145644.
[1] V. Adam, S. Eleftheriadis, A. Artemev, N. Durrande, and J. Hensman. Doubly sparse variational Gaussian processes. In International Conference on Artificial Intelligence and Statistics, pages 2874-2884. PMLR, 2020.
[2] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):859-877, 2017.
[3] K. Blomqvist, S. Kaski, and M. Heinonen. Deep convolutional Gaussian processes. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2019, Würzburg, Germany, September 16–20, 2019, Proceedings, Part II, pages 582–597. Springer, 2020.
[4] V. Borovitskiy, A. Terenin, P. Mostowsky, et al. Matérn Gaussian processes on Riemannian manifolds. Advances in Neural Information Processing Systems, 33:12426-12437, 2020.
[5] N. Brown, M. Fiscato, M. H. Segler, and A. C. Vaucher. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling, 59(3):1096-1108, 2019.
[6] D. Burt, C. E. Rasmussen, and M. Van Der Wilk. Rates of convergence for sparse variational Gaussian process regression. In International Conference on Machine Learning, pages 862-871. PMLR, 2019.
[7] D. R. Burt, C. E. Rasmussen, and M. Van Der Wilk. Convergence of sparse variational inference in Gaussian processes regression. The Journal of Machine Learning Research, 21(1):5120-5182, 2020.
[8] C.-A. Cheng and B. Boots. Variational inference for Gaussian process models with linear complexity. Advances in Neural Information Processing Systems, 30, 2017.
[9] L. Csato and M. Opper. Sparse representation for Gaussian process models. Advances in neural information processing systems, 13, 2000.
[10] D. Dua and C. Graff. UCI Machine Learning Repository. Irvine, CA: University of California, School of Information and Computer Science, 2017. URL http://archive.ics.uci.edu/ml.
[11] V. Dutordoir, N. Durrande, and J. Hensman. Sparse Gaussian processes with spherical harmonic features. In International Conference on Machine Learning, pages 2793-2802. PMLR, 2020.
[12] D. Eriksson, M. Pearce, J. R. Gardner, R. Turner, and M. Poloczek. Scalable global optimization via local Bayesian optimization. In Advances in Neural Information Processing Systems, volume 32, 2019.
[13] P. I. Frazier. Bayesian optimization. In *Recent advances in optimization and modeling of contemporary problems*, pages 255-278. Informs, 2018.
A Methodology
A.1 The Evidence Lower Bound (ELBO)
Here we provide detailed derivation of the evidence lower bound (ELBO) for the proposed decoupled SVGP model fitting. The goal of variational learning remains the same:
$$
\min  \operatorname {K L} (q (\mathbf {f}, \mathbf {f} _ {m}) | | p (\mathbf {f}, \mathbf {f} _ {m} | \mathbf {y})) = \min  \left(\mathbb {E} [ \log (q (\mathbf {f}, \mathbf {f} _ {m})) ] - \mathbb {E} [ \log (p (\mathbf {f}, \mathbf {f} _ {m}, \mathbf {y})) ] + \log (p (\mathbf {y}))\right),
$$
where in the standard case  $q(\mathbf{f}, \mathbf{f}_m) = p(\mathbf{f}|\mathbf{f}_m)\phi(\mathbf{f}_m)$ , but we further approximate the training conditional  $p(\mathbf{f}|\mathbf{f}_m)$  by a decoupled one  $\psi(\mathbf{f}|\mathbf{f}_m)$  and have  $q(\mathbf{f}, \mathbf{f}_m) = \psi(\mathbf{f}|\mathbf{f}_m)\phi(\mathbf{f}_m)$ . Following the standard variational inference [2], we have the ELBO for decoupled SVGP:
$$
\begin{array}{l} \begin{array}{l} \log (p (\mathbf {y})) \geq \operatorname {E L B O} (q) = \mathbb {E} [ \log (p (\mathbf {f}, \mathbf {f} _ {m}, \mathbf {y})) ] - \mathbb {E} [ \log (q (\mathbf {f}, \mathbf {f} _ {m})) ] \\ \quad \mathbb {E} [ \log ((\mathbf {r} ^ {\prime} | \mathbf {f}, \mathbf {f} _ {m})) ] = K I _ {1} ((\mathbf {f}, \mathbf {f} _ {m}) | | _ {L ^ {2}} (\mathbf {f}, \mathbf {f} _ {m})) \end{array} \tag {7} \\ = \mathbb {E} [ \log (p (\mathbf {y} | \mathbf {f}, \mathbf {f} _ {m})) ] - \mathrm {K L} (q (\mathbf {f}, \mathbf {f} _ {m}) | | p (\mathbf {f}, \mathbf {f} _ {m})). \\ \end{array}
$$
The first likelihood term in Eq. 7 is
$$
\begin{array}{l} \mathbb {E} [ \log (p (\mathbf {y} | \mathbf {f}, \mathbf {f} _ {m})) ] = \int \int \log (p (\mathbf {y} | \mathbf {f}, \mathbf {f} _ {m})) q (\mathbf {f}, \mathbf {f} _ {m}) d \mathbf {f} d \mathbf {f} _ {m} \\ = \int \phi (\mathbf {f} _ {m}) \left(\int \log (p (\mathbf {y} | \mathbf {f}, \mathbf {f} _ {m})) \psi (\mathbf {f} | \mathbf {f} _ {m}) d \mathbf {f}\right) d \mathbf {f} _ {m} \\ = \int \phi (\mathbf {f} _ {m}) \left(\int \log (p (\mathbf {y} | \mathbf {f})) \psi (\mathbf {f} | \mathbf {f} _ {m}) d \mathbf {f}\right) d \mathbf {f} _ {m} \\ = \sum_ {i = 1} ^ {n} \left\{\log \mathcal {N} (y _ {i} | \tilde {\mu} _ {\bf f} ({\bf x} _ {i}), \sigma^ {2}) - \frac {\tilde {\sigma} _ {\bf f} ({\bf x} _ {i}) ^ {2}}{2 \sigma^ {2}} \right\}, \\ \end{array}
$$
where the predictive mean  $\tilde{\mu}_{\mathbf{f}}(\mathbf{x}_i)$  and latent function variance  $\tilde{\sigma}_{\mathbf{f}}(\mathbf{x}_i)^2$  are
$$
\tilde {\mu} _ {\mathbf {f}} (\mathbf {x} _ {i}) = \mathbf {Q} _ {i m} \mathbf {Q} _ {m m} ^ {- 1} \mathbf {m}, \tilde {\sigma} _ {\mathbf {f}} (\mathbf {x} _ {i}) ^ {2} = \tilde {\mathbf {K}} _ {i i} + \mathbf {Q} _ {i m} \mathbf {Q} _ {m m} ^ {- 1} \mathbf {S} \mathbf {Q} _ {m m} ^ {- 1} \mathbf {Q} _ {m i}.
$$
The second KL divergence term in Eq. 7 is
$$
\begin{array}{l} \operatorname {K L} (q (\mathbf {f}, \mathbf {f} _ {m}) | | p (\mathbf {f}, \mathbf {f} _ {m})) = \int \psi (\mathbf {f} | \mathbf {f} _ {m}) \phi (\mathbf {f} _ {m}) \log \frac {\psi (\mathbf {f} | \mathbf {f} _ {m}) \phi (\mathbf {f} _ {m})}{p (\mathbf {f} | \mathbf {f} _ {m}) p (\mathbf {f} _ {m})} d \mathbf {f} d \mathbf {f} _ {m} \\ = \int \psi (\mathbf {f} | \mathbf {f} _ {m}) \phi (\mathbf {f} _ {m}) \log \frac {\psi (\mathbf {f} | \mathbf {f} _ {m})}{p (\mathbf {f} | \mathbf {f} _ {m})} d \mathbf {f} d \mathbf {f} _ {m} + \int \psi (\mathbf {f} | \mathbf {f} _ {m}) \phi (\mathbf {f} _ {m}) \log \frac {\phi (\mathbf {f} _ {m})}{p (\mathbf {f} _ {m})} d \mathbf {f} d \mathbf {f} _ {m} \\ = \int \phi (\mathbf {f} _ {m}) \mathrm {K L} \left(\psi (\mathbf {f} | \mathbf {f} _ {m}) | | p (\mathbf {f} | \mathbf {f} _ {m})\right) d \mathbf {f} _ {m} + \mathrm {K L} \left(\phi (\mathbf {f} _ {m}) | | p (\mathbf {f} _ {m})\right) \\ = \underbrace {\mathbb {E} _ {\phi (\mathbf {f} _ {m})} [ \mathrm {K L} (\psi (\mathbf {f} | \mathbf {f} _ {m}) | | p (\mathbf {f} | \mathbf {f} _ {m})) ]} _ {:= \Omega} + \mathrm {K L} (\phi (\mathbf {f} _ {m}) | | p (\mathbf {f} _ {m})). \\ \end{array}
$$
The KL divergence term is almost the same as that in the standard ELBO, with an additional term  $\Omega \coloneqq \int \phi (\mathbf{f}_m)\mathrm{KL}(\psi (\mathbf{f}|\mathbf{f}_m)||p(\mathbf{f}|\mathbf{f}_m))d\mathbf{f}_m$