SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xinran Zhu $^{1}$  Kaiwen Wu $^{2}$  Natalie Maus $^{2}$  Jacob R. Gardner $^{2}$  David Bindel $^{1}$
$^{1}$ Cornell University  $^{2}$ University of Pennsylvania
{xz584,bindel}@cornell.edu {kaiwenwu,nmaus,jacobrg}@seas.upenn.edu
Variational Gaussian processes (GPs) approximate exact GP inference by using a small set of inducing points to form a sparse approximation of the true posterior, with the fidelity of the model increasing with additional inducing points. Although the approximation error in principle can be reduced by using more inducing points, this leads to scaling optimization challenges and computational complexity. To achieve scalability, inducing point methods typically introduce conditional independencies and then approximations to the training and test conditional distributions. In this paper, we consider an alternative approach to modifying the training and test conditionals, in which we make them more flexible. In particular, we investigate decoupling the parametric form of the predictive mean and covariance in the conditionals, and learn independent parameters for predictive mean and covariance. We derive new evidence lower bounds (ELBO) under these more flexible conditionals, and provide two concrete examples of applying the decoupled conditionals. Empirically, we find this additional flexibility leads to improved model performance on a variety of regression tasks and Bayesian optimization (BO) applications.
Gaussian processes (GPs) are a powerful class of non-parametric probabilistic models [38]. Their flexibility and ability to make probabilistic predictions make them particularly useful in situations where uncertainty estimates are important [23, 30, 43, 45, 56]. Although elegant, exact GPs become intractable for large datasets since the computational cost scales cubically with the size of the dataset.
To overcome this limitation, various sparse approximate GP approaches have been developed, mostly relying on sparse approximations of the true posterior [28, 29, 36]. Among those, variational GPs have become increasingly popular because they enable stochastic minibatching and apply to both regression [17] and classification [18] tasks. However, the approximate posterior and variational inference leave an unavoidable gap from the exact model. Especially with too few inducing points, variational GPs could yield suboptimal accuracy performance due to a lack of expressiveness. Using more inducing points closes this gap in principle, but leads to additional computational complexity and optimization difficulties [15]. While several works have studied ways of allowing more or better placed inducing points to improve accuracy of variational GPs, [34, 41, 50, 57], it is nevertheless challenging to use more than a small fraction of the dataset size in inducing points.
In this work, we take a different approach to improving model expressiveness for better accuracy of variational GPs through decoupling conditionals. To start, we point out that inducing point approximations rely on two key conditionals (see Sec. 3.1) – the training and the test conditional, where the mean and covariance are parameterized by inducing points and kernel hyperparameters. Typically, prior works have focused on various relaxations of these conditionals that enable significant computational complexity benefits [36, 44, 46, 47, 48]. In this work, we consider alternative changes
(a) Test NLL: 0.60,  $l$ : 0.85
(b) Test NLL: -0.61,  $l_{\mathrm{mean}}$ : 0.27,  $l_{\mathrm{covar}}$ : 2.15
(c) Test NLL: -0.90,  $l$ : 0.31
Figure 1: We compare model fit on a 1D latent function using 100 training samples. Solid curves with shading area depict the predictive mean and  $95\%$  confidence interval. Using 10 inducing points, in **subplot** (a) SVGP underfits the latent function with large lengthscale  $l = 0.85$ ; while in **subplot** (b), our DCSVGP model (see Sec. 3.2) fits better and learns different decoupled lengthscales  $l_{\mathrm{mean}}$  and  $l_{\mathrm{covar}}$ . Using 100 inducing points, in **subplot** (c) and (d), both models fits well with similar lengthscales around 0.3. See Sec. 1 for more details.
(d) Test NLL: -0.93,  $l_{\mathrm{mean}}$ : 0.33,  $l_{\mathrm{covar}}$ : 0.31
that increase the flexibility of these conditionals. In Sec. 3.2, we describe two concrete examples of this idea: decoupling kernel lengthscales and decoupling entire deep feature extractors.
As a simple illustration, Fig. 1 illustrates, for example, how decoupled lengthscales improve model fitting. Our model DCSVGP (see Sec. 3.2) learns decoupled lengthscales  $l_{\mathrm{mean}}$  and  $l_{\mathrm{covar}}$  for mean and covariance respectively, and we compare with baseline SVGP (see Sec. 2) which learns one lengthscale  $l$ .
To summarize our contributions: 1) We propose decoupled conditionals in variational GPs to improve model expressiveness for better accuracy. 2) We show that our idea is compatible with the variational framework and rigorously derive an ELBO for variational inference with decoupled conditionals. 3) We provide two concrete examples of applying decoupled conditionals and empirically show the superior performance of our models through extensive regression tasks and BO applications.
We assume familiarity with GPs [38] and briefly introduce them for notational clarity. Given observation locations  $\mathbf{X} = \{\mathbf{x}_i\}_{i=1}^n \subset \mathbb{R}^d$ , a GP prior induces a multivariate Normal belief for latent function values  $\mathbf{f} = \{f(\mathbf{x}_i)\}$ :  $\mathbf{f} \sim \mathcal{N}(\mu_{\mathbf{X}}, \mathbf{K}_{nn})$ , where  $\mu_{\mathbf{X}}, \mathbf{K}_{nn}$  are the mean values and covariance matrix at data  $\mathbf{X}$ . Givend observations  $\mathbf{y} = \mathbf{f} + \epsilon$  with Gaussian noise  $\epsilon \sim \mathcal{N}(0, \sigma^2 I)$ , the posterior distribution of the function value  $\mathbf{f}^*$  at a new data point  $x^*$  is  $p(\mathbf{f}^| \mathbf{y}) = \mathcal{N}(\mu^*, \Sigma^{**})$ , where
$$
\mu^ {*} = \mu (\mathbf {x} ^ {*}) + \mathbf {K} _ {* n} (\mathbf {K} _ {n n} + \sigma^ {2} \mathbf {I}) ^ {- 1} (\mathbf {y} - \mu_ {\mathbf {X}}),
$$
$$
\Sigma^ {* *} = \mathbf {K} (\mathbf {x} ^ {*}, \mathbf {x} ^ {*}) - \mathbf {K} _ {* n} (\mathbf {K} _ {n n} + \sigma^ {2} \mathbf {I}) ^ {- 1} \mathbf {K} _ {* n} ^ {T}.
$$
Model hyperparameters such as kernel hyperparameters and noise  $\sigma$  are typically estimated by Maximum Likelihood using standard numerical solvers such as LBFGS [35]. If no approximations are used, each evaluation to optimize the log marginal likelihood function costs  $\mathcal{O}(n^3)$  flops and  $\mathcal{O}(n^2)$  memory, thus motivating approximate methods for large training datasets.
2.1 Sparse Gaussian Processes
To overcome the scalability limitations of exact GPs, many authors have proposed a variety of sparse GPs by introducing inducing points  $\mathbf{Z} = \{\mathbf{z}_i\}_{i=1}^m$  [17, 44, 47, 48, 49]. Inducing points are associated with inducing values  $\mathbf{f}_m$ , which represent latent function values at  $\mathbf{Z}$  under the same GP assumption.
Although inducing values  $\mathbf{f}_m$  are marginalized out in the predictive distribution, they typically reduce training costs from  $\mathcal{O}(n^3)$  to  $\mathcal{O}(n^2 m + m^3)$  for each gradient step, where  $m \ll n$ .
SGPR. Based on inducing point methods, Sparse Gaussian Process Regression (SGPR) [49] further introduces variational inference into sparse GPs. Assuming that inducing data is adequate for inference, Titsias [49] further introduces a variational distribution  $\phi (\mathbf{f}_m)\sim \mathcal{N}(\mathbf{m},\mathbf{S})$  that approximates the posterior  $p(\mathbf{f}_m|\mathbf{y})$ . Thus, the predictive density can be approximated as
$$
q (\mathbf {f} ^ {*}) = \int p (\mathbf {f} ^ {*} | \mathbf {f} _ {m}) p (\mathbf {f} _ {m} | \mathbf {y}) d \mathbf {f} _ {m} = \int p (\mathbf {f} ^ {*} | \mathbf {f} _ {m}) \phi (\mathbf {f} _ {m}) d \mathbf {f} _ {m} = \mathcal {N} (\mu_ {\mathbf {f}} (\mathbf {x} ^ {*}), \sigma_ {\mathbf {f}} (\mathbf {x} ^ {*})) ^ {2}).
$$
Here the predictive mean  $\mu_{\mathbf{f}}(\mathbf{x}^{*})$  and the latent function variance  $\sigma_{\mathbf{f}}(\mathbf{x}^{*})^{2}$  are:
$$
\mu_ {\mathbf {f}} \left(\mathbf {x} ^ {*}\right) = \mathbf {K} _ {* m} \mathbf {K} _ {m m} ^ {- 1} \mathbf {m}, \sigma_ {\mathbf {f}} \left(\mathbf {x} ^ {*}\right) ^ {2} = \tilde {\mathbf {K}} _ {* *} + \mathbf {K} _ {* m} \mathbf {K} _ {m m} ^ {- 1} \mathbf {S} \mathbf {K} _ {m m} ^ {- 1} \mathbf {K} _ {m *}, \tag {1}
$$
where  $\tilde{\mathbf{K}}_{**} = \mathbf{K}_{**} - \mathbf{K}_{*m}\mathbf{K}_{mm}^{-1}\mathbf{K}_{*m}^{T}$ . The variational distribution  $\phi (\mathbf{f}_m) = \mathcal{N}(\mathbf{m},\mathbf{S})$  is then learned by maximizing the variational ELBO [18, 21], which is a lower bound on the log marginal likelihood:
$$
\log p (\mathbf {y}) \geq \mathbb {E} _ {q (\mathbf {f})} [ \log p (\mathbf {y} | \mathbf {f}) ] - \mathrm {K L} [ \phi (\mathbf {f} _ {m}) | | p (\mathbf {f} _ {m}) ],
$$
where  $q(\mathbf{f}) = \int p(\mathbf{f}|\mathbf{f}_m)\phi (\mathbf{f}_m)d\mathbf{f}_m$ , and KL  $[\phi (\mathbf{f}_m)||p(\mathbf{f}_m)]$  is the KL divergence [27].
SVGP. To enable data subsampling during training, SVGP [17] avoids analytic solutions to the ELBO, but rather decomposes the ELBO as a sum of losses over training labels and enables stochastic gradient descent (SGD) [39] training:
$$
\mathrm {E L B O} _ {\mathrm {S V G P}} = \sum_ {i = 1} ^ {n} \left\{\log \mathcal {N} \left(y _ {i} \mid \mu_ {\mathbf {f}} \left(\mathbf {x} _ {i}\right), \sigma^ {2}\right) - \frac {\sigma_ {\mathbf {f}} \left(\mathbf {x} _ {i}\right) ^ {2}}{2 \sigma^ {2}} \right\} - \beta \mathrm {K L} [ \phi (\mathbf {f} _ {m}) | | p (\mathbf {f} _ {m}) ], \tag {2}
$$
where  $\mu_{\mathbf{f}}(\cdot),\sigma_{\mathbf{f}}(\cdot)^2$  are the predictive mean and latent function variance from Eq. 1 respectively, and  $\beta$  is an optional regularization parameter for the KL divergence and can be tuned in practice [22].
PPGPR. To achieve heteroscedastic modeling and improve predictive variances, the Parametric Gaussian Process Regressor (PPGPR) [22] targets the predictive distribution directly in the loss function. It shares the same predictive mean  $\mu_{\mathbf{f}}(\cdot)$  and latent function variance  $\sigma_{\mathbf{f}}(\cdot)^{2}$  as SVGP, but uses a slightly different stochastic ELBO loss:
$$
\mathrm {E L B O} _ {\mathrm {P P G P R}} = \sum_ {i = 1} ^ {n} \log \mathcal {N} \left(y _ {i} \mid \mu_ {\mathbf {f}} (\mathbf {x} _ {i}), \sigma^ {2} + \sigma_ {\mathbf {f}} (\mathbf {x} _ {i}) ^ {2}\right) - \beta \mathrm {K L} \left[ \phi \left(\mathbf {f} _ {m}\right) | | p \left(\mathbf {f} _ {m}\right) \right].
$$
Whitening. In practice, the variational distribution  $\phi (\mathbf{f}_m)$  is usually "whitened" to accelerate the optimization of the variational distribution [32]. Conventionally, the whitened variational distribution is  $\tilde{\phi} (\mathbf{f}_m) = \mathcal{N}(\tilde{\mathbf{m}},\tilde{\mathbf{S}}) = \mathcal{N}(\mathbf{K}_{mm}^{-1 / 2}\mathbf{m},\mathbf{K}_{mm}^{-1 / 2}\mathbf{SK}_{mm}^{-1 / 2})$ , where  $\mathbf{K}_{mm}^{1 / 2}$  is the square root of  $\mathbf{K}_{mm}$ . With whitening, the KL divergence and predictive distribution in ELBO (Eq. 2) are both simplified:
$$
\operatorname {K L} \left[ \phi \left(\mathbf {f} _ {m}\right) | | p \left(\mathbf {f} _ {m}\right) \right] = \operatorname {K L} \left(\tilde {\phi} \left(\mathbf {f} _ {m}\right) | | p _ {0} \left(\mathbf {f} _ {m}\right)\right), \text {w h e r e} p _ {0} \left(\mathbf {f} _ {m}\right) = \mathcal {N} (\mathbf {0}, \mathbf {I}),
$$
$$
\mu_ {\mathbf {f}} (\mathbf {x} _ {i}) = \mathbf {K} _ {i m} \mathbf {K} _ {m m} ^ {- 1 / 2} \tilde {\mathbf {m}}, \sigma_ {\mathbf {f}} (\mathbf {x} _ {i}) ^ {2} = \tilde {\mathbf {K}} _ {i i} + \mathbf {K} _ {i m} \mathbf {K} _ {m m} ^ {- 1 / 2} \tilde {\mathbf {S}} \mathbf {K} _ {m m} ^ {- 1 / 2} \mathbf {K} _ {m i}.
$$
Here, we present variational GPs with decoupled conditionals. In Sec. 3.1 and 3.2, we introduce decoupled conditionals under a unifying framework for approximate GPs, followed by two concrete examples. In Sec. 3.3, we derive an ELBO to do variational inference with decoupled conditionals.
3.1 Conditionals of Approximate GPs
Quinonero-Candela and Rasmussen [36] introduced a unifying framework for sparse GPs. Through the framework, an approximate GP can be interpreted as "an exact inference with an approximated
Table 1: Exact and approximate conditionals. See Sec. 3.1 and Sec. 3.2 for details. Here  $\tilde{\mathbf{K}}_{nn} = \mathbf{K}_{nn} - \mathbf{K}_{nm}\mathbf{K}_{mm}^{-1}\mathbf{K}_{mn}$  and  $\tilde{\mathbf{K}}_{**} = \mathbf{K}_{**} - \mathbf{K}_{*m}\mathbf{K}_{mm}^{-1}\mathbf{K}_{m*}$ .
Training Conditionals	Test Conditionals
DTC	q(f|f_m) = N(K_nm K^{-1}_{mm} f_m, 0)	p(f*|f_m) = N(K_{*m}K^{-1}_{mm}f_m,\tilde{K}_{**})
FITC	q(f|f_m) = N(K_nm K^{-1}_{mm} f_m, diag[\tilde{K}_{nn}])	p(f*|f_m) = N(K_{*m}K^{-1}_{mm}f_m,\tilde{K}_{**})
SVGP	p(f|f_m) = N(K_nm K^{-1}_{mm} f_m,\tilde{K}_{nn})	p(f*|f_m) = N(K_{*m}K^{-1}_{mm}f_m,\tilde{K}_{**})
Ours	ψ(f|f_m) = N(Q_nm Q^{-1}_{mm} f_m,\tilde{K}_{nn})	ψ(f*|f_m) = N(Q_{*m}Q^{-1}_{mm}f_m,\tilde{K}_{**})
prior" in contrast to "an approximate inference with an exact prior". With inducing points  $\mathbf{Z} = \{\mathbf{z}_i\}_{i=1}^m$  and inducing values  $\mathbf{f}_m$ , most sparse GPs approximate the joint Gaussian prior as:
$$
p (\mathbf {f} ^ {*}, \mathbf {f}) = \int p (\mathbf {f} ^ {*}, \mathbf {f} | \mathbf {f} _ {m}) p (\mathbf {f} _ {m}) d \mathbf {f} _ {m} \approx \int q (\mathbf {f} ^ {*} | \mathbf {f} _ {m}) q (\mathbf {f} | \mathbf {f} _ {m}) p (\mathbf {f} _ {m}) d \mathbf {f} _ {m} = q (\mathbf {f} ^ {*}, \mathbf {f}),
$$
where dependencies between training data  $\mathbf{f}$  and test data  $\mathbf{f}^*$  are induced by inducing values  $\mathbf{f}_m$ . Different approximations can be made of the inducing training conditionals  $q(\mathbf{f}|\mathbf{f}_m)$  and the inducing test conditionals  $q(\mathbf{f}^*|\mathbf{f}_m)$ . Table 1 provides the exact expressions of the two conditionals and examples of approximations used in various approximate GP methods. For example, the Deterministic Training Conditional (DTC) approximation uses a deterministic training conditional and the exact test conditional [44]; The Fully Independent Training Conditional (FITC) approximation uses an approximate training conditional with diagonal corrections and the exact test conditional [48].
3.2 Decoupled Conditionals and Prediction
In this paper, we consider augmenting the exact conditionals by a more flexible form that decouples the kernel hyperparameters in the mean and covariance in both conditionals:
$$
\text {t r a i n i n g c o n d i t i o n a l} \psi (\mathbf {f} | \mathbf {f} _ {m}) = \mathcal {N} \left(\mathbf {Q} _ {n m} \mathbf {Q} _ {m m} ^ {- 1} \mathbf {f} _ {m}, \tilde {\mathbf {K}} _ {n n}\right) \approx p (\mathbf {f} | \mathbf {f} _ {m}),
$$
$$
\text {t e s t} \quad \psi (\mathbf {f} ^ {*} | \mathbf {f} _ {m}) = \mathcal {N} \left(\mathbf {Q} _ {* m} \mathbf {Q} _ {m m} ^ {- 1} \mathbf {f} _ {m}, \tilde {\mathbf {K}} _ {* *} \right) \approx p \left(\mathbf {f} ^ {*} | \mathbf {f} _ {m}\right), \tag {3}
$$
where the  $\mathbf{Q}$  and  $\mathbf{K}$  matrices are formed by the same family of kernel functions, but some kernel hyperparameters are decoupled. See Table 1 for a comparison with other approximations. Decoupled conditionals improve model flexibility without relying on more inducing points and it applies to various SVGP-based models. We provide two examples of decoupled models that we will evaluate.
Example 1: Decoupled lengthscales. By decoupling the kernel lengthscale  $l$  into  $l_{\mathrm{mean}}$  for the mean and  $l_{\mathrm{covar}}$  for the covariance, we enable the model to learn in settings where the function value changes more rapidly than the variance. In Eq. 3, this corresponds to the Q and K kernel matrices being formed using separate lengthscales. For example, an RBF kernel gives
$$
\mathbf {Q} _ {m m} = \left[ \exp \left(- \frac {\| \mathbf {z} _ {i} - \mathbf {z} _ {j} \| ^ {2}}{2 l _ {\mathrm {m e a n}} ^ {2}}\right) \right] _ {\mathbf {z} _ {i}, \mathbf {z} _ {j} \in \mathbf {Z}}, \mathbf {K} _ {m m} = \left[ \exp \left(- \frac {\| \mathbf {z} _ {i} - \mathbf {z} _ {j} \| ^ {2}}{2 l _ {\mathrm {c o v a r}} ^ {2}}\right) \right] _ {\mathbf {z} _ {i}, \mathbf {z} _ {j} \in \mathbf {Z}}.
$$
We will denote the application of decoupled lengthscales to SVGP as Decoupled Conditional SVGP (DCSVGP).
Example 2: Decoupled deep kernel learning. Wilson et al. [53] proposed deep kernel learning (DKL) which stacks a deep neural network feature extractor  $h$  with a GP layer. Combining DKL with variational GP models is straightforward, with the feature extractor  $h$  learned through ELBO with all other model (hyper)parameters [54]. The feature extractor  $h$  can then be decoupled: one  $h_{\mathrm{mean}}$  for the mean and one  $h_{\mathrm{covar}}$  for the covariance. Again using the RBF kernel as a base example:
$$
\mathbf {Q} _ {m m} = \left[ \exp \left(- \frac {\| h _ {\mathrm {m e a n}} (\mathbf {z} _ {i}) - h _ {\mathrm {m e a n}} (\mathbf {z} _ {j}) \| ^ {2}}{2 l ^ {2}}\right) \right] _ {\mathbf {z} _ {i}, \mathbf {z} _ {j} \in \mathbf {Z}},
$$
$$
\mathbf {K} _ {m m} = \left[ \exp \left(- \frac {\| h _ {\mathrm {c o v a r}} (\mathbf {z} _ {i}) - h _ {\mathrm {c o v a r}} (\mathbf {z} _ {j}) \| ^ {2}}{2 l ^ {2}}\right) \right] _ {\mathbf {z} _ {i}, \mathbf {z} _ {j} \in \mathbf {Z}}.
$$
We denote the application of decoupled deep feature extractors to SVGP as SVGP-DCDKL.
Prediction. Using decoupled conditionals, the predictive posterior at a new point  $\mathbf{x}^*$  is
$$
q (\mathbf {y} ^ {*}) = \int p (\mathbf {y} ^ {*} | \mathbf {f} ^ {*}) \psi (\mathbf {f} ^ {*} | \mathbf {f} _ {m}) \phi (\mathbf {f} _ {m}) d \mathbf {f} _ {m} d \mathbf {f} ^ {*} = \mathcal {N} (\mathbf {y} ^ {*} | \tilde {\mu_ {\mathbf {f}}} (\mathbf {x} ^ {*}), \tilde {\sigma_ {\mathbf {f}}} (\mathbf {x} ^ {*})) ^ {2} + \sigma^ {2} \mathbf {I}),
$$
where the predictive mean  $\tilde{\mu}_{\mathbf{f}}(\mathbf{x}^{*})$  and latent function variance  $\tilde{\sigma}_{\mathbf{f}}(\mathbf{x}^{*})^{2}$  are similar to Eq. 1:
$$
\tilde {\mu} _ {\mathbf {f}} \left(\mathbf {x} ^ {*}\right) = \mathbf {Q} _ {* m} \mathbf {Q} _ {m m} ^ {- 1} \mathbf {m}, \quad \tilde {\sigma} _ {\mathbf {f}} \left(\mathbf {x} ^ {*}\right) ^ {2} = \tilde {\mathbf {K}} _ {* *} + \mathbf {Q} _ {* m} \mathbf {Q} _ {m m} ^ {- 1} \mathbf {S} \mathbf {Q} _ {m m} ^ {- 1} \mathbf {Q} _ {m *}. \tag {4}
$$
3.3 The Evidence Lower Bound (ELBO)
In this section, we derive an ELBO for DCSVGP model fitting (other examples of decoupled models follow the same derivation). All model parameters and hyperparameters are learned by maximizing the resulting ELBO. Following the standard variational inference [2], we approximate the true posterior distribution  $p(\mathbf{f}, \mathbf{f}_m | \mathbf{y})$  by the variational distribution  $q(\mathbf{f}, \mathbf{f}_m)$  and minimize the KL divergence: min  $\mathrm{KL}(q(\mathbf{f}, \mathbf{f}_m) || p(\mathbf{f}, \mathbf{f}_m | \mathbf{y}))$ . In the standard case,  $q(\mathbf{f}, \mathbf{f}_m) = p(\mathbf{f} | \mathbf{f}_m) \phi(\mathbf{f}_m)$ , but a decoupled model has  $q(\mathbf{f}, \mathbf{f}_m) = \psi(\mathbf{f} | \mathbf{f}_m) \phi(\mathbf{f}_m)$  since it further approximates the training conditional  $p(\mathbf{f} | \mathbf{f}_m)$  by a decoupled one  $\psi(\mathbf{f} | \mathbf{f}_m)$ . This difference leads to the following ELBO for the decoupled model:
$$
\begin{array}{l} \log (p (\mathbf {y})) \geq \operatorname {E L B O} (q) = \mathbb {E} [ \log (p (\mathbf {y} | \mathbf {f}, \mathbf {f} _ {m})) ] - \operatorname {K L} (q (\mathbf {f}, \mathbf {f} _ {m}) | | p (\mathbf {f}, \mathbf {f} _ {m})) \\ = \sum_ {i = 1} ^ {n} \left\{\log \mathcal {N} \left(y _ {i} | \tilde {\mu} _ {\mathbf {f}} (\mathbf {x} _ {i}), \sigma^ {2}\right) - \frac {\tilde {\sigma} _ {\mathbf {f}} (\mathbf {x} _ {i}) ^ {2}}{2 \sigma^ {2}} \right\} - \int \psi (\mathbf {f} | \mathbf {f} _ {m}) \phi (\mathbf {f} _ {m}) \log \frac {\psi (\mathbf {f} | \mathbf {f} _ {m}) \phi (\mathbf {f} _ {m})}{p (\mathbf {f} | \mathbf {f} _ {m}) p (\mathbf {f} _ {m})} d \mathbf {f} d \mathbf {f} _ {m} \\ = \sum_ {i = 1} ^ {n} \left\{\log \mathcal {N} (y _ {i} | \tilde {\mu} _ {\mathbf {f}} (\mathbf {x} _ {i}), \sigma^ {2}) - \frac {\tilde {\sigma} _ {\mathbf {f}} (\mathbf {x} _ {i}) ^ {2}}{2 \sigma^ {2}} \right\} - \mathrm {K L} (\phi (\mathbf {f} _ {m}) | | p (\mathbf {f} _ {m})) - \int \psi (\mathbf {f} | \mathbf {f} _ {m}) \phi (\mathbf {f} _ {m}) \log \frac {\psi (\mathbf {f} | \mathbf {f} _ {m})}{p (\mathbf {f} | \mathbf {f} _ {m})} d \mathbf {f} d \mathbf {f} _ {m} \\ = \sum_ {i = 1} ^ {n} \left\{\log \mathcal {N} (y _ {i} | \tilde {\mu} _ {\mathbf {f}} (\mathbf {x} _ {i}), \sigma^ {2}) - \frac {\tilde {\sigma} _ {\mathbf {f}} (\mathbf {x} _ {i}) ^ {2}}{2 \sigma^ {2}} \right\} - \mathrm {K L} (\phi (\mathbf {f} _ {m}) | | p (\mathbf {f} _ {m})) - \underbrace {\mathbb {E} _ {\boldsymbol {\phi} (\mathbf {f} _ {m})} [ \mathrm {K L} (\psi (\mathbf {f} | \mathbf {f} _ {m}) | | p (\mathbf {f} | \mathbf {f} _ {m})) ]} _ {:= \Omega}. \\ \end{array}
$$
We refer to App. A.1 for additional derivation details. Adding regularization parameters  $\beta_{1}$  and  $\beta_{2}$  to the KL divergence terms as is often done in practice, the ELBO for DCSVGP is
$$
\mathrm {E L B O} _ {\mathrm {D C S V G P}} = \sum_ {i = 1} ^ {n} \left\{\log \mathcal {N} \left(y _ {i} \mid \tilde {\mu} _ {\mathbf {f}} \left(\mathbf {x} _ {i}\right), \sigma^ {2}\right) - \frac {\tilde {\sigma} _ {\mathbf {f}} \left(\mathbf {x} _ {i}\right) ^ {2}}{2 \sigma^ {2}} \right\} - \beta_ {1} \mathrm {K L} [ \phi (\mathbf {f} _ {m}) | | p (\mathbf {f} _ {m}) ] - \beta_ {2} \Omega \tag {5}
$$
where the predictive mean  $\tilde{\mu}_{\mathbf{f}}(\mathbf{x}_i)$  and latent function variance  $\tilde{\sigma}_{\mathbf{f}}(\mathbf{x}_i)^2$  are same as Eq. 4
$$
\tilde {\mu} _ {\mathbf {f}} \left(\mathbf {x} _ {i}\right) = \mathbf {Q} _ {i m} \mathbf {Q} _ {m m} ^ {- 1} \mathbf {m}, \quad \tilde {\sigma} _ {\mathbf {f}} \left(\mathbf {x} _ {i}\right) ^ {2} = \tilde {\mathbf {K}} _ {i i} + \mathbf {Q} _ {i m} \mathbf {Q} _ {m m} ^ {- 1} \mathbf {S} \mathbf {Q} _ {m m} ^ {- 1} \mathbf {Q} _ {m i}. \tag {6}
$$
The explicit expression of  $\Omega$ . The  $\Omega$  term can be computed explicitly (see App. A.1):
$$
\Omega = \mathbb {E} _ {\boldsymbol {\phi} (\mathbf {f} _ {m})} [ \mathbf {K L} (\boldsymbol {\psi} (\mathbf {f} | \mathbf {f} _ {m}) | | p (\mathbf {f} | \mathbf {f} _ {m})) ] = \frac {1}{2} \mathbb {E} [ \mathbf {f} _ {m} ^ {T} \mathbf {T} \mathbf {f} _ {m} ] = \frac {1}{2} \left(\mathbf {T r} (\mathbf {T S}) + \mathbf {m} ^ {T} \mathbf {T m}\right),
$$
where  $\mathbf{T} = \mathbf{A}^T\bar{\mathbf{K}}_{nn}^{-1}\mathbf{A},\mathbf{A} = \mathbf{Q}_{nm}\mathbf{Q}_{mm}^{-1} - \mathbf{K}_{nm}\mathbf{K}_{mm}^{-1}$ , and  $\phi (\mathbf{f}_m) = \mathcal{N}(\mathbf{m},\mathbf{S})$
Comparing  $\mathbf{ELBO}_{\mathrm{DCSVGP}}$  and  $\mathbf{ELBO}_{\mathrm{SVGP}}$ . The  $\mathbf{ELBO}_{\mathrm{SVGP}}$  in Eq. 2 and  $\mathbf{ELBO}_{\mathrm{DCSVGP}}$  in Eq. 5 both consist of an approximate likelihood term and a KL divergence part. There are two differences: 1)  $\mathbf{ELBO}_{\mathrm{DCSVGP}}$  involves different predictive mean  $\tilde{\mu}_{\mathbf{f}}(\mathbf{x}_i)$  and variance  $\bar{\sigma}_{\mathbf{f}}(\mathbf{x}_i)^2$  derived from decoupled conditionals, see Eq. 6; 2)  $\mathbf{ELBO}_{\mathrm{DCSVGP}}$  contains an additional KL divergence term  $\Omega$ , which is an expected KL divergence of the two training conditionals over the variational distribution  $\phi(\mathbf{f}_m)$ , regularizing the difference between the decoupled conditional  $\psi(\mathbf{f}|\mathbf{f}_m)$  and the exact one  $p(\mathbf{f}|\mathbf{f}_m)$ .