Title: Variational Gaussian Processes with Decoupled Conditionals
Venue: NeurIPS 2023
Accepted: yes
Keywords: Gaussian processes

variational inference

variational Gaussian processes

Bayesian optimization
Abstract: Variational Gaussian processes (GPs) approximate exact GP inference by using a small set of inducing points to form a sparse approximation of the true posterior, with the fidelity of the model increasing with additional inducing points. Although the approximation error in principle can be reduced through the use of more inducing points, this leads to scaling optimization challenges and computational complexity. To achieve scalability, inducing point methods typically introduce conditional independencies and then approximations to the training and test conditional distributions. In this paper, we consider an alternative approach to modifying the training and test conditionals, in which we make them more flexible. In particular, we investigate decoupling the parametric form of the predictive mean and covariance in the conditionals, and learn independent parameters for predictive mean and covariance. We derive new evidence lower bounds (ELBO) under these more flexible conditionals, and provide two concrete examples of applying the decoupled conditionals. Empirically, we find this additional flexibility leads to improved model performance on a variety of regression tasks and Bayesian optimization (BO) applications.

Variational Gaussian Processes with Decoupled Conditionals

Abstract

Variational Gaussian processes (GPs) approximate exact GP inference by using a small set of inducing points to form a sparse approximation of the true posterior, with the fidelity of the model increasing with additional inducing points. Although the approximation error in principle can be reduced by using more inducing points, this leads to scaling optimization challenges and computational complexity. To achieve scalability, inducing point methods typically introduce conditional independencies and then approximations to the training and test conditional distributions. In this paper, we consider an alternative approach to modifying the training and test conditionals, in which we make them more flexible. In particular, we investigate decoupling the parametric form of the predictive mean and covariance in the conditionals, and learn independent parameters for predictive mean and covariance. We derive new evidence lower bounds (ELBO) under these more flexible conditionals, and provide two concrete examples of applying the decoupled conditionals. Empirically, we find this additional flexibility leads to improved model performance on a variety of regression tasks and Bayesian optimization (BO) applications.

1 Introduction

Gaussian processes (GPs) are a powerful class of non-parametric probabilistic models [38]. Their flexibility and ability to make probabilistic predictions make them particularly useful in situations where uncertainty estimates are important [23, 30, 43, 45, 56]. Although elegant, exact GPs become intractable for large datasets since the computational cost scales cubically with the size of the dataset.

To overcome this limitation, various sparse approximate GP approaches have been developed, mostly relying on sparse approximations of the true posterior [28, 29, 36]. Among those, variational GPs have become increasingly popular because they enable stochastic minibatching and apply to both regression [17] and classification [18] tasks. However, the approximate posterior and variational inference leave an unavoidable gap from the exact model. Especially with too few inducing points, variational GPs could yield suboptimal accuracy performance due to a lack of expressiveness. Using more inducing points closes this gap in principle, but leads to additional computational complexity and optimization difficulties [15]. While several works have studied ways of allowing more or better placed inducing points to improve accuracy of variational GPs, [34, 41, 50, 57], it is nevertheless challenging to use more than a small fraction of the dataset size in inducing points.

In this work, we take a different approach to improving model expressiveness for better accuracy of variational GPs through decoupling conditionals . To start, we point out that inducing point approximations rely on two key conditionals (see Sec. 3.1) - the training and the test conditional, where the mean and covariance are parameterized by inducing points and kernel hyperparameters. Typically, prior works have focused on various relaxations of these conditionals that enable significant computational complexity benefits [36, 44, 46, 47, 48]. In this work, we consider alternative changes

Training data

Inducing points

-

1

0

1

2

(a) Test NLL: 0.60,

l

:

0.85

SVGP with 10 inducing points

(b) Test NLL: -0.61,

l

mean: 0.27,

l

covar :

2.15

DCSVGP with 10 inducing points (ours)

-

2

-

1

0

1

2

3

4

-

1

0

1

2

(c) Test NLL: -0.90,

l

:

0.31

SVGP with 100 inducing points

-

2

-

1

0

1

2

3

4

(d) Test NLL: -0.93,

l

mean: 0.33,

l

covar :

0.31

DCSVGP with 100 inducing points (ours)

that increase the flexibility of these conditionals. In Sec. 3.2, we describe two concrete examples of this idea: decoupling kernel lengthscales and decoupling entire deep feature extractors.

As a simple illustration, Fig. 1 illustrates, for example, how decoupled lengthscales improve model fitting. Our model DCSVGP (see Sec. 3.2) learns decoupled lengthscales l mean and l covar for mean and covariance respectively, and we compare with baseline SVGP (see Sec. 2) which learns one lengthscale l .

To summarize our contributions: 1) We propose decoupled conditionals in variational GPs to improve model expressiveness for better accuracy. 2) We show that our idea is compatible with the variational framework and rigorously derive an ELBO for variational inference with decoupled conditionals. 3) We provide two concrete examples of applying decoupled conditionals and empirically show the superior performance of our models through extensive regression tasks and BO applications.

2 Background

We assume familiarity with GPs [38] and briefly introduce them for notational clarity. Given observation locations X = { x i } n i =1 ⊂ R d , a GP prior induces a multivariate Normal belief for latent function values f = { f ( x i ) } : f ∼ N ( µ X , K nn ) , where µ X , K nn are the mean values and covariance matrix at data X . Givend observations y = f + ϵ with Gaussian noise ϵ ∼ N (0 , σ 2 I ) , the posterior distribution of the function value f ∗ at a new data point x ∗ is p ( f ∗ | y ) = N ( µ ∗ , Σ ∗∗ ) , where

Model hyperparameters such as kernel hyperparameters and noise σ are typically estimated by Maximum Likelihood using standard numerical solvers such as LBFGS [35]. If no approximations are used, each evaluation to optimize the log marginal likelihood function costs O ( n 3 ) flops and O ( n 2 ) memory, thus motivating approximate methods for large training datasets.

2.1 Sparse Gaussian Processes

To overcome the scalability limitations of exact GPs, many authors have proposed a variety of sparse GPs by introducing inducing points Z = { z i } i m =1 [17, 44, 47, 48, 49]. Inducing points are associated with inducing values f m , which represent latent function values at Z under the same GP assumption.

Although inducing values f m are marginalized out in the predictive distribution, they typically reduces training costs from O ( n 3 ) to O ( n 2 m + m 3 ) for each gradient step, where m ≪ n .

SGPR. Based on inducing point methods, Sparse Gaussian Process Regression (SGPR) [49] further introduces variational inference into sparse GPs. Assuming that inducing data is adequate for inference, Titsias [49] further introduces a variational distribution ϕ ( f m ) ∼ N ( m , S ) that approximates the posterior p ( f m | y ) . Thus, the predictive density can be approximated as

Here the predictive mean µ f ( x ∗ ) and the latent function variance σ f ( x ∗ ) 2 are:

where ˜ K ∗∗ = K ∗∗ -K ∗ m K -1 mm K T ∗ m . The variational distribution ϕ ( f m ) = N ( m , S ) is then learned by maximizing the variational ELBO [18, 21], which is a lower bound on the log marginal likelihood:

where q ( f ) = ∫ p ( f | f m ) ϕ ( f m ) d f m , and KL [ ϕ ( f m ) || p ( f m )] is the KL divergence [27].

SVGP. To enable data subsampling during training, SVGP [17] avoids analytic solutions to the ELBO, but rather decomposes the ELBO as a sum of losses over training labels and enables stochastic gradient descent (SGD) [39] training:

where µ f ( · ) , σ f ( · ) 2 are the predictive mean and latent function variance from Eq. 1 respectively, and β is an optional regularization parameter for the KL divergence and can be tuned in practice [22].

PPGPR. To achieve heteroscedastic modeling and improve predictive variances, the Parametric Gaussian Process Regressor (PPGPR) [22] targets the preditive distribution directly in the loss function. It shares the same predictive mean µ f ( · ) and latent function variance σ f ( · ) 2 as SVGP, but uses a slightly different stochastic ELBO loss:

Whitening. In practice, the variational distribution ϕ ( f m ) is usually "whitened" to accelerate the optimization of the variational distribution [32]. Conventionally, the whitened variational distribution is ˜ ϕ ( f m ) = N ( ˜ m , ˜ S ) = N ( K -1 / 2 mm m , K -1 / 2 mm SK -1 / 2 mm ) , where K 1 / 2 mm is the square root of K mm . With whitening, the KL divergence and predictive distribution in ELBO (Eq. 2) are both simplified:

3 Methodology

Here, we present variational GPs with decoupled conditionals. In Sec. 3.1 and 3.2, we introduce decoupled conditionals under a unifying framework for approximate GPs, followed by two concrete examples. In Sec. 3.3, we derive an ELBO to do variational inference with decoupled conditionals.

3.1 Conditionals of Approximate GPs

Quinonero-Candela and Rasmussen [36] introduced a unifying framework for sparse GPs. Through the framework, an approximate GP can be interpreted as "an exact inference with an approximated

prior" in contrast to "an approximate inference with an exact prior". With inducing points Z = { z i } i m =1 and inducing values f m , most sparse GPs approximate the joint Gaussian prior as:

where dependencies between training data f and test data f ∗ are induced by inducing values f m . Different approximations can be made of the inducing training conditionals q ( f | f m ) and the inducing test conditionals q ( f ∗ | f m ) . Table 1 provides the exact expressions of the two conditionals and examples of approximations used in various approximate GP methods. For example, the Deterministic Training Conditional (DTC) approximation uses a deterministic training conditional and the exact test conditional [44]; The Fully Independent Training Conditional (FITC) approximation uses an approximate training conditional with diagonal corrections and the exact test conditional [48].

3.2 Decoupled Conditionals and Prediction

In this paper, we consider augmenting the exact conditionals by a more flexible form that decouples the kernel hyperparameters in the mean and covariance in both conditionals:

where the Q and K matrices are formed by the same family of kernel functions, but some kernel hyperparameters are decoupled . See Table 1 for a comparison with other approximations. Decoupled conditionals improve model flexibility without relying on more inducing points and it applies to various SVGP-based models. We provide two examples of decoupled models that we will evaluate.

Example 1: Decoupled lenghthscales. By decoupling the kernel lengthscale l into l mean for the mean and l covar for the covariance, we enable the model to learn in settings where the function value changes more rapidly than the variance. In Eq. 3, this corresponds to the Q and K kernel matrices being formed using separate lengthscales. For example, an RBF kernel gives

.

We will denote the application of decoupled lengthscales to SVGP as Decoupled Conditional SVGP (DCSVGP) .

Example 2: Decoupled deep kernel learning. Wilson et al. [53] proposed deep kernel learning (DKL) which stacks a deep neural network feature extractor h with a GP layer. Combining DKL with variational GP models is straightforward, with the feature extractor h learned through ELBO with all other model (hyper)parameters [54]. The feature extractor h can then be decoupled: one h mean for the mean and one h covar for the covariance. Again using the RBF kernel as a base example:

We denote the application of decoupled deep feature extractors to SVGP as SVGP-DCDKL .

Prediction. Using decoupled conditionals, the predictive posterior at a new point x ∗ is

where the predictive mean ˜ µ f ( x ∗ ) and latent function variance ˜ σ f ( x ∗ ) 2 are similar to Eq. 1:

3.3 The Evidence Lower Bound (ELBO)

In this section, we derive an ELBO for DCSVGP model fitting (other examples of decoupled models follow the same derivation). All model parameters and hyperparameters are learned by maximizing the resulting ELBO. Following the standard variational inference [2], we approximate the true posterior distribution p ( f , f m | y ) by the variational distribution q ( f , f m ) and miminize the KL divergence: min KL ( q ( f , f m ) || p ( f , f m | y )) . In the standard case, q ( f , f m ) = p ( f | f m ) ϕ ( f m ) , but a decoupled model has q ( f , f m ) = ψ ( f | f m ) ϕ ( f m ) since it further approximates the training conditional p ( f | f m ) by a decoupled one ψ ( f | f m ) . This difference leads to the following ELBO for the decoupled model:

We refer to App. A.1 for additional derivation details. Adding regularization parameters β 1 and β 2 to the KL divergence terms as is often done in practice, the ELBO for DCSVGP is

where the predictive mean ˜ µ f ( x i ) and latent function variance ˜ σ f ( x i ) 2 are same as Eq. 4

The explicit expression of Ω . The Ω term can be computed explicitly (see App. A.1):

Comparing ELBODCSVGP and ELBOSVGP. The ELBOSVGP in Eq. 2 and ELBODCSVGP in Eq. 5 both consist of an approximate likelihood term and a KL divergence part. There are two differences: 1) ELBODCSVGP involves different predictive mean ˜ µ f ( x i ) and variance ˜ σ f ( x i ) 2 derived from decoupled conditionals, see Eq. 6; 2) ELBODCSVGP contains an additional KL divergence term Ω , which is an expected KL divergence of the two training conditionals over the variational distribution ϕ ( f m ) , regularizing the difference between the decoupled conditional ψ ( f | f m ) and the exact one p ( f | f m ) .

The regularization parameter. In Eq. 5, ELBODCSVGP contains two KL divergence terms with regularization parameters β 1 and β 2 , respectively. Varying β 1 controls the regularization on prior similarities, same as SVGP, while varying β 2 controls the regularization on the difference between the decoupled conditional and the exact one. In the limit where β 2 → + ∞ , decoupling is disallowed and DCSVGP degenerates to SVGP. See Sec. 5.2 for more discussion and empirical study.

Whitening. Decoupling the mean and covariance hyperparameters introduces a challenge where one can whiten with either K 1 / 2 mm or Q 1 / 2 mm , with the drawback of only simplifying one term. K -whitening simplifies the KL divergence but leaves the predictive distribution significantly 'less linear', while Q -whitening does the opposite. For example, the predictive distribution from the two whitening choices are:

Empirically Q -whitening performs better because a simplified predictive distribution is more favorable for model fitting. See App. A.2 for KL divergence terms and derivation details.

4 Related Work

Variational GPs have been successfully extended to various settings [1, 3, 4, 11, 16, 19, 20, 31, 51, 55, 54]. Among these vast enhancements, much attention has been devoted to the study of inducing points, the core part that leads to an expressiveness-and-complexity trade-off. Many works have studied better placement of inducing points, including selecting from training data [9, 24, 42, 44, 47, 7, 34] and from the input domain [40, 50], but typically inducing points are learned as model parameters [17, 48]. On the other hand, several schemes have been developed to reduce complexity and allow more inducing points, e.g. Fourier methods [19, 29] and methods that decouple inducing points used in variational mean and covariance [8, 15, 41]. However, more inducing points could be less informative under certain data characteristics [6, 16, 50] or result in suboptimal model fitting [15, 57]. Therefore, our work takes a different direction to improve variational GPs via more flexible mean and covariance modeling rather than replying on more inducing points. The work that is most closely related to ours is the ODSVGP model [41] that decouples the inducing points for mean and variance to allow more inducing points used in mean modeling. With similar decoupling idea, we are motivated by increasing model flexibility rather than reducing model complexity and we simply decouple the kernel hyperparameters with negligible additional costs.

5 Experiments

We evaluate the performance of decoupled models proposed in Sec. 3.2: DCSVGP (variational GPs using decoupled lengthscales) and SVGP-DCDKL (variational GPs with deep kernel learning using decoupled deep feature extractors). Because PPGPR [22] is orthogonal to our decoupling method, similarly we also evaluate DCPPGPR (decoupled lengthscales) and PPGPR-DCDKL (decoupled feature extractors). All experiments use an RBF kernel and a zero prior mean and are accelerated through GPyTorch [14] on a single GPU. Code is available at https://github.com/xinranzhu/ Variational-GP-Decoupled-Conditionals .

5.1 Regression Tasks

We consider 10 UCI regression datasets [10] with up to 386508 training examples and up to 380 dimensions. We present main results with non-ARD RBF kernels, the Q -whitening scheme described in Sec. 3.3, and set β 2 = 1 e3 . Results are averaged over 10 random dataset splits. For additional results with ARD kernels and K -whitening, see App. B.1.4 and B.1.5. For more experiment setup and training details such as the number of inducing points, we refer to App. B.1.

DCSVGP and DCPPGPR. We first compare DCSVGP to baseline SVGP [17] and ODSVGP [41] and similarly compare DCPPGPR to PPGPR [22] and ODPPGPR. Table 2 shows the test RMSE and NLL results. We observe that DCSVGP (and similarly DCPPGPR) yields the lowest RMSE and NLL on 8 datasets, demonstrating the improved flexibility of our method. We report all learned lengthscales in App. B.1.1. On the other hand, we also note that on Protein and Elevators, DCPPGPR slightly overfits with worse test NLL. We show in Sec. 5.2 that increasing β 2 resolves this issue. We also find that decoupling lengthscales generally improves model calibration (see App. B.1.3 for results and discussions).

SVGP-DCDKL and PPGPR-DCDKL. We then compare SVGP-DCDKL with baseline model SVGP-DKL [53] and comare PPGPR-DCDKL with baseline model PPGPR-DKL [22]. Here, all models learn inducing points in the input space rather than in the feature space and we refer to App. B.1.6 for more discussion on this choice and supporting results. Table 3 shows the test RMSE and NLL and we observe that SVGP-DCDKL yields better (or equivalent) RMSE and NLL on all datasets but the Slice dataset; PPGPR-DCDKL always yields better (or equivalent) RMSE and only obtains worse NLL on 3 datasets due to an overfitting issue similar to DCPPGPR.

5.2 Ablation Study On β 2

In Sec. 5.1, we evaluate decoupled models with fixed β 2 = 1 e3 and observe mostly superior performance except for occasional overfiting with DCPPGPR and PPGPR-DCDKL. Here, we study how the regularization parameter β 2 affects model performance. We evaluate DCPPGPR with β 2 ∈ { 0 , 0 . 001 , 0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1 . 0 } on two datasets-Pol and Protein- that represent our general findings.

The first row of Fig. 2 shows performance on Pol: DCPPGPR gets lower RMSE and NLL with decreasing β 2 , and β 2 → 0 is desirable. We also see how the decoupled lengthscales l mean and l covar diverge from PPGPR's lengthscale as β 2 decreases. The second row of Fig. 2 shows results on Protein: DCPPGPR gets better RMSE but worse NLL as β 2 decreases and thus overfits when β 2 < 0 . 1 . In this case, β 2 = 0 . 1 is the best. We refer to App. B.2 for more supporting results, and conclude that 1) small β 2 or even 0 is usually ideal and 2) adding back some small regularization β 2 = 0 . 1 appears to resolve the instances of overfitting we noticed.

DCPPGPR

PPGPR

0

.

20

0

.

25

0

.

30

Performance on Pol

Test RMSE

-

0

.

6

-

0

.

4

-

0

.

2

Test NLL

0

.

4

0

.

6

0

.

8

l

mean

1

2

3

4

l

covar

10

-

3

10

-

2

10

-

1

10

0

β

2

0

.

63

0

.

64

0

.

65

0

.

66

Performance on Protein

10

-

3

10

-

2

10

-

1

10

0

β

2

0

.

78

0

.

80

0

.

82

0

.

84

0

.

86

10

-

3

10

-

2

10

-

1

10

0

β

2

0

.

12

0

.

14

0

.

16

0

.

18

0

.

20

10

-

3

10

-

2

10

-

1

10

0

β

2

0

.

200

0

.

225

0

.

250

0

.

275

0

.

300

2000

3000

4000

5000

Number of Evaluations

0

2

4

Mean Reward

Rover (dim=60)

SVGP

DCSVGP

200

400

600

800

1000

Number of Evaluations

100

200

300

Objective Value

Lunar (dim=12)

SVGP

DCSVGP

0

50K

100K

150K

200K

Number of Evaluations

0.74

0.76

0.78

0.80

0.82

Mean Reward

Ranolazine (dim=256)

PPGPR-DKL

PPGPR-DCDKL

5.3 Applications to Bayesian Optimization (BO)

In this section, we apply our decoupled variational GPs to BO tasks [13]. We consider 3 real BO tasks: the rover trajectory planning problem (Rover) [12, 52], the lunar landing reinforcement learning (Lunar) [12], and a challenging molecule design task Ranolazine MPO (Ranolazine) [5, 33]. We take TuRBO [12] as the BO algorithm and use different variational GPs in TuRBO. On Rover and Lunar, we use SVGP with TuRBO and compare SVGP with DCSVGP. On Ranolazine we first reduce the problem dimension to 256 using a Variational Autoencoder (VAE) [26], then perform BO in the latent space using PPGPR-DKL with TuRBO [33]. We compare PPGPR-DKL with PPGPR-DCDKL on Ranolazine. Consistent with our ablation study in Sec. 5.2, we find that β 2 = 0 performs the best on Rover and Ranolazine while Lunar requires more regularization with β 2 = 0 . 1 . Fig. 3 summarizes optimization performance averaged on at least 10 runs - all decoupled models outperform their coupled counterparts, showing the advantage of decoupling conditionals in BO applications.

5.4 Plausible Extensions

We further explore extensions of decoupled conditionals to achieve more flexible predictive mean and variance of variational GPs. Beyond decoupling the same parametric form, we could model the predictive mean and covariance differently for a better fit. Despite lack of theoretical grounding, we evaluate two possibilities, both targeting a more complex predictive mean than the covariance.

Replace the predictive mean of SVGP by a neural network. In fact, the importance of the parametric form of the covariance field is mentioned in Jankowiak et al. [22]: "a good ansatz for the predictive variance is important for good uncertainty prediction". They suggest, but never implement

or evaluate, that one could replace the mean function with a neural network so that inducing points are only used for variance prediction. We empirically evaluate this idea, denoted as NNSVGP (and NNPPGPR). With a totally flexible predictive mean, NNSVGP and NNPPGPR yield best test RMSE over 9 out of 10 datasets (see supplement for the RMSE table). However, with the predictive mean fully independent of the GP framework, NNSVGP and NNPPGPR yield worse NLL typically. Table 4 reports test NLL, where results are generally mixed.

Simplified deep kernels for covariances. In the DKL setting, instead of using decoupled feature maps, one could use entirely different neural network architectures for the predictive mean and covariance. We empirically find that a simpler feature mapping, or even no feature mapping, for the covariance still make a plausible model with good performance but smaller training cost. We denote the method with no feature mapping for the covariance as SVGP-MeanDKL and PPGPR-MeanDKL.

We present results in Table 5. We observe SVGP-MeanDKL does not significantly outperform the decoupled models SVGP-DCDKL, but it improves baseline SVGP-DKL on 8 out of 10 datasets. PPGPR-MeanDKL performs similarly in terms of the RMSE metric (better than baseline PPGPRDKLbut worse than PPGPR-DCDKL). However, in terms of NLL, it gives poor uncertainty prediction (worse NLL). This suggests that it is more beneficial to have different models (decoupled feature extractors in SVGP-DCDKL) for conditional mean and covariance rather than only modeling the conditional mean (SVGP-MeanDKL) or having same models (SVGP-DKL), and SVGP-based model is more robust than the PPGPR-based model in uncertainty prediction.

6 Conclusion

Variational GPs scale approximate GP inference to millions of training examples but may yield suboptimal accuracy due to lack of model expressiveness or poor model fitting. We propose a simple idea to improve model expressiveness by decoupling the parametric form of the mean and variance in the conditionals of variational GPs. We derive an ELBO for our model with decoupled conditionals, which end up being similar to the ELBO of standard variational GP with an additional a regularization

term. Our method is simple yet effective, and it applies to various variational GPs. We provide two concrete examples, one decoupling kernel lengthscales in the basic variational GP setting and one decoupling the feature mapping in the deep kernel learning setting. Through extensive empirical study, we show that the decoupled conditionals effectively improve the accuracy of variational GPs in terms of both mean and uncertainty prediction. We also empirically study two plausible extensions of our method, motivated by the idea of modeling the mean and variance differently, but we conclude that they are not as effective as our decoupled models. Current limitations of our work and therefore future directions include but not limit to: 1) the application of decoupled conditionals to more SVGP-based models; 2) further generalization of flexible forms of training and testing conditionals that improve model performance; 3) the application of decoupled variational GP framework to tasks other than regression or BO, such as classification tasks.

References

V. Adam, S. Eleftheriadis, A. Artemev, N. Durrande, and J. Hensman. Doubly sparse variational Gaussian processes. In International Conference on Artificial Intelligence and Statistics , pages 2874-2884. PMLR, 2020.

D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association , 112(518):859-877, 2017.

K. Blomqvist, S. Kaski, and M. Heinonen. Deep convolutional Gaussian processes. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2019, Würzburg, Germany, September 16-20, 2019, Proceedings, Part II , pages 582-597. Springer, 2020.

V. Borovitskiy, A. Terenin, P. Mostowsky, et al. Matérn Gaussian processes on Riemannian manifolds. Advances in Neural Information Processing Systems , 33:12426-12437, 2020.

N. Brown, M. Fiscato, M. H. Segler, and A. C. Vaucher. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling , 59(3):1096-1108, 2019.

D. Burt, C. E. Rasmussen, and M. Van Der Wilk. Rates of convergence for sparse variational Gaussian process regression. In International Conference on Machine Learning , pages 862-871. PMLR, 2019.

D. R. Burt, C. E. Rasmussen, and M. Van Der Wilk. Convergence of sparse variational inference in Gaussian processes regression. The Journal of Machine Learning Research , 21(1):5120-5182, 2020.

C.-A. Cheng and B. Boots. Variational inference for Gaussian process models with linear complexity. Advances in Neural Information Processing Systems , 30, 2017.

L. Csató and M. Opper. Sparse representation for Gaussian process models. Advances in neural information processing systems , 13, 2000.

D. Dua and C. Graff. UCI Machine Learning Repository. Irvine, CA: University of California, School of Information and Computer Science, 2017. URL http://archive.ics.uci.edu/ ml .

V. Dutordoir, N. Durrande, and J. Hensman. Sparse Gaussian processes with spherical harmonic features. In International Conference on Machine Learning , pages 2793-2802. PMLR, 2020.

D. Eriksson, M. Pearce, J. R. Gardner, R. Turner, and M. Poloczek. Scalable global optimization via local Bayesian optimization. In Advances in Neural Information Processing Systems , volume 32, 2019.

P. I. Frazier. Bayesian optimization. In Recent advances in optimization and modeling of contemporary problems , pages 255-278. Informs, 2018.

J. R. Gardner, G. Pleiss, D. Bindel, K. Q. Weinberger, and A. G. Wilson. Gpytorch: Blackbox matrix-matrix Gaussian process inference with GPU acceleration. In Advances in Neural Information Processing Systems , volume 31, 2018.

M. Havasi, J. M. Hernández-Lobato, and J. J. Murillo-Fuentes. Deep Gaussian processes with decoupled inducing inputs. arXiv preprint arXiv:1801.02939 , 2018.

J. Hensman and N. D. Lawrence. Nested variational compression in deep Gaussian processes. arXiv preprint arXiv:1412.1370 , 2014.

J. Hensman, N. Fusi, and N. D. Lawrence. Gaussian processes for big data. In Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence , page 282-290, Arlington, Virginia, USA, 2013. AUAI Press.

J. Hensman, A. Matthews, and Z. Ghahramani. Scalable variational Gaussian process classification. In Artificial Intelligence and Statistics , pages 351-360. PMLR, 2015.

J. Hensman, N. Durrande, A. Solin, et al. Variational fourier features for Gaussian processes. J. Mach. Learn. Res. , 18(1):5537-5588, 2017.

T. N. Hoang, Q. M. Hoang, and B. K. H. Low. A distributed variational inference framework for unifying parallel sparse Gaussian process regression models. In International Conference on Machine Learning , pages 382-391. PMLR, 2016.

M. D. Hoffman, D. M. Blei, C. Wang, and J. Paisley. Stochastic variational inference. Journal of Machine Learning Research , 14(5), 2013.

M. Jankowiak, G. Pleiss, and J. Gardner. Parametric Gaussian process regressors. In International Conference on Machine Learning , pages 4702-4712. PMLR, 2020.

D. E. Jones, D. C. Stenning, E. B. Ford, R. L. Wolpert, T. J. Loredo, C. Gilbertson, and X. Dumusque. Improving exoplanet detection power: Multivariate Gaussian process models for stellar activity. arXiv preprint arXiv:1711.01318 , 2017.

S. Keerthi and W. Chu. A matching pursuit approach to sparse Gaussian process regression. Advances in neural information processing systems , 18, 2005.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations , 2014.

D. P. Kingma and M. Welling. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114 , 2013.

S. Kullback and R. A. Leibler. On information and sufficiency. The annals of mathematical statistics , 22(1):79-86, 1951.

M. Lázaro-Gredilla and A. Figueiras-Vidal. Inter-domain Gaussian processes for sparse inference using inducing features. Advances in Neural Information Processing Systems , 22, 2009.

M. Lázaro-Gredilla, J. Quinonero-Candela, C. E. Rasmussen, and A. R. Figueiras-Vidal. Sparse spectrum Gaussian process regression. The Journal of Machine Learning Research , 11:18651881, 2010.

M. Lorenzi, G. Ziegler, D. C. Alexander, and S. Ourselin. Efficient Gaussian process-based modelling and prediction of image time series. In International Conference on Information Processing in Medical Imaging , pages 626-637. Springer, 2015.

W. J. Maddox, S. Stanton, and A. G. Wilson. Conditioning sparse variational Gaussian processes for online decision-making. Advances in Neural Information Processing Systems , 34:6365-6379, 2021.

A. G. d. G. Matthews. Scalable Gaussian process inference using variational methods . PhD thesis, University of Cambridge, 2017.

N. Maus, H. Jones, J. Moore, M. J. Kusner, J. Bradshaw, and J. Gardner. Local latent space Bayesian optimization over structured inputs. Advances in Neural Information Processing Systems , 35:34505-34518, 2022.

H. B. Moss, S. W. Ober, and V. Picheny. Inducing point allocation for sparse Gaussian processes in high-throughput Bayesian optimisation. In International Conference on Artificial Intelligence and Statistics , pages 5213-5230. PMLR, 2023.

J. Nocedal and S. Wright. Numerical optimization . Springer Science & Business Media, 2006.

J. Quinonero-Candela and C. E. Rasmussen. A unifying view of sparse approximate Gaussian process regression. The Journal of Machine Learning Research , 6:1939-1959, 2005.

A. Ramdas, N. García Trillos, and M. Cuturi. On wasserstein two-sample testing and related families of nonparametric tests. Entropy , 19(2):47, 2017.

C. E. Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning , pages 63-71. Springer, 2003.

H. Robbins and S. Monro. A stochastic approximation method. The Annals of Mathematical Statistics , pages 400-407, 1951.

S. Rossi, M. Heinonen, E. Bonilla, Z. Shen, and M. Filippone. Sparse Gaussian processes revisited: Bayesian approaches to inducing-variable approximations. In International Conference on Artificial Intelligence and Statistics , pages 1837-1845. PMLR, 2021.

H. Salimbeni, C.-A. Cheng, B. Boots, and M. Deisenroth. Orthogonally decoupled variational Gaussian processes. Advances in neural information processing systems , 31, 2018.

J. Schreiter, D. Nguyen-Tuong, and M. Toussaint. Efficient sparsification for Gaussian process regression. Neurocomputing , 192:29-37, 2016.

P. Schulam and S. Saria. Reliable decision support using counterfactual models. In Advances in Neural Information Processing Systems , volume 30, 2017.

M. W. Seeger, C. K. Williams, and N. D. Lawrence. Fast forward selection to speed up sparse Gaussian process regression. In International Workshop on Artificial Intelligence and Statistics , pages 254-261. PMLR, 2003.

B. Shahriari, K. Swersky, Z. Wang, R. P. Adams, and N. De Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE , 104(1):148-175, 2015.

B. W. Silverman. Some aspects of the spline smoothing approach to non-parametric regression curve fitting. Journal of the Royal Statistical Society: Series B (Methodological) , 47(1):1-21, 1985.

A. Smola and P. Bartlett. Sparse greedy Gaussian process regression. Advances in neural information processing systems , 13, 2000.

E. Snelson and Z. Ghahramani. Sparse Gaussian processes using pseudo-inputs. Advances in neural information processing systems , 18, 2005.

M. Titsias. Variational learning of inducing variables in sparse Gaussian processes. In Artificial intelligence and statistics , pages 567-574. PMLR, 2009.

A. K. Uhrenholt, V. Charvet, and B. S. Jensen. Probabilistic selection of inducing points in sparse Gaussian processes. In Uncertainty in Artificial Intelligence , pages 1035-1044. PMLR, 2021.

M. Van der Wilk, C. E. Rasmussen, and J. Hensman. Convolutional Gaussian processes. Advances in Neural Information Processing Systems , 30, 2017.

Z. Wang, C. Gehring, P. Kohli, and S. Jegelka. Batched large-scale Bayesian optimization in high-dimensional spaces. In International Conference on Artificial Intelligence and Statistics , pages 745-754. PMLR, 2018.

A. G. Wilson, Z. Hu, R. Salakhutdinov, and E. P. Xing. Deep kernel learning. In Artificial intelligence and statistics , pages 370-378. PMLR, 2016.

A. G. Wilson, Z. Hu, R. R. Salakhutdinov, and E. P. Xing. Stochastic variational deep kernel learning. Advances in neural information processing systems , 29, 2016.

L. Wu, G. Pleiss, and J. P. Cunningham. Variational nearest neighbor Gaussian process. In International Conference on Machine Learning , pages 24114-24130. PMLR, 2022.

Z. Yi, R. Calandra, F. Veiga, H. van Hoof, T. Hermans, Y. Zhang, and J. Peters. Active tactile object exploration with Gaussian processes. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 4925-4930. IEEE, 2016.

X. Zhu, J. R. Gardner, and D. Bindel. Efficient variational Gaussian processes initialization via kernel-based least squares fitting. In NeurIPS Workshop on Gaussian Processes, Spatiotemporal Modeling, Decision-making Systems , Dec. 2022.

A Methodology

A.1 The Evidence Lower Bound (ELBO)

Here we privide detailed derivation of the evidence lower bound (ELBO) for the proposed decoupled SVGP model fitting. The goal of variational learning remains the same:

where in the standard case q ( f , f m ) = p ( f | f m ) ϕ ( f m ) , but we further approximate the training conditional p ( f | f m ) by a decoupled one ψ ( f | f m ) and have q ( f , f m ) = ψ ( f | f m ) ϕ ( f m ) . Following the standard variational inference [2], we have the ELBO for decoupled SVGP:

The first likelihood term in Eq. 7 is

where the predictive mean ˜ µ f ( x i ) and latent function variance ˜ σ f ( x i ) 2 are

The second KL divergence term in Eq. 7 is

The KL divergence term is almost the same as that in the standard ELBO, with an additional term Ω := ∫ ϕ ( f m ) KL ( ψ ( f | f m ) || p ( f | f m )) d f m .

Analytic expression of Ω We could compute explicitly the analytical expression for the Ω term. For notational simplicity, let decoupled and exact training conditionals be

The Ω term is an expectation of a quadratic form:

where T = A T Σ -1 A , A = Q nm Q -1 mm -K nm K -1 mm , and ϕ ( f m ) = N ( m , S ) .

A.2 Whitening

In this section, we provide detailed discussion of the whitening scheme used in the decoupled models. We first present the standard whitening scheme used in variational GP models, then derive two plausible whitening schemes for decoupled models - the K -whitening and the Q -whitening. We implemented both ways of whitening for decoupled models, and empirically found that the latter ( Q -whitening) performs better (See Section B.1.5).

A.2.1 The standard whitening

For standard models like SVGP, using the square root of K mm , the variational distribution is "whitened" (reparameterized) in the following way:

Under such reparameterization, the KL divergence is equivalent to a simpler form:

where p 0 ( f m ) = N ( 0 , I ) is the standard Normal, and M is the number of inducing points. Now plug ¯ m and ¯ S in the predictive distribution p ( f ) , we have the predictive mean µ f ( x i ) and latent function variance σ f ( x i ) 2 in simplified forms as well:

A.2.2 The K -whitening

For decoupled models like DCSVGP, we could similarly use the square root of K mm to whiten the variational distribution:

Same as the standard whitening, the KL divergence is simplified:

where p 0 ( f m ) = N ( 0 , I ) is the standard Normal, and M is the number of inducing points. However, if we plug ¯ m and ¯ S in the predictive distribution p ( f ) , the predictive mean ˜ µ f ( x i ) and latent function variance ˜ σ f ( x i ) 2 are not simplified in the same way as the standard case:

A.2.3 The Q -whitening

For decoupled models like DCSVGP, another plausible way of whitening for decoupled models is to use the square root of Q mm :

In this way, the KL divergence term is equivalent to a different form:

where ¯ p ( f m ) = N ( 0 , ¯ K ) and ¯ K = ¯ L ¯ L T , ¯ L = Q -1 / 2 mm K 1 / 2 mm and M is the number of inducing points. Now plug ¯ m and ¯ S in the predictive distribution p ( f ) , we have the predictive mean ˜ µ f ( x i ) and latent function variance ˜ σ f ( x i ) 2 simplified in a similar way as the standard case:

As discussed in the main paper, it is not surprising that the Q -whitening performs better in practice. This is because the K -whitening simplifies the KL divergence but leaves the predictive distribution complicated, while the Q -whitening does the opposite, and a complicated predictive distribution makes the optimization much harder than a complicated KL divergence term.

B Experiments

B.1 Regression Tasks

Experiment setup Table 6 provides details on the 10 UCI regression datasets [10] and the number of inducing points used for each dataset. For ODSVGP, since the goal is to use more inducing points for mean and less for covariance under the same budget, we use the setup in the original paper [41]: if SVGP uses 4X inducing points, then ODSVGP uses 7X inducing points for mean and 3X inducing points for covariance. We use the Adam [25] optimizer with a multistep scheduler to train all models on all datasets, and we train for 300 epochs using training batch size 1024. We selected the best training hyperparameters for SVGP and use the same ones for all models - learning rate lr = 5 e3 and a multistep learning rate scheduler (multiplicative factor γ = 0 . 2 ).

B.1.1 Learned lengthscales

Table 6 also provides learned lengthscales of all models, averaged over 10 random dataset splits, as supplementary results for Table 2. As is shown in Table 6, we observe how lengthscales are decoupled in DCSVGP and DCPPGPR - the lengthscale for mean function l mean is typically smaller than the shared lengthscale l , while the lengthscale for covariance function l covar is typically larger than the shared lengthscale l . This is indeed consistent with the motivation of decoupling lengthscales since a smaller lengthscale for the mean function would typically lead to a better mean fit when the number of inducing points is limited compare to total training data; when the mean function is well fit, the lengthscale for the covariance can be larger to further reduce the predictive covariance.

B.1.2 Error bars of main results

Here we provide standard errors of main results Table 2 and Table 3 - see Figure 4 and Figure 5 respectively. Error bars for other similar results, i.e. Tables in the Appendix, are similarly small and not informative and therefore we avoid redundant results.

0.2

0.3

SVGP

ODSVGP

DCSVGP

Pol

0.375

0.380

Elevators

0.225 0.250 0.275

Bike

0.16

0.18

0.60

0.65

Protein

0.085

0.090

0.095

Keggdir

0.05

0.10

Slice

0.120

0.122

0.45

0.50

3Droad

0.78

0.79

Song

0.2

0.3

PPGPR

DCPPGPR

0.38

0.39

0.30

0.35

0.3

0.4

0.64

0.66

0.088

0.090

0.092

0.1

0.2

0.122

0.124

0.126 0.54

0.55

0.56

0.778

0.780

0.25

0.00

0.25

SVGP

ODSVGP

DCSVGP

0.43

0.44

0.45

0.0

0.2

0.4

0.2

0.95

1.00

1.06

1.04

1.02

1.0

0.5

0.70

0.69

0.68

0.6

0.7

0.8

1.17

1.18

1.19

0.50

0.25

PPGPR

DCPPGPR

0.38

0.40

0.9

0.8

1.0

0.8

0.80

0.85

1.65

1.60

1.50

1.25

1.00

1.9

1.8

0.25

0.30

1.108

1.110

1.112

0.050

0.055

0.060

SVGP-DKL

SVGP-DCDKL

Pol

0.342

0.344

Elevators

0.011 0.012 0.013

Bike

0.05

0.06

0.58

0.59

0.60

Protein

0.085

0.086

0.087

Keggdir

0.020

0.022

Slice

0.118

0.120

0.30

0.32

3Droad

0.773

0.774

Song

0.06

0.08

PPGPR-DKL

PPGPR-DCDKL

0.3425 0.3450 0.3475

0.026

0.028

0.030

0.06

0.07

0.08

0.59250.59500.5975

0.086

0.088

0.017

0.018

0.120

0.121

0.35

0.36

0.37

0.7705

0.7710

0.7715

1.6

1.5

1.4

SVGP-DKL

SVGP-DCDKL

0.345

0.350

2.8

2.7

2.6

1.6

1.4

1.2

0.89

0.90

1.06

1.04

2.5

2.4

0.72

0.71

0.2

0.3

1.162

1.164

2.6

2.4

PPGPR-DKL

PPGPR-DCDKL

0.30

0.35

3.20

3.15

3.10

2.2

2.0

1.8

0.8

1.0

1.2

1.65

1.60

2.70

2.65

2.60

1.95

1.90

1.85

0.30

0.25

1.10

1.11

1.12

5

0

5

0.0

0.2

0.4

0.6

0.8

Z-score Distribution

SVGP, D=0.33

Standard Normal

Model z-score

10

5

0

5

0.0

0.1

0.2

0.3

0.4

0.5

DCSVGP, D=0.18

4

2

0

2

0.0

0.1

0.2

0.3

0.4

0.5

0.6

PPGPR, D=0.18

4

2

0

2

4

0.0

0.1

0.2

0.3

0.4

DCPPGPR, D=0.02

B.1.3 Model Calibration

To study how decoupled conditionals affect model calibration, we evaluate the z-score distribution from model predictions on all UCI datasets and compare it to the standard Normal. The closer the z-score distribution from model predictions is to the standard Normal, the better the model is calibrated. As an example, Figure 6 shows the z-score distributions of SVGP, DCSVGP, PPGPR and DCPPGPR on the Kin40k dataset. As Figure 6 shows, on the Kin40k dataset, decoupled models DCSVGP and DCPPGPR are better calibrated compare to their counterparts SVGP and PPGPR, respectively. Quantitatively, we compute the Wasserstein distance D [37] between the model z-score distribution and standard Normal (see subtitles in Figure 6 for example), the smaller the better. Averaged over all 10 UCI datasets, SVGP and DCSVGP both have D = 0 . 27 , PPGPR has D = 0 . 22 and DCPPGPR has the best D = 0 . 17 . Therefore, we conclude that decoupling conditionals does not lead to a poorly calibrated model, but can even improve model calibration on average.

B.1.4 ARD kernels

Here we provide additional results on regression tasks using ARD kernels . In Table 2, we show that decoupled models outperform the baseline models using non-ARD kernels (kernels with onedimensional lengthscale). Here, we use ARD kernels (kernels with multi-dimensional lengthscales, one for each input dimension) for all models and show consistent advantages of using decoupled models . Table 7 summarizes results, similar to the main results using non-ARD kernels (Table 2 in main paper), we observe that all decoupled models outperform baseline models.

B.1.5 Whitening of decoupled models

In Section A.2, we discussed two plausible whitening schemes for the decoupled models and claimed that the Q -whitening is more favorable than the K -whitening due to the simplified predictive

distribution. Here, we empirically compare the two whitening schemes and verify such claim. Table 8 summarizes a comparison of the K -whitening and Q -whitening in the DCSVGP model, and we observe that Q -whitening outperforms K -whitening on 8 out of 10 datasets . Table 9 shows similar results on the DCPPGPR model.

B.1.6 DKL discussion

In the main paper, we evaluate SVGP-DKL, PPGPR-DKL and their decoupled counterparts SVGPDCDKL and PPGPR-DCDKL on 10 UCI datasets, and we choose to learn inducing points in the input space for all models. Despite the convention of learning inducing points in the feature space, we found that for baseline models SVGP-DKL and PPGPR-DKL, learning inducing points in the input space performs better on the UCI datasets, see Table 10 and Table 11.

B.2 Ablation Study on β 2

In addition to the ablation study results on DCPPGPR in the main paper, here we provide more ablation study results on DCSVGP and DCPPGPR with varying β 2 values on all datasets. Together with the representative results shown in the main paper, We obtain the same conclusion that

β 2 = 0 . 001 or even β 2 = 0 are good choices for both DCSVGP and DCPPGPR (see Table 12 and Table 13);

on rare cases (only on the Pol and Elevators dataset, see Table 13), DCPPGPR could overfit and β 2 = 0 . 1 or β 2 = 1 . 0 would fix the issue and improve the prediction performance.

More detailed results and discussions follows.

1DExample Using the same 1D toy example from the main paper, here in Figure 7 we show ablation study on this 1D example. Consistently, DCSVGP outperforms SVGP with 10 inducing points, and DCSVGP keeps improving as β 2 decreases. We also observe that the decoupled lengthscales l mean and l covar differ more and more as β 2 decreases and thus allowing the model to better fit mean and covariance differently.

DCSVGP From Table 12, we observe that, DCSVGP favors small β 2 values and β 2 = 0 . 001 or even β 2 = 0 are good choices. The general trend is that DCSVGP performs better as β 2 decreases. To clearly see this trend, in Figure 8 we provide detailed performance study on 3 datasets: the 1D synthetic dataset, the Pol dataset, and the Song dataset. We observe that both RMSE and NLL decrease as β 2 decreases, and the learned lengthscales differ more and more as β 2 decreases to better model the mean and covariance differently.

DCPPGPR From Table 13, we observe similar trend that DCPPGPR also favors small β 2 values and β 2 = 0 . 001 or β 2 = 0 are still good choices. In the main paper, we observe that DCPPGPR overfits on dataset Elevators and Protein, and on Protein we show how varying β 2 improves test NLL in detail (Figure 2 in the main paper). Here we also observe from Table 13 that regularization with β 2 = 1 . 0 would fix the issue for both dataset Elevators and Protein. Figure 13 shows how the test RMSE and NLL change with β 2 on the Elevators dataset - we observe that RMSE does not change much as β 2 varies, but NLL decreases as β 2 increases, showing how regularization improves test NLL.

-

1

0

1

2

(a) Test NLL:0.63,

l

:

1.13

SVGP

(b) Test NLL:0.54,

l

mean: 0.66,

l

covar :

0.68

DCSVGP (ours),

β

2

= 1

.

0

-

2

-

1

0

1

2

3

4

-

1

0

1

2

(c) Test NLL:0.17,

l

mean: 0.34,

l

covar :

0.52

DCSVGP (ours),

β

2

= 0

.

1

-

2

-

1

0

1

2

3

4

(d) Test NLL:-0.55,

l

mean: 0.28,

l

covar :

0.71

DCSVGP (ours),

β

2

= 0

.

01

Latent Function

Training data

Inducing points

B.3 Plausible Extension

RMSE results of NNSVGP We provide the RMSE results of NNSVGP and NNPPGPR here in Table 14 to support the observation that they yield the best test RMSE on 9 out of 10 datasets.

DCSVGP

DCSVGP-0

0

.

2

0

.

3

0

.

4

Performance on 1D

Test RMSE

-

0

.

2

0

.

0

0

.

2

0

.

4

Test NLL

0

.

4

0

.

6

0

.

8

l

mean

1

.

0

1

.

5

2

.

0

l

covar

0

.

150

0

.

175

0

.

200

0

.

225

0

.

250

Performance on pol

-

0

.

4

-

0

.

2

0

.

0

0

.

3

0

.

4

0

.

5

1

.

5

2

.

0

2

.

5

3

.

0

3

.

5

10

-

3

10

-

2

10

-

1

10

0

β

2

0

.

776

0

.

777

0

.

778

0

.

779

0

.

780

Performance on song

10

-

3

10

-

2

10

-

1

10

0

β

2

1

.

166

1

.

168

1

.

170

10

-

3

10

-

2

10

-

1

10

0

β

2

0

.

36

0

.

38

0

.

40

10

-

3

10

-

2

10

-

1

10

0

β

2

1

.

6

1

.

8

2

.

0

2

.

2

DCPPGPR

10

-

3

10

-

2

10

-

1

10

0

β

2

0

.

390

0

.

392

0

.

394

0

.

396

Performance on elevators

Test RMSE

10

-

3

10

-

2

10

-

1

10

0

β

2

0

.

37

0

.

38

0

.

39

0

.

40

0

.

41

Test NLL

10

-

3

10

-

2

10

-

1

10

0

β

2

0

.

8

0

.

9

1

.

0

1

.

1

l

mean

10

-

3

10

-

2

10

-

1

10

0

β

2

1

.

1

1

.

2

1

.

3

l

covar

Peer reviews:
comments: SUMMARY:
This paper addresses enhancing the performance of sparse approximate Gaussian processes, which has scalability with the dataset size but causes performance degradation.
The authors proposed a novel parametrization of conditionals as decoupled forms for training and testing ones, where lengthscales in kernel representations in their means and covariances can have different values.
Experimental results on multiple public datasets demonstrate that the proposed method performs better than existing methods.

STRENGTHS:
- Experimental results of the proposed method showed promising performance.

WEAKNESSES:
### Originality:
- Originality is limited to the proposal of Eq.3.

### Clarity:
- The important concept of how to select or generate inducing points is not described in the main text. For example, inducing points are different between the left and right in Figure 1, but it is not described why. 
- It is better to provide the definition of SVGP in Introduction.
- In the equation under l.53, some variables, such as mu, are not defined. The notation for x* is inconsistent
- In the description of Eq.1, it is better to define Kmm explicitly. Accordingly, the difference between Kmms for the proposed method and the existing methods is not clear.
- It is better to introduce p(f∗|fm) with more intuition.
- It is unclear how to determine the regularization parameter.

QUESTIONS:
NA

LIMITATIONS:
There is a study of performances over different hyperparameters.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies sparse Gaussian processes which allow different kernel hyperparameters such as length scale to be used, for instance, for the variational posterior mean and covariance. They derive a variational approximation that enables training of models of this kind. This extra scalable approximation flexibility is shown to lead to better predictions on UCI datasets, as well as a few benchmark-style Bayesian optimization problems.\n\nEDIT: As indicated below, I've acknowledged the authors' rebuttal and previously updated my review and score appropriately.\n\nSTRENGTHS:\nThe paper is technically sound, generally well-written, and includes all of the content and evaluation I would expect for this topic. The authors have done a reasonable and comprehensive job here. They evaluate on both modeling tasks (UCI dataset regression) and decision-making tasks (Bayesian optimization), as well as include an ablation study for one of the key non-obvious hyperparameters they introduce (the \\beta regularization term which controls how the test conditional component enters the objective).\n\nMy most common reaction while reading this paper was \"that makes sense\" - which is actually very good, but also means that I don't have too many comments on what to add since the paper was largely clear, and therefore this review will be a bit short. The authors should not be discouraged by this - the shortness in this case should be understood as a sign they wrote the paper well.\n\nWEAKNESSES:\nThis paper is studying what I view as a relatively old-school topic, and as a result much of the value here is from good evaluation and execution of ideas, rather than from the ideas themselves. Inducing point approximations have been studied very extensively, and at this stage there are so many variations out there that it isn't clear why we need to write more papers about them, since these methods are mature enough that readers can probably figure out new ones themselves when they are needed for applied work.\n\nThe need to add $\\beta$ regularization parameters is a bit suspicious, since this suggests there might sometimes be a need to downscale the extra term coming from different conditionals because it starts behaving badly in the limit of many test points. It also forces the user to consider even more hyperparameters, and in most cases there are already enough of those out there. As a result, I'm curious for what choices of conditionals the respective infinite-dimensional variational approximation problem between stochastic processes is well-defined. See questions.\n\nThe empirical evaluation is overall good, but it focuses too much on RMSE tables on UCI datasets. These take up a lot of space and are not super informative, so it would be better to have just one and move extra results into the appendix, creating more room for describing the experiments. I would be particularly interested to see more figures which evaluate the technique's produced uncertainty, particularly since inducing points tend to blow up uncertainty and over-smooth the data in situations where there are not enough inducing points, and it would be much more interesting to see how the authors' technique behaves in this situation, rather than more RMSE tables.\n\nQUESTIONS:\nIs there a way to view the inducing point approximation you propose as a random function via pathwise conditioning? By this, I mean that, since the true posterior satisfies $(f|y)(.) = f(.) + K_{(.)x} (K_{xx} + \\sigma^2 I)^{-1} (y - f(x) - \\epsilon)$ where $f \\sim GP(0,k)$ and $\\epsilon \\sim N(0,\\sigma^2 I)$, and the approximation of Titsias satisfies $(f|u)(.) = f(.) + K_{(.)z} K_{zz}^{-1} K_{zx} (K_{xz} K_{zz}^{-1} K_{zx} + \\sigma^2 I)^{-1} (y - g(x) - \\epsilon)$ where $g(x) \\sim N(0, K_{xz} K_{zz}^{-1} K_{zx})$ and $\\epsilon \\sim N(0,\\sigma^2 I)$, your posterior approximation might admit a similar formula, and its structure should give insight into the approximation's behavior. Do you know what the respective formula is?\n\nIs your variational optimization problem consistent in the limit of infinitely many test points? By this, I mean, does it lead to a valid Kullback-Leibler divergence minimization problem between stochastic processes, where the true posterior and variational approximation are absolutely continuous so that the KL is well-defined?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops a variational approximation for learning sparse Gaussian Processes. The central idea is to \"decouple\" the mean and covariance parameters. From my read, this decoupling simply means formulating two different kernel matrices (called $Q$ and $K$ in the paper), and then working those new parameters through the inference and learning mechanisms: to that end, the paper provides appropriately-detailed derivations, including for some empirical concerns (such as whitening). The paper formulates the proposed approach in two cases, evaluating on 10 benchmark regression tasks. Overall, the proposed approach yields improved benchmark tasks. Ablation studies are also performed, which help provide insight into the approach.\n\nSTRENGTHS:\nOverall, this paper achieves an acceptable balance of mathematical derivation vs. explanation. However, please see \"Clarity\" in weakness for qualifications.\n\nExperiments: Experiments were averaged across 10 runs, and sufficient details of standard error & significance were included. There's an appropriate amount of rigor, and ablation studies run.\n\nWEAKNESSES:\nClarity: Key summary statements, to act as milestones, throughout would definitely help. This is especially true of Table 1, section 3.2 (Eqn 3), and section 3.3: broadly, the equations are presented, but without a lot of discussion contextualizing the meaning and importance of those equations. \n\nLimited discussion: Given the number of experiments run, the discussion does not have a good pay-off. For example, Table 5 (one of the ablation studies) has fewer than 3 lines of prose dedicated to discussing those results in the main paper. \n\nLack of error analysis: The battery of regression benchmark tasks gives some evidence of broad performance, but there's no error analysis or insight into the numbers and performance. Currently, the results read as \"just numbers\": it's not clear, e.g., what a reduction from 0.321 to 0.156 on \"Pol\" means.\n\nThis is more of a minor point, but not without impact: splitting related work into sections 2.1 and 4 was not effective for me. It made 2.1 very much a slog, and section 4 seem unimportant.\n\nQUESTIONS:\nQ1: Please provide a few paragraphs that would form the basis of an error analysis & extended discussion section. \n\nQ2: Please provide a limitation discussion (or if I missed it in the paper, please point me to it).\n\nLIMITATIONS:\nNo: no obvious limitations section was provided in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of increasing the expressivity of sparse Gaussian processes without increasing the number of inducing points by considering a decoupled ELBO. In their setup, the gram matrices appearing in the predictive mean and covariance have different parameterizations. They consider two such parameterizations, the first corresponding to learning different length-scales in a non-ARD RBF kernel, and the second using different neural networks in a deep kernel learning setup. An ELBO is derived in this setup, and a version is given for the PPGPR analogue. Whitening is considered with mean-whitening being favored. Numerical examples are given on UCI regression and Bayesian optimization tasks. The effects of differing levels of regularization are examined. Extensions parameterized with further neural networks are considered.\n\nSTRENGTHS:\nThe paper is well structured and written. Numerical examples are given and their methods appear to perform well. The new parameter that's introduced in their derivation is studied.\n\nWEAKNESSES:\nThere are typos/grammatical errors, as well as inconsistencies in use of abbreviations (see e.g. 5.2 vs the previous paragraph), so this should be thoroughly checked by the authors. The non-DKL baselines would be illuminating in the DKL experiments. Only non-ARD kernels are considered when considering different length-scales without justification - it would seem natural to consider different length scales in each dimension as is common practice. The authors give choice between one of two different whitening matrices, however no justification is given to why both are not used. Since the model separates the posterior mean and covariance, a metric that measures both and their tradeoff would be useful, or at least discussion on how this can be inferred from the reported results.\n\nQUESTIONS:\nHow do the DKL models compare to SVGP?\nWhy are only non-ARD kernels considered?\nWhy is a model that utilizes both whitening matrices not considered?\nCan you report performance in a way that considers the tradeoff between accuracy and uncertainty?\n\nLIMITATIONS:\nThese are discussed above from my end. No limitations of the study are listed in the work, however the risk of harm or civilisation collapse based on this work is minimal.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}