Variational Gaussian Processes with Decoupled Conditionals
Authors: Xinran Zhu
Kaiwen Wu
Natalie Maus
Jacob R. Gardner
David Bindel
Venue: NeurIPS 2023 poster (2023)
Keywords: Gaussian processes
variational inference
variational Gaussian processes
Bayesian optimization

Abstract: Variational Gaussian processes (GPs) approximate exact GP inference by using a small set of inducing points to form a sparse approximation of the true posterior, with the fidelity of the model increasing with additional inducing points. Although the approximation error in principle can be reduced through the use of more inducing points, this leads to scaling optimization challenges and computational complexity. To achieve scalability, inducing point methods typically introduce conditional independencies and then approximations to the training and test conditional distributions. In this paper, we consider an alternative approach to modifying the training and test conditionals, in which we make them more flexible. In particular, we investigate decoupling the parametric form of the predictive mean and covariance in the conditionals, and learn independent parameters for predictive mean and covariance. We derive new evidence lower bounds (ELBO) under these more flexible conditionals, and provide two concrete examples of applying the decoupled conditionals. Empirically, we find this additional flexibility leads to improved model performance on a variety of regression tasks and Bayesian optimization (BO) applications.

Variational Gaussian Processes with Decoupled Conditionals
Xinran Zhu'
Kaiwen Wu²
Natalie Maus Jacob Gardner²
David Bindel'
'Corelll University
Uniierrsity of Pennsylvania
(x2584, biooe1)0coreell.. edu (kaiwenwu, nmaus, (acobrg)@seas upenn. edu
Abstract
Variational Gaussian processes (GPs) approximate exact GP inference by using small set of inducing points to form a sparse approximation of the true posterior, with the fidelity of the model increasing with additional inducing points. Although the approximation error in principle can be reduced by using more inducing points, this leads scaling optimization challenges and computational complexity. To achieve scalability, inducing point methods typically introduce conditional inddpen- dencies and then approximations the training and test conditional distributions. In this paper, we consider an alternative approach to modifying the training and test conditionals, in which we make them more flexible. In particular, we investigate decoupling the parametric form of the predictive mean and covariance in the condi- tionals, and learn independent parameters for predictive derive new evidence lower bounds (ELBO) under these more flexible conditionals, and provide two concrete examples of applying the decoupled conditionals. Empir- ically, we find this additional flexibility leads to improved model performance on variety of regression tasks and Bayesian optimization (BO) applications.
and covariance. We
1 Introduction
Gaussian processes (GPs) are powerful class of non-parametric probabilistic models (38] Their flexibility and ability to make probabilistic predictions make them particularly useful in situations
where uncertainty estimates are important [23 30 43 45 56). Although elegant, exact GPs become intractable for large datasets since the computational cost scales cubically with the size of the dataset.
To overcome this limitation, various sparse approximate GP approaches have been developed, mostly relying on sparse approximations of the true posterior [28 29 36). Among those, variational GPs have become increasingly popular because they enable stochastic minibatching and apply to both regression [77 and classification [8] tasks. However, the approximate posterior and variational inference leave unavoidable gap from the exact model. Especially with too few inducing points, variational GPs could yield suboptimal accuracy performance due to a lack of expressiveness. Using more inducing points closes this gap in principle, but leads to additional computational complexity and optimization difficulties [15). While several works have studied ways of allowing more or better placed inducing points improve accuracy of variational GPs, (34 50, 57], is nevertheless challenging to use more than a small fraction of the dataset size in inducing points.
In this work, we take different approach to improving model expressiveness for better accuracy of variational GPs through decoupling conditionals. To start, we point out that inducing point approximations rely two key conditionals (see Sec. 3.I) the training and the test conditional, where the mean and covariance are parameterized by inducing points and kernel hyperparameters. Typically, prior works have focused various relaxations of these conditionals that enable significant computational complexity benefits (36
48 In this work, we consider alternative changes
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(A) Test 0.00, 0.85
(b) Test -0.61, Ameani 0.27, leowart
2
SVGP induring pointss
DCSVGP induring prints (murs)
[c] Test NLL: -0.00, 0.11
(d) Test NLL: 0.990 Ameam: 0.30, leomus: 0.31
2
inducin poinls
induring points (ours)
0-
-2
Latent Function
Training data
laducing points
Figure 1: We compare model fit on 1D latent function using 100 training samples. Solid curves with shading area depict the predictive mean and 95% confidence interval. Using 10 inducing points, in subplot (a) SVGP underfits the latent function with large lengthscale / = 0.85; while in subplot (b), our DCSVGP model (see Sec. (32) fits better and learns different decoupled lengthscales Imman and loovar. Using 100 inducing points, in subplot (c) and (d), both models fits well with similar lengthscales around 0.3. See Sec. [oor more details.
that increase the flexibility of these conditionals. In Sec. 3.2 describe two concrete examples of this idea: decoupling kernel lengthscales and decoupling entire deep feature extractors.
As simple illustration, Fig. illustrates, for example, how decoupled lengthscales improve model fitting. Our model DCSVGP (see Sec. learns decoupled lengthscales (mean and leowar for mean and covariance respectively, and we compare with baseline SVGP (see Sec. 2 which learns one lengthscale
To summarize our contributions: 1) We propose decoupled conditionals in variational GPs to improve model expressiveness for better accuracy. 2) We show that our idea is compatible with the variational framework and rigorously derive an ELBO for variational inference with decoupled conditionals. 3) We provide two conerete examples of applying decoupled conditionals and empirically show the superior performance of our models through extensive regression tasks and BO applications.
2 Background
We assume familiarity with GPs 38 and briefly introduce them for notational clarity. Given observation locations X {x latent function values f = covariance matrix at data X. Givend observations y = f e with Gaussian noise ~ (0,,02)) the posterior distribution of the function value f" at new data point xt is p(f" y) N(p", M*)), where
1n
C Rdd a GP prior induces multivariate Normal belief for K,n), where μ, K,n are the mean values and
f
11 = =x(x") + K.n (K on + o 2T) - - px); x** = K((x, K.n (Knn + o
Model hyperparameters such as kernel hyperparameters and noise are typically estimated by Maximum Likelihood using standard numerical solvers such as LBFGS (35). If no approximations are used, each evaluation to optimize the log marginal likelihood function costs (nn²) flops and Onn²) memory, thus motivating approximate methods for large training datasets.
2.1 Sparse Gaussian Processes
To overcome the scalability limitations of exact GPs, many authors have proposed a variety of sparse GPs by introducing inducing points {n, M
48 49). Inducing points are associated with inducing values fn, which represent latent function values Z under the same GP assumption.

---

Although inducing values fm are marginalized out in the predictive distribution, they typically reduces training costs from (n²)) ) to Onn m²) for each gradient step, where m < n.
SGPR. Based on inducing point methods, Sparse Gaussian Process Regression (SGPR) 49 further introduces variational inference into sparse GPs. Assuming that inducing data is adequate for inference, Titsias [49] further introduces variational distribution @(fm) M(m, S) that approximates the posterior p(fmly). Thus, the predictive density can be approximated as
((()) - 1 pf (() y)df = ff p(FF (m)))()))))))) = NNur(xx),, ((r(x))).or(x))))))
Here the predictive mean (x((x)) and the latent function variance of xf(x*) are:
[r((x)) K
K m, dx(x*)²= = K.. + K. K mm SK K
where K., K.. K.m K_1 1 KT
mm am- The variational distribution of (f)) = (m, S) is then learned
by maximizing the variational ELBO [8
21, which is lower bound the log marginal likelihood:
logp(y) Borrlooppyy/ (f)
where ((f) = (ff ) and [L( lpp)))l))))))))))
(f.)) is the KL divergence 27
SVGP.
To enable data subsampling during training, SVGP [1 avoids analytic solutions the ELBO, but rather decomposes the ELBO as sum of losses over training labels and enables stochastic gradient descent (SGD) 39 training:
oF(x;)2 ,KL (o(fm) (p(mm) 202
ELBOsvaP
Σ ((vlllex)))))) o²)
(2)
where (r(+), (-)) are the predictive mean and latent function variance from Eq. 1 respectivelly and is optional regularization parameter for the KL divergence and can be tuned practice [22
PPGPR. To achieve heteroscedastic modeling and improve predictive variances, the Parametric Gaussian Process Regressor (PPGPR) [22] targets the preditive distribution directly in the loss function. It shares the same predictive mean (-(() and latent function variance ar(-)² ) as SVGP, but uses slightly different stochastic ELBO loss:
ELBOpPGPR
log N(giluu(x)) +r((x)) 3KL(o(Em))ll(t,))
Whitening. In practice, the variational distribution (ff, is usually "whitened" accelerate the optimization of the variational distribution (32). Conventionally. the whitened variational distribution is Ō(fm.. (m, $) (KKm/// 1/2 m, / Skkk() where K1/2 is the square root of K
With whitening, the KL divergence and predictive distribution in ELBO (Eq.2) are both simplified:
(L(()(m))) [p((m.)) = (L (poo()))))( where Po = I),
(x.)) KinK K-112 m, xx(x)) K Kim K-1/2ŠK 1111///K///KK... Km
3
Methodology
we introduce decoupled conditionals under unifying framework for approximate GPs. followed by two concrete examples. In Sec. 3.3 we derive ELBO to do variational inference with decoupled conditionals.
Here, we present variational GPs with decoupled conditionals. In Sec. 3.1 and 3.2
3.1
Conditionals of Approximate GPs
Quinonero-Candee and Rasmussen (36 introduced a unifying framework for sparse GPs. Through the framework, approximate GP can be interpreted as "an exact inference with an approximated

---

Table 1: Exact and approximate conditionals. See Sec. 3.1 and Sec. 3.2 for details. Here Knn
Knn Kam K mm Kmn and K.. K.. K.m
mm Km..
Training Conditionals
Test Conditionals
DTC
(f -N (am K- fm. mmm 0)
p(f fm) N(K Kmm fmm K..)
FTTC (f((fm) = N NN(KKnnm (Km K mi fym diag kmn))
p(f" fm] NNKKKn K Kim fom K..) )
SVGP p(f(fm) -N (Kmm = fmms Knn)
p(f" fm] -N(Kmm I fims K.,
Ours
(f(()m) (Q Qunnn fmi Knn)
( -V(Qum Omm lim: K..)
prior" contrast to "an approximate inference with an exact prior". With inducing points = {z, mm and inducing values fm. most sparse GPs approximate the joint Gaussian prior as:
p(f", f) = p(f", ((ff)) o(
q(
where dependencies between training data and test data f are induced by inducing values fm. Different approximations can be made of the inducing training conditionals (f((f.)) and the inducing test conditionals examples of approximations used in various approximate GP methods. For example, the Deterministic Training Conditional (DTC) approximation uses deterministic training conditional and the exact test conditional 44); The Fully Independent Training Conditional (FITC) approximation uses an approximate training conditional with diagonal corrections and the exact test conditional [48]
Table [ provides the exact expressions of the two conditionals and
3.2 Decoupled Conditionals and Prediction
In this paper, we consider augmenting the exact conditionals by more flexible form that decouples the kernel hyperparameters in the mean and covariance in both conditionals:
training conditional (f [f.) = NV (Qum nm Qmin f. Knn) R p(f(Em))
(3)
test conditional (fm = (VQQ mm Qunn f. K..) 23 p(f"
where the Q and K matrices are formed by the same family of kernel functions, but some kernel hyperparameters are decoupled. See Table for comparison with other approximations. Decoupled conditionals improve model flexibility without relying on more inducing points and it applies to various SVGP-based models. We provide two examples of decoupled models that we will evaluate.
Decoupled lenghthscales. By decoupling the kernel lengthscale into Imean for the mean and Leovar for the covariance, we enable the model learn in settings where the function value changes more rapidly than the variance. In Eq. [ this corresponds to the Q and K kernel matrices being formed using separate lengthscales. For example, an RBF kernel gives
Example 1:
Izz
Zz
|Z
Z
Q 17222
exp
K Kmm
exp
2/2
2/2 covar
mean
Z222EZ
We will denote the application of decoupled lengthscales to SVGP as Decoupled Conditional SVGP (DCSVGP).
Example 2: Decoupled deep kernel learning. Wilson et al. 53 proposed deep kernel learning (DKL) which stacks deep neural network feature extractor h with GP layer. Combining DKL with variational GP models is straightforward, with the feature extractor h learned through ELBO with all other model (hyper) (54) The feature extractor can then be decoupled: /m for the mean and one /evwar for the covariance. Again using the RBF kernel as a base example:
(_ | (2:) - 11mean (2,, 112
Qmm
exp
212
22,Z,ZZ
|/ (z,) heooaa (o,oarzZ)) (1
K mm
exp
2/2
We denote the application of decoupled deep feature extractors to SVGP as SVGP-DCDKL.

---

Prediction. Using decoupled conditionals, the predictive posterior at a new point x" is
df = =N(y"" (Vy""ir(x")) or(xx)² + I where the predictive mean ixp(x*) and latent function variance ör of(x*)² are similar to Eq.
gly') a(y")= p(y"|()) ((
(+r(x") Qam Qmn m, Ōx(x")² K.. Qem Q SQ Qm.
3.3 The Evidence Lower Bound (ELBO)
In this section, we derive an ELBO for DCSVGP model fitring (other examples of decoupled models follow the same derivation). All model parameters and hyperparameters
learned by maximizing the resulting ELBO. Following the standard variational inference (2], we approximate
the true posterior distribution p(f, fmly)) by the variational distribution q(f, f.))) and miminize the
KL divergence: min (L(g(f, ((f, fm (y). In the standard case, qf,, fm) ((f(ff))((fm) but
decoupled model has qf, fm (f (f( since it further approximates the training conditional
p(f(fmm by a decoupled one model:
This difference leads to the following ELBO for the decoupled
log(p(y)) 2 ELBO(4) = [[log(p(y|[ bn)) (m))p(E,,, Gm)
Ør (x,))
15 (()(m))((()))
log. log.N(vslr(xx)))
df
20²
(p
oz (x,))
(f(fm) df
Nogl((vl)((xx)))
(L(o(fm))ll((mm)) - (f (ff pf((fm)
202
ox1)),²
log log.((ylū+((x))...o 0²)
-L((((E))ll((..)) - ExCt..) KL
(f Il
20²
We refer to App. A.1 for additional derivation details. Adding regularization parameters ;1 and 82 to the KL divergence terms as often done in practice, the ELBO for DCSVGP is
or(x;)²
ELBODcSvaP Σ
KL (o( (pp B252
(5)
202
where the predictive mean (x;( and latent function variance or (x,)² are same as Eq.
(x,) = Qim Q m, or(x;)² K2 + Qsm Q SQm Qmi-
(6)
The explicit expression of Ω The term can be computed explicitly (see App. A.I
Eo((..)) KKL (
Bffm T Tffm (Tr(TS) + m Tm)
where T A
Qum Q-11
mi and Φ(f,.) N(m,
Comparing ELBODDSVGP and ELBOsvGP. The ELBOSvGP in Eq 2 and ELBOpesVGP in Eq. both consist of an approximate likelihood term and KL divergence part. There two differences: ELBOppsvGP involves different predictive mean jur (x:) and variance ãr (x,)² derived from decoupled conditionals, Eq. [6] 2) ELBOpcsvGP contains additional KI. divergence term Ω. which is an expected KL divergence of the two training conditionals over the variational distribution o(f, (ffm). regularizing the difference between the decoupled conditional fm) and the exact one p(f ff..)
The regularization parameter. In Eq. 5 ELBOpcsvar contains two KL divergence terms with regularization parameters 31 and B2, respectively. Varying 31 controls the regularization on prior similarities, same SVGP, while varying 32 controls the regularization on the difference between the decoupled conditional and the exact one. In the limit where B3 - +00, decoupling is disallowed and DCSVGP degenerates SVGP. See Sec. 5.2 for more discussion and empirical study.
5

---

Whitening, Decoupling the mean and covariance hyperparameters introduces a challenge where one can whiten with either Kmn or Q with the drawback of only simplifying one term. K- whitening simplifies the KL divergence but leaves the predictive distribution significantly "less linear", while Q- --whitening does the opposite. For example, the predictive distribution from the two whitening choices are:
Q-whitening je (X:) Qim 1/22 m, xx()))² Kii + Qinn Q Q S/222/0011/220000 Q/ mis
K-whitening jir(x;,) = Qim Q K1/2
mm m, or(x;)² K Qim Q
1 SK1/2 Q mm
Qmi-
mmmm
mm
Empirically Q-whitening performs better because simplified predictive distribution is more favor- able for model fitting. See App. A.2 for KL divergence terms and derivation details.
4 Related Work
Variational GPs have been successfully extended to various settings 13
19
55
54].
Among these vast enhancements, much attention has been devoted the study of inducing points, the core part that leads to an expressiveness-an wawpeveeesss-dddcoopppe and-compyexity trade-off. Many works have studied
better placement of inducing points, including selecting from training data p 24
44
47
7 34
and from the input domain [40 50]. but typically inducing points are learned as model parameters
48 On the other hand, several schemes have been developed to reduce complexity and allow more inducing points, e.g. Fourier methods [19, [29 and methods that decouple inducing points used in variational mean and covariance 88 15
However, more inducing points could be less
informative under certain data characteristics 16 16 50 or result in suboptimal model fitting [S 57)
Therefore, our work takes different direction to improve variational GPs via more flexible mean and covariance modeling rather than replying on more inducing points. The work that is most closely related to ours is the ODSVGP model [4] that decouples the inducing points for mean and variance to allow more inducing points used in mean modeling. With similar decoupling idea, are motivated by increasing model ficxibility rather than reducing model complexity and we simply decouple the kernel hyperparameters with negligible additional costs.
5 Experiments
We evaluate the performance of decoupled models proposed in Sec. 3.2 DCSVGP (variational GPs
using decoupled lengthscales) and SVGP-DCDKL. (variational GPs with deep kernel learning using decoupled deep feature extractors). Because PPGPR [22 is orthogonal our decoupling method, similarly we also evaluate DCPPGPR (decoupled lengthscales) and PPGPR-DCDKS feature extractors). All experiments use an RBF kernel and a zero prior mean and are accelerated through Torch 14] on a single GPU, Code is available at attttts://githii com/ wwt.ccooo/iiaaeeee xiiranzhuuu Variational-Coe GP wew.aeeooooeeeeoooooiiooooooooio
(decoupled
5.1 Regression Tasks
We consider 10 UCI regression datasets [00 with up to 386508 training examples and to 380 dimensions. We present main results with non-ARD RBF kernels, the Q-whitening scheme described in Sec. 3.3 and set /3 1e-3. Results are averaged over 10 random dataset splits. For additional results with ARD kernels and K K-whitening, see App. B.I.4aand and B.1.5] For more experiment setup and training details such as the number of inducing points, we refer to App. B.I
DCSVGP and DCPPGPR. We first compare DCSVGP to baseline SVGP and ODSVGP 4 and similarly compare DCPPGPR to PPGPR 22 and ODPPGPR. Table 2 shows the test RMSE and NLL results. We observe that DCSVGP (and similarly DCPPGPR) yields the lowest RMSE and NLL on 8 datasets, demonstrating the improved flexibility of our method. We report all learned lengthscales in App. I. On the other hand, we also note that on Protein and Elevators, DCPPGPR slightly overfits with worse test NL... We show in Sec. 5.2 that increasing /3 resolves this issue. We also find that decoupling lengthscales generally improves model calibration (see App. B.1.3 for results and discussions).

---

Table 2: Test RMSE and NLL on 10 regression datasets (lower is better). Results are averaged over 10 random rrin/validation/test splits. Statistical significance is indicated by bold. See the supplement for standard errors.
Test RMSE
Pol
Elevators
Bike
Kin40k
Protein
Keggdir
Slice
Keggundir
3Droad
Song
SVGP
0.313
0,380
0.294
0,186
0.662
0.089
0.131
0.122
0.511
0.797
ODSVGP
0.321
0.373
0.222
0.175
0.667
0.093
0.087
0.121
0,534
0.794
DCSVGP
0.156
0,379
0,286
0.150
0.604
0.086
0.039
0.121
0.434
0.777
PPGPR 0.306
0.392
0.377
0.282
0.659
0.091
0.205
0,125
0.552
0.780
ODPPGPR
0.333
0.376
0.277
0,394
0.647
0.090
0.092
0.123
0,565
0.778
DCPPGPR
0.178
0,395
0.348
0,226
0.632
0.089
0.042
0.124
0,543
0.779
Test NII
Pol
Flevators
Bike
Kin40k
Protein
Keggdir
Slice
Keggundir
3Droad
Song
SVGP
0.331
0,452
0.207
-0.188
1.013
-1.018
-0.409
-0.683
0.752
1.192
ODSVGP
0.278
0,433
-0,094
-0.354
1.011
-1.029
-0.578
-0,694
0.797
1.187
DCSVGP
-0.373
0.450
0.165
-0,502
0.919
-1.047
-1.293
-0,697
0.586
1.166
PPGPR -0.056
0.377
-0.715
-0.772
0.804
-1.606
-0.906
-1.801
0,260
1.112
ODPPGPR -0.064
0.382
-0.842
-0,972
0,812
-1.603
-0.937
-1.791
0.316
1.111
DCPPGPR -0.588
0.403
-0,901
-1.067
0.854
-1.648
-1.574
-1,904
0.227
1.109
Table 3: Test RMSE and NLL on 10 regression datasets (lower is better). Results are averaged over 10 random train/vat validatioo//estt splits. Statistical significance is indicated by bold. See the supplement for standard errors.
Test RMSE
Pol
Elevators
Bike
Kin40k
Protein
Koggdic
Slice
Keggundir
3Droad
Song
SVGP-DKL
0.0614
0.343
0.013
0.067
0.598
0.0864
0.0189
0.120
0.329
0.773
SVGP-DCDSS
0.0513
0.343
0.011
0.047
0.577
0.0855
0.0223
0.119
0.288
0.774
PPGPR-DKL
0.0847
0.343
0.030
0.082
0.597
0.0871 0.0870
0.0176
0.121 0.120
0.375
0.771
PPGPR-DCDKL.
0.0593
0.346
0.027
0.061
0.594
0.0171
0.350
0.771
Test NLL
Pol
Elevators
Bike
Kin40k
Protein
Keggdir
Slice
Keggundir
3Droad
Song
SVGP-DKL
-1.888
0.347
-2.590
-1.227
0.905
-..44
-2.550
-0.712
0.320
1.161
SVGP-DCDKL
-1600
0.348
-2.786
-1.637
0.888
-1.052
-2.379
-0.713
0.173
1.164
PKG
-2.324
0.303
-3.096
-1.739
0.712
-1.627
-2,625
-1.951
-0.232
1.098
PPGPR-DCDKL
-2.584
0.358
-3.192
-2.186
1.194
-1.650
-2.676
-1.911
-0.340
1.117
SVGP-DCDKL and PPGPR-DCDKL. We then compare SVGP-DCDKL with baseline model SVGP-DKL 53 and comare OPGPR-DCDKS with baseline model PPGPR-DKL [22. Here, all models learn inducing points in the input space rather than in the feature space and we refer to App. B.1.6 for more discussion this choice and supporting results. Table 3 shows the test RMSE and NLL and we observe that DCDKL yields better (or equivalent) RMSE and NLL on all datasets but the Slice dataset; PPGPR-DCDE always yields better (or equivalent) RMSE and only obtains worse NLL on 3 datasets due to an overfitting issue similar to DCPPGPR.
5.2 Ablation Study On 82
In Sec. 5.1 we evaluate decoupled models with fixed 82 1e-3 and observe mostly superior performance except for occasional overfiting with DCPPGPR and PPGPR-DCDKL_ Here, we study how the regularization parameter B3 affects model performance. We evaluate DCPPGPR with 32 E {0, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0} on two datasets- and Protein- that represent our general findings.
The first row of Fig. shows performance on Pol: DCPPGPR gets lower RMSE and NLL with decreasing 32, and 52 + desirable. We also see how the decoupled lengthscales Imean and levar diverge from PPGPR's lengthscale as 82 decreases. The second row of Fig. 2 shows results on Protein: DCPPGPR gets better RMSE but worse NLL as 32 decreases and thus overfits when 32 < 0.1. In this case, 32 = 0.1 is the best. We refer to App. B.2 for more supporting results, and conclude that 1) small 32 or even 0 is usually ideal and 2) adding back some small regularization 32 = 0.1 appears to resolve the instances of overfitting noticed.
7

---

The RMSE
Tol: SLL
fasan
Aarvar
00 Pa
-122
0.26
0.6
0.4
0.:31
0.300
11.6
0.36
0.291
M44
0.75
0.12
0.000
100-7 30
10 10-1
10 10 10 10t
10
10
10 10
10-11 10
107
Figure 2: We evaluate DCPPGPR with varying 32 dataset Pol (first row) and Protein (second row). Results are averaged over 10 random dataset splits with standard errors included. Solid blue lines show DCPPGPR with nonzero 32 and green dashed lines show DCPPGPR with B2 0. Baseline PPGPR is shown in orange dotted line. First two columns contain test RMSE and NLL (lower is better). Last two columns show the learned decoupled lengthscales Imean and Lovar PPGPR has equal Imcan and loovar due to no decouling. See Sec. 5.2 for details.
(dim=12)
Ranolazine (dim=256)
300
0.82
Mean Reward
Objective Value
Mean Reward
0.80
200
SVGP
100
SVGP
0.76
PPGPR-DKL
DCSVGP
DCSVGP
0.74
PPGPR-DCIS
2000 3000
4000
5000
200
400
600
800
1000
50K
100K
150K
200K
Number Evaluations
Number
Evaluations
Figure 3: Optimization performance in terms of objective values on tasks Rover (dim=60), Lunar (dim=12) and Ranolazine (dim=256) shown (higher is better). We compare baseline GP models (SVGP or PPGPR-DKL.) with their decoupled counterparts (DCSVGP or PPGPR-DCDKL). Decou- pling conditionals results in significant improvement on all tasks. See Sec. 5.3 for details.
5.3 Applications to Bayesian Optimization (BO)
In this section, apply our decoupled variational GPs to BO tasks 13. We consider 3 real BO tasks: rover trajectory planning problem (Rover) 222 52 the lunar landing reinforcement learning (Lunar) [1). and challenging molecule design task Ranolazine MPO (Ranolazine) (3] We take TuRBO 12 as the BO algorithm and use different variational GPs in TuRBO. On Rover and Lunar, first reduce the problem dimension to 256 using a Variational Autoencoder (VAE) (26). then perform BO in the latent We compare PPGPR-DKE with PPGPR-DCDKL. Sec.5.2
SVGP with TuRBO and compare SVGP with DCSVGP. On Ranolazine
space using PPGPR-DKL with TuRBO 33
Ranolazine. Consistent with our ablation study in Sec. 5.2 we find that 32 = performs the best on Rover and Ranolazine while Lunar requires more regularization with B2 0.1. Fig. summarizes optimization performance averaged on at least 10 runs all decoupled models outperform their coupled counterparts, showing the advantage of decoupling conditionals BO applications.
5.4 Plausible Extensions
We further explore extensions of decoupled conditionals achieve more flexible predictive mean and variance of variational GPs. Beyond decoupling the same parametric form, we could model the predictive mean and covariance differently for better Despite lack theoretical grounding, we evaluate two possibilities, both targeting more complex predictive mean than the covariance.
Replace the predictive mean of SVGP by a neural network. In fact, the importance of the parametric form of the covariance field is mentioned in Jankowiak et [22]: "a good ansatz for the predictive variance is important for good uncertainty prediction". They suggest, but never implement

---

Table 4: Test NLL on 10 regression datasets (lower is better). Results are averaged over 10 random train/validation/eest splits. Best ones with statistical significance are bold. See Sec. 5.4 for details.
Pol
Fllevators
Bike
Kin40k
Protein
Keggdir
Slice
Keggundir
3Droad
Song
SVGP 0.331
0.452
0.207
-0.188
1.013
-1.018
-0.409
-0.683
0.752
1.192
DCSVGP
-0.373
0.450
0.165
-0,502
0.919
-1.047
-1.293
-0.697
0,586
1.166
NNSVGP
-1.050
0,419
-1,231
-0.827
0.895
-1.60
-0.948
-0.725
0.776
1.171
PPGPR -0.056
0.377
-0.715
-0.772
0.804
-1.606
-0,906
-1.801
0.260
1.112
DCPPGPR -0.588
0.403
-0.901
-1.067
0.854
-1.648
-1.574
-1,904
0.227
1.109
NNPPGPR -1.169
0,552
-1,560
-1.074
0.766
-1,514
-1.369
-1.632
0.412
1.113
Table 5: Test RMSE and NLL on 10 regression datasets (lower is better). Results are averaged over 10 random vrann/aliiatiion/esst splits. Best ones with statistical significance are bold. See Sec. 5.4 for details.
Test RMSE
Pol
Elevators
Bike
Kin40k
Protein
Keggdir
Slice
Keggundir
3rood
Song
SVGP-DKL
0.0614
0.343
0.013
0.067
0.598
0.0864
0.0189
0.120
0.329
0.773
SVGP-DCDKL.
0.0513
0.343
0.011
0.047
0.577
0.0855
0.0223
0.119
0.288
0.774
SVGP-McanDKL
0.0576
0.344
0.027
0.048
0.576
0.0859
0.0259
0.119
0.285
0.771
PPGPR-DKE
0.0847
0.343
0.030
0,082
0.597
0.0871
0.0176
0.121
0.375
0.771
PPGPR-DCDSS 0,0593
0.346
0.027
0.061
0.594
0.0870
0.0171
0.120
0,350
0.771
PPGPR-MemDKLL
0.0736
0.344
0.055
0.062
0,594
0.0868
0.0223
0.121
0.349
0.771
Test NLL
Pol
Blevators
Bike
Kin40k
Protein
Keggdir
Slice
Keggundir
3rood
Song
SVGP-DKI,
-1.388
0.347
-2.900
-1,227
0.905
-1.045
-2.550
-0.712
0.320
1.161
SVGP-DCDKL
-1.600
0.348
-2.786
-1.637
0.888
-1.052
-2.379
-0.713
0.173
1.164
SVGP-MeanDKL
-1.450
0.349
-1,989
-1.645
0.883
-1.053
-1.777
-0.715
0.163
1.160
PPGPR-DKU DKL
-2.324
0.303
-3,096
-1.739
0.712
-1627
-2.625
-1.951
-0.232
1.098
PPGPR-DCDKL
-2.584
0.358
-3.192
-2.186
1.194
-1650
-2.676
-1.911
-0.340
1.117
PPGPR-MeanDKK
-2.362
0.312
-2.831
-2.150
1.048
-1,666
-2.175
-1.906
-0.344
1.099
or evaluate, that one could replace the mean function with a neural network that inducing points are only used for variance prediction. We empirically evaluate this idea, denoted as NNSVGP (and NNPPGPR). With totally flexible predictive mean, NNSVGP and NNPPGPR yield best test RMSE over 9 out of 10 datasets (see supplement for the RMSE table). However, with the predictive mean fully independent of the GP framework. NNSVGP and NNPPGPR yield worse NLL typically. Table reports test NLL, where results are generally mixed.
Simplified deep kernels for covariances. In the DKL setting, instead of using decoupled feature maps, one could use entirely different neural network architectures for the predictive mean and covariance. We empirically find that a simpler feature mapping, or even no feature mapping, for the covariance still make a plausible model with good performance but smaller training cost. We denote the method with no feature mapping for the covariance as SVGP-McanDKL and PPGPR-MeanDKL
We present results in Table 5 We observe SVGP-MeanDKL does not significantly outperform the decoupled models SYGP-DCDKL but it improves baseline SVGP-DKL on 8 out of 10 datasets. PPGPR-McanDKS performs similarly in terms of the RMSE metric (better than baseline PPGPR- DKL but worse than SOGGR-DDDDLL However, in terms of NLL, it gives poor uncertainty prediction (worse NLL). This suggests that it is more beneficial to have different models (decoupled feature extractors SVGP-DCDKS conditional mean (S LVGG-MMeaDDKK or having same models (SVGP-DKL), and SVGP- model is more robust than the PPGPR-bare based model in uncertainty prediction.
for conditional mean and covariance rather than only modeling the
6 Conclusion
Variational GPs scale approximate GP inference to millions of training examples but may yield suboptimal accuracy due to lack of model expressiveness or poor model fitting. We propose simple idea to improve model expressiveness by decoupling the parametric form of the mean and variance in the conditionals of variational GPs. We derive an ELBO for our model with decoupled conditionals, which end up being similar to the ELBO of standard variational GP with an additional regularization

---

term. Our method is simple yet effective, and it applies to various variational GPs. We provide two concrete examples, one decoupling kernel lengthscales in the basic variational GP setting and one decoupling the feature mapping in the deep kernel learning setting. Through extensive empirical study, we show that the decoupled conditionals effectively improve the accuracy of variational GPs in terms of both mean and uncertainty prediction. We also empirically study two plausible extensions of our method, motivated by the idea of modeling the mean and variance differently, but we conclude that they not as effective as our decoupled models. Current limitations of our work and therefore future directions include but not limit to: 1) the application of decoupled conditionals to more SVGP-based models; 2) further generalization of flexible forms of training and testing conditionals that improve model performance; 3) the application of decoupled variational GP framework to tasks other than regression or BO, such as classification tasks.
Acknowledgements
We acknowledge support from Simons Foundation. This work was supported by a grant from the Simons Foundation (601956, DSB). JRG is supported by NSF award IIS-2145644.
References
[1] Adam, S. Eleftheriadis, A. Artemev, N. Durrande, and J. Hensman. Doubly sparse variational Gaussian processes. In International Conference on Artificial Intelligence and Statistics, pages PMLR, 2020.
[2] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):599-87,, 2017.
[3] K. Blomqvist, S. Kaski, and M. Heinonen. Deep convolutional Gaussian processes. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2019, Würzburg, Germany, September 16-20, 2019, Proceedings, Part II. pages 582-597. Springer, 2020.
[4] V. Borovitskiy, A. Terenin, P. Mostowsky, et Matérn Gaussian processes on Riemannian manifolds. Advances in Neural Information Processing Systems, 13:1266--1237,, 2020.
[5] N. Brown, M. Fiscato, M. H. Segler, and A. C. Vaucher. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling. 59(3):1096-110 2019.
[6] D. Burt, C. E. Rasmussen, and M. Van Der Wilk. Rates of convergence for sparse variational Gaussian process regression. In International Conference on Machine Learning, pages 862-871. PMLR, 2019,
[7] D. Burt, C. E. Rasmussen, and M. Van Der Wilk. Convergence of sparse variational inference in Gaussian processes regression. The Journal of Machine Learning Research, 21 21(1):5100-5182, 2020.
[8] C.-A. Cheng and B. Boots. Variational inference for Gaussian process models with linear complexity. Advances in Neural Information Processing Systems, 30, 2017.
[9] L. Csató and M. Opper. Sparse representation for Gaussian process models. Advances in neural information processing systems, 13, 2000.
[10] D. Dua and C. Graff. UCI Machine Learning Repository. Irvine, CA: University of California, School of Information and Computer Science, 2017. URL http://archive. icss uci edu/
m1
[11] V. Dutordoir, N. Durrande, and J. Hensman. Sparse Gaussian processes with spherical harmonic features. In International Conference on Machine Learning, pages 2793-2802. PMLR, 2020.
[12] D. Eriksson, M. Pearce, J. R. Gardner, Turner, and M. Poloczek. Scalable global optimization via local Bayesian optimization. In Advances in Neural Information Processing Systems, volume 32, 2019.
[13] Frazier. Bayesian optimization. Recent advances optimization and modeling of contemporary problems, pages 255-278. Informs, 2018.
10

---

[14] J. R. Gardner, G. Pleiss, D. Bindel, K. Q. Weinberger. and A. G. Wilson. Gpytorch: Blackbox mearix-mmatixx Gaussian process inference with GPU acceleration. In Advances in Neural Information Processing Systems, volume 31, 2018.
[15] M. Havasi, J. M. Hernández-Lobato, and J. J. Murillo-Fuentes. Deep Gaussian processes with decoupled inducing inputs. arXiv preprint ar0i:: 02939, 2018.
[16] J. Hensman and N. D. Lawrence. Nested variational compression in deep Gaussian processes. arXiv preprint arXiv: 14/2. 1370, 2014.
[17] Hensman, Fusi, and N. D. Lawrence. Gaussian processes for big data. In Proceedings of the Twenty-Ninth Conference Uncertainty in Artificial Intelligence, page 282-290, Arlington, Virginia, USA, 2013. AUAI Press.
[18] Hensman, A. Matthews, and Z.. Ghahramani. Scalable variational Gaussian process classifica- tion. In Artificial Intelligence and Statistics, pages 351-360. PMLR, 2015.
[19] J. Hensman, N. Durrande, A. Solin, et al. Variational fourier features for Gaussian processes. J. Mach. Learn. Res., 18(1):5537-5558 2017.
[20] T. N. Hoang. Q. M. Hoang, and B. K. H. Low. A distributed variational inference framework for unifying parallel sparse Gaussian process regression models. In International Conference on Machine Learning, pages 382-391. PMLR, 2016.
[21] M. D. Hoffmann, D. M. Blei, C. Wang, and J. Paisley, Stochastic variational inference. Journal of Machine Learning Research, 14(5), 2013.
[22] M. Jankowiak, G. Pleiss, and J. Gardner. Parametric Gaussian process regressors. In Interna-
tional Conference on Machine Learning, pages 4702-4717 4712. PMLR, 2020.
[23] D. E. Jones, D. C. Stenning, E. B. Ford, R. L. Wolpert, T. J. Loredo, C. Gilbertson, and X. Dumusque. Improving exoplanet detection power: Multivariate Gaussian process models for stellar activity. arXiv preprint arXiv; 1711: 1.01318, 2017.
[24] S. Keerthi and W. Chu. A matching pursuit approach to sparse Gaussian process regression. Advances in neural information processing systems, 18, 2005.
[25] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, 2014.
[26] D. Kingma and M. Welling. Auto-encoding variational Bayes. arXiv preprint
131266 14, 2013.
[27] S. Kullback and R. A. Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1))79-86, 1951.
(28) M. Lázaro-Gredinn and A. EggueirasViidal Inter-domain Gaussian processes for sparse in- ference using inducing features. Advances in Neural Information Processing Systems, 22, 2009.
[29] M. Lázaro-Gredilla, J. Quinonero-Candel C. E. Rasmussen, and A. R. Figueiras- Vidal. Sparse spectrum Gaussian process regression. The Journal of Machine Learning Research, 11:1865- 1881, 2010.
[30] M. Lorenzi, G. Ziegler, D. C. Alexander, and S. Ourselin. Efficient Gaussian process modelling and prediction of image time series. In International Conference on Information Processing in Medical Imuging, pages 626-637. Springer, 2015.
process-based based
[31] W. J. Maddox, S. Stanton, and A. G. Wilson. Conditioning sparse variational Gaussian processes for online decision decision-making. Advances in Neural Information Processing Systems, 34:6365 (4:7365-6379, 2021.
[32] A. G. d. G. Matthews. Scalable Gaussian process inference using variational methods. PhD thesis, University of Cambridge, 2017.
[33] N. Maus, H. Jones, J. Moore, M. J. Kusner, J. Bradshaw, and J. Gardner. Local latent space Bayesian optimization over structured inputs. Advances in Neural Information Processing Systems, 35:34505-34511 2022.
[34] H. B. Moss, S. W. Ober, and V. Picheny. Inducing point allocation for sparse Gaussian processes in high-throughput Bayesian optimisation. In International Conference on Artificial Intelligence and Statistics, pages 5213-5230. PMLR, 2023.

---

[35] J. Nocedal and S. Wright. Numerical optimization. Springer Science & Business Media, 2006.
[36] J. Siinonero-Caander and C. E. Rasmussen. A unifying view of sparse approximate Gaussian process regression. The Journal of Machine Learning Research, 6:1939-1998 2005.
[37] A. Ramdas, N. Gareía Trillos, and M. Cuturi. On wasserstein two-sample testing and related families of nonparametric tests. Entropy, 19(2):47, 2017.
[38] C. E. Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning, pages 63-71. Springer, 2003.
[39] H. Robbins and S. Monro. A stochastic approximation method. The Annals of Mathematical Statistics, pages 400-407, 1951.
[40] S. Rossi, M. Heinonen, E. Bonilla, Z. Shen, and M. Filippone. Sparse Gaussian processes revis- ited: Bayesian approaches
mnducing-variabll approximations. In International Conference on Artificial Intelligence and Statistics, pages 1837-1845. PMLR, 2021.
[41] H. Salimbeni, C.-A. Cheng, B. Boots, and M. Deisenroth. Orthogonally decoupled variational Gaussian processes. Advances in neural information processing systems, 31, 2018.
[42] J. Schreiter, D. Nguyen-Tuong, and M. Toussaint. Efficient sparsification for Gaussian process regression. Neurocomputingg, 192:29-37, 2016.
[43] P. Schulam and S. Saria. Reliable decision support using counterfactual models. In Advances in Neural Information Processing Systems, volume 30, 2017.
[44] M. W. Seeger, C. K. Williams, and N. D. Lawrence. Fast forward selection to speed up sparse Gaussian process regression. In International Workshop on Artificial Intelligence and Statistics, pages 261-261. PMLR, 2003.
[45] B. Shahriari, K. Swersky, Z. Wang. R. P. Adams, and N. De Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, (04(1):148-17) 175, 2015.
[46] B. W. Silverman. Some aspects of the spline smoothing approach to non-parametric regression curve fitting. Journal of the Royal Statistical Society: Series B (Methodological), 1-(1):1-21, 1985.
[47] A. Smola and P. Bartlett. Sparse greedy Gaussian process regression. Advances in neural information processing systems, 13, 2000.
[48] E. Snelson and Z.. Ghahramani. Sparse Gaussian processes using pseudo-inputs. Advances in neural information processing systems, 18, 2005.
[49] M. Titsias. Variational learning of inducing variables sparse Gaussian processes. In Artificial intelligence and statistics, pages 567-574. PMLR, 2009.
[50] A. K. Uhrenholt, V. Charvet, and B. S. Jensen. Probabilistic selection of inducing points in sparse Gaussian processes. In Uncertainty in Artificial Intelligence, pages 1035-1044. PMLR, 2021.
[51] M. Van der Wilk, C. F. Rasmussen, and Hensman. Convolutional Gaussian processes. Advances in Neural Information Processing Systems, 30, 2017.
[52] Z. Wang, C. Gehring, P. Kohli, and S. Jegelka. Batched large-scale Bayesian optimization in high-dimensional spaces. In International Conference on Artificial Intelligence and Statistics, pages 745-754, PMLR, 2018.
[53] A. G. Wilson, 7.. Hu, R. Salakhutdinov, and E. P. Xing. Deep kernel learning. In Artificial intelligence and statisfics, pages 370-378. PMLR, 2016.
[54] A. G. Wilson, 7. Hu, R. R. Salakhutdinov, and F. P. Xing. Stochastic variational deep kernel learning. Advances in neural information processing systems, 29, 2016.
[55] L. Wu, G. Pleiss, and J. P. Cunningham. Variational nearest neighbor Gaussian process. In International Conference on Machine Learning, pages 24114-24130. PMIR, 2022.
[56] Z. Yi, R. Calandra, F. Veiga, H. van Hoof, T. Hermans, Y. Zhang, and J. Peters. Active tactile object exploration with Gaussian processes. In 2016 IEFF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4925-4930. IEEE, 2016.
[57] X. Zhu, J. R. Gardner, and D. Bindel. Efficient variational Gaussian processes initialization via kernel-based least squares fitting. NewrlPS Workshop on Gaussian Processes, Spatiotemporal Modeling, Decision-making Systems, Dec. 2022.
12

---

A Methodology
A.1 The Evidence Lower Bound (ELBO)
Here we privide detailed derivation of the evidence lower bound (ELBO) for the proposed decoupled SVGP model fittin.. The goal of variational learning remains the same:
min (g(f, fm (l)(, fmly) min (E(log(a(f, fm)]] [log(p(ff y))] log(p(y))),
where in the standard case fm. (fm)) but we further approximate the training
conditional fff) by a decoupled one (f f.) and have q fmm = ( Following the
standard variational inference [2), we have the ELBO for decoupled SVGP:
log(p(y)) 2 (L) [F]oo(e(ff, fm.y)) [loog(g(f, fm)))
(7)
(plog(p(y ff, fm)) - KL(g(f, fm (lp() (m))
The first likelihood term in Eq. 7 is
/ (o (m)) fm) df
E [og]ppy/[f. fm))] =
/ ((m)) (ʃ (flostoisIE ( (f))) dfm
/ o(fm) (oog(oiS)E)))CEEE, df dfm
Σ ((wl(())))x o²) (x(x;)
20
where the predictive mean (x;(x;) and latent function variance or ( are (r:(x;) Qim Qmin m, or(x:)² Kis + Qim Qm SQmi Qmi
The second KL divergence term in Eq. ii
/ (flf.. (fmm) log
(f( df
KL(g(f, (l.)(((f...)))))
p(f fm (f(m)
/ (CTE log (f(fmm)
dfdfm I (TIIm)) log ((fm) dr p(fm)
pff
/ (LLC))) (m)) drm + (L((((())l((fmm) RoCC... (ILG (LGm))l))))))))
The KL divergence term is almost the same that in the standard ELBO, with an additional term = ] (ffm (fff.))) (f) dfm.
Analytic expression of Ω We could compute explicitly the analytical expression for the Q term. For notational simplicity, let decoupled and exact training conditionals
(f]ff.) (Q nm Q fm Knn) == -N E), pff (KK nm Kminf fm Kmn) N Σ).
The Ω term is an expectation of quadratic form:
Ω
df.
/ (ffm) ((M) - 222 /
T " - (22) dfm
(8)
Effm T Tfm! (Tr(TS) + mm' Tm)
where T = - A. A Q ra Q_1 Knm K K-1
mm and @(fm) A (m,
13

---

A.2 Whitening
In this section, we provide detailed discussion whitening scheme used the decoupled models. We first present the standard whitening scheme used in variational GP models, then derive two plausible whitening schemes for decoupled models K-whitening and the Q-whitening. We implemented both ways of whitening for decoupled models, and empirically found that the latter (Q-wiitennng) performs better (See Section B.1.5).
A.2.1 The standard whitening
For standard models like SVGP, using the square root of Kmm, the variational distribution "whitened" (reparameterized) in the following way:
1/2 ((fm) = (m, S) ((K
mom m, mi
K-11/SKK11/""
Under such reparametterization, the KL divergence equivalent to a simpler form:
1
KL(g(fm)ll (L((())))l((mm))
2 log [S]
log Kmml
Tr(K mmm)) S)
m² F= m M]
1
2 log S Tr(K 1O2KK1/KKK 1/2K 1K"K SK mmn m K1/2 K 112222122
11111/1///// m M
2 log [S] Tr(S) + m mm M
KL fm (PPo(l)))
where po (fm) (0, I) is the standard Normal, and M is the number of inducing points. Now plug in and S in the predictive distribution p(f), we have the predictive mean prr (x:) and latent function variance (x,)² in simplified forms well:
(x;) = Kim K K-11/m 1/2m.
ox(x,)2 K₁ Kin K Kmm Kms Kin K
K -i/K-1/22K 1/2K K mi-
A.2.2 The K-whitening
For decoupled models like DCSVGP, could similarly use the square root of Kmm whiten the variational distribution:
/V mm m, K K-1//SKK11/
Same as the standard whitening, the KL divergence is simplified:
1 | log [S] + log K Tr(K + m K m M]
(L(((t))ll((mm))
2
[ log S Tr(S) + T M]
2
(L(( (Poo(()))
where (fm)) = (0, I) is the standard Normal, and M is the number of inducing points. However, we plug and S in the predictive distribution pff), the predictive mean (r(x;) and latent function variance oz(x,)² are not simplified in the same way as the standard case:
(x;: = Qim Q-1 11/2
mam m.
or(x;)² = Kii K K-1 mm Kms + Qim Q-1 K2/2
1/2 SK 1/2 mam
Q-1 Qmi-
mm
A.2.3 The Q-whitening
For decoupled models like DCSVGP, another plausible way of whitening for decoupled models is to use the square root of Q
((fm) M(m, S) _N(QQ 1/2
mm m, Q 1/10//1////02 1S/22112/
14

---

In this way, the KL divergence term is equivalent to different form:
(L (lpmm))))(mm))))) [- log [S] + log mmm
+ Tr(K mum S) + m K
M]
mm
2 log Qum
1/2
+ log K + Tr(K mm
1/2 mmm
SQ
1/2 + m mm
1/2 K Q1/2 m M "Q1/ mem
mm
mm wim
1
log Q1/2 SQ mam
log
1/2 K
K
1/2 mm
Tr(K 1om
2
1/2
SQ 1011/11 +
Q mim
K 1-12
1/2 1 Q M
mmm
2
Wam
mmm
nom
2
1
log [S] + log
mm
1/2
K 1/2
K
1/2
Tr( 1/2
mm
1/2
SQ K m Q 1/2 K 1/2K 1/2
mm
Q//2 20
M
1
[ log [S] log LL' + SSL
SL () + m - I M]
2
| log [S] - log K + ( + m k im M]
2
(7( (((
where p(fm) ( K) and K L - T 2 Now plug m and S in the predictive distribution pff), we have the predictive mean jor (x(xx) and latent function variance (x,) simplified in similar way as the standard case:
mim and M is the number of inducing points.
(x;) Qim 11122 m,
ox(x;)² K;; Kin K-1 Toms Km + Qim Qm1// 1122SQQ
1/2S1111 1/2 1/2QQi
As discussed in the main paper, it is not surprising that the Q-whitening performs better in practice. This is because the K-whitening simplifies the KL divergence leaves the predictive distribution complicated, while the Q-whitening does the opposite, and complicated predictive distribution makes optimization much harder than complicated KL divergence term.
B
Experiments
B.1 Regression Tasks
Experiment setup Table @joovide details on the 10 UCI regression datasets 10] and the number of inducing points used for each dataset. For ODSVGP, since the goal is to use more inducing points for mean and less for covariance under the same budget, we use the setup in the original paper [41: SVGP uses 4X inducing points, then ODSVGP uses 7X inducing points for mean and 3X inducing optimizer with multistep scheduler to train all models
points for covariance. We use the Adam 25 on all datasets, and we train 300 epochs using training batch size 1024. We selected the best training hyperparameters for SVGP and use the same ones for all models learning rate Ir 5e-3 and multistep learning rate scheduler (multiplicative factor y = 0.2).
B.1.1 Learned lengthscales
Table allso provides learned lengthscales of all models, averaged over 10 random dataset splits, as supplementary results for Table As shown in Table(6) we observe how lengthscales are decoupled in DCSVGP and DCPPGPR the lengthscale for mean function /mean is typically smaller than the shared lengthscale I, while the lengthscale for covariance function leowar typically larger than shared lengthscale 1. This is indeed consistent with the motivation of decoupling lengthscales since smaller lengthscale for the mean function would typically lead to a better mean fit when the number of inducing points is limited compare total training data; when the mean function is well fit, the lengthscale for the covariance can larger to further reduce the predictive covariance.
B.1.2 Error bars of main results
Here we provide standard errors of main results Table 2 and Table 3 see Figure 4 and Figure respectively. Error bars for other similar results, i.e. Tables in the Appendix, are similarly small and not informative and therefore avoid redundant results.
15

---

Table 6: We present details on UCI datasets and number of inducing points used in SVGP, DCSVGP, PPGPR and DCPPGPR here. We also compare the learned lengthscale values (averaged over 10 random dataset splits): lengthscale I of baseline models SVGP, ODSVGP, PPGPR and ODPPGPR, lengthscales mean and covar of decoupled models DCSVGP and DCPPGPR. We observe that the lengthscale for mean meaan typically smaller than the shared lengthscale I, while the lengthscale for
covariance leowar is typically larger than shared lengthscale I. See App. B.I and App. B.I.1 for
more details.
Pol
Elevators
Bike
Kin4Okk
Protein
Koggdir
Slice
Keggundir
3Droad
Song
Total number of data
11250
12449
13034
30000
34297
36620
40125
47706
326155
386508
Input dimension
18
8
9
380
26
90
#inducing points
500
500
500
800
800
800
800
800
1000
1000
SVGP I
0.976
1.370
2.330
1.220
0.260
1.150
19.600
2.270
0.120
1.110
ODSVGP
1.418
1.561
1.710
0,412
1.726
20.694
2.665
1.644
DCSVGP Inean
0.291
0.761
1.174
0.673
0.104
0.791
4.956
1.273
0.038
0.346
DCSVGP Agous
3,304
2.257
3.459
1.721
0.555
3.190
28. 112
5.876
0.170
2.124
PPGPR 0.789
1.060
2.890
1,300
0,200
1.030
21.000
2.510
0.110
0.490
ODPPGPR
1.079
1.171
3.387
1.825
0.261
1.298
20.536
2.583
0.137
0.591
DCPPGPR Ineon
0.290
0.790
1.423
0.797
0.122
0.923
5.053
2,161
0,100
0.349
DCPPGPR lenon
3.811
1.361
4.556
1.793
0.298
2.122
26.734
4.814
0.138
0.640
Ergas
107
0500
230 2311
A1B
130
121
441
130
622
AND
409909
23998
-280
-0.0
3.20 3.0
-2.3 -6.0 +1.0
104
0.0
5.00
1101
123
Figure 4: Test RMSE and NLL results with standard errors on UCI datasets, same results as Table main paper. We compare DCSVGP SVGP and ODSVGP and compare DCPPGPR to PPGPR and ODPPGPR. Row and include RMSE results and row 3 and 4 include NLL results (same row order as Table in the main paper). See App. B.1.2 for more details.
Fdi
Pogi
raggd:
Sice
Fagguner
20rood
54%
190400
50000
Figure Test RMSE and NLL results with standard errors on UCI datasets, same results as Table 3 in main paper. We compare SVGP-DCDKL to SVGP-DKL, and compare PPGPR-DCDKL. to PPGPR-DKL. Row and include RMSE results and row 3 and 4 include NLL results (same row order Table 3 in the main paper). See App. 1.1.2 for more details.
16

---

SVOP. 0.0
DESVER, D-0.18
POPL, D-0.10
DCPPaT D
0.5 Standard Norma
0.4
Model 2-score
0.5-
0-4
0.4
0.3
0.2
0.2
0.0
0.0 -10
0.0 -4
0.0 -2
Figure 6: On dataset Kin40K, we plot z-scores distributions of model prediction (orange curves), and compare with the standard Normal (black dashed curves). The closer the model predictive distribution is to the standard Normal, the better the model is calibrated. Clearly, decoupled models DCSVGP and DCPPGPR are better calibrated compare to their counterparts SVGP and PPGPR, respectively. Quantitatively, in the titles we show the Wasserstein distance D between the two distributions, the smaller the better. See App. B.1.3 for more details.
Table 7: Test RMSE and NLL on 10 univariate regression datasets (lower is better) with ARD kernels. We compare decoupled models DCSVGP and DCPPGPR with their coupled counterparts SVGP and PPGPR. Results are averaged over 6 random train/ validation//est splits. Best ones with statistical
significance are bold. See App. B.1.4 for more details.
Test RMSE
Pol
Elevators
Bike
Kin40k
Protein
Koggdir
Slice
Keggundir
3Droad
Song
SVGP
0.167
0.370
0,089
0,166
0.646
0.089
0.111
0.122
0.376
0.794
DCSVGP
0.078
0.366
0.058
0.116
0.590
0.087
0.041
0.120
0.323
0.772
PPGPR
0.166
0.376
0.127
0.240
0.643
0.092
0.150
0.124 0.123
0.450
0.779
DCPPGPR
0.084
0.372
0.064
0.163
0.614
0.090
0.047
0.458
0.776
Test NLL
Pol
Elevators
Bike
Kin40k
Protein
Keggdir
Slice
Keggundir
3Droad
Song
SVGP
-0.286
0.426
-0,873
-0,315
0.991
-1.027
-0.482
-0.689
0.457
1.188
DCSVGP
-1.102
0.414
-1.453
-0.712
0.901
-1038
-1.279
-0.710
0.289
1.161
PPGPR
-0,871
0,359
-1,818
-0.844
0.783
-1.603
-0.919
-1.900
-0.084
1.108
DCPPGPR
-1.494
0,392
-2.223
-1.177
0.885
-1.622
-1.505
-1.989
-0.083
1.121
B.1.3 Model Calibration
To study how decoupled conditionals affect model calibration, we evaluate the z-score distribution from model predictions on all UCI datasets and compare it to the standard Normal. The closer the z-score distribution from model predictions is to the standard Normal, the better the model is calibrated. As an example, Figure the z-score distributions of SVGP, DCSVGP, PPGPR and DCPPGPR the Kin40k dataset. As Figure 6 shows, the Kin40k dataset, decoupled models DCSVGP and DCPPGPR are better calibrated compare to their counterparts SVGP and PPGPR, respectively. Quantitatively, distribution and standard Normal (see subtitles in Figure 6 for example). the smaller the better. Averaged over all 10 UCI datasets, SVGP and DCSVGP both have D 0.27, PPGPR has D 0.22 and DCPPGPR has the best = 0.17. Therefore, we conclude that decoupling conditionals does not lead to poorly calibrated model, but even improve model calibration on average.
compute the Wasserstein distance D 37 between the model z-score
B.1.4 ARD kernels
Here provide additional results regression tasks using ARD kernels. In Table 2 show that decoupled models outperform the baseline models using non-ARD kernels (kernels with one- dimensional lengthscale). Here, one for each input dimension) for all models and show consistent advantages of using decoupled models. Table summarizes results, similar the main results using non-ARD kernels (Table 2 in main paper), we observe that all decoupled models outperform baseline models.
ARD kernels (kernels with multi-dimensional lengthscales,
B.1.5 Whitening of decoupled models
In Section A.2 we discussed two plausible whitening schemes for the decoupled models and claimed that the Q --whitening is more favorable than the K K-whitening due to the simplified predictive
17

---

Table 8: Test RMSE and NLL on 10 univariate regression datasets (lower is better). Using the DCSVGP model, we compare the K-whitening and Q-whitening discussed in App. A.2 Results are averaged over 10 random vrann/validaiionnesstt splits. Best ones with statistical significance are bold. See App. B.1.5 for more details.
Test RMSE
Pol
Elevators
Bike
Kin40k
Protein
Keggdir
Slice
Keggundir
3Droad
Song
K K-whitening
0.163
0.380 0.379
0,260
0.158
0.635
0.087 0.086
0.041 0.039
0.119 0.121
0.437
0.774
Q-whitening
0.156
0.286
0.150
0.604
0.434
0.777
Test NILI
Pol
Flevators
Bike
Kin40k
Protein
Keggdir
Slice
Keggundir
3Droad
Song
K- K-whitening Q-whitening
-0.099
0.457
0,083
-0.453
0,966
-1.045
-1.243
-0.703
0,591
1.163
-0.373
0.450
0.165
-0.502
0,919
-1.47
-1.293
-0.697
0.586
1.166
Table 9: Test RMSE and NLL on 10 univariate regression datasets (lower is better). Using the DCPPGPR model, we compare the K-whitening and Q-whitening discussed App. A.2 Results are averaged over 10 random train/validation/tess splits. Best ones with statistical significance are bold. See App. B.1.5 for more details.
Test RMSE
Pol
Elevators
Bike
Kin40k
Protein
Keggdir
Slice
Keggundir
3Droad
Song
K K-whitening
0.178
0.387 0.395
0.296
0.214 0,226
0.659 0.632
0.089 0,089
0.046 0.042
0.123 0.124
0.567
0.781
Q-whitening
0.178
0.348
0.543
0.779
Test NLL
Pol
Elevators
Bike
Kin40k
Protein
Keggdir
Slice
Keggundir
3Droad
Song
K-whitening 0.434
0.475 0.403
-0.786
-0,941
0.813
-1.638
-1.722
-1.861
0.277
1.112
Q-whitening
-0,588
-1.067
0.854
-1.648
-1574
-1.04
0.227
1.109
Table 10: Test RMSE and NLL on 10 univariate regression datasets (lower is better) using SVGP- DKL. We compare learning inducing points in feature space with learning inducing points in the input space. Results are averaged over 10 random vrain/vaiiationn/est splits. Best ones with statistical significance are bold. See App. B.1.6 for more details.
Test RMSE
Pol
Elevators
Bike
Kin40k
Protein
Keggdir
Slice
Keggundir
3Droad
Song
feature space
0.0867
0.347
0.067 0.013
0.086 0.067
0.614 0.598
0.0905 0.0864
0.0237
0.122
0.382
0.772
imput space
0.0614
0.343
0.0189
0.120
0.329
0.773
Test NLL
Pol
Elevators
Bike
Kin4Uk
Protein
Keggdir
Slice
Keggundir
3rooad
Song
feature space
-1.014
0.361
-1.292
-1.070
0.931
-1.007
-2.214
-0.691
0.464
1.161
input space
-1.388
0.347
-2.590
-1.227
0.905
-1.045
-2.550
-0.712
0.320
1.161
distribution. Here, empirically compare the whitening schemes and verify such claim. Table g summarizes a comparison of the K K-whitening and Q Q-whitening in the DCSVGP model, and we observe that Q-whitening outperforms K-whitening on out of 10 datasets. Table Table similar results on the DCPPGPR model.
B. DKL discussion
In the main paper, evaluate SVGP-DKL, PPGPR-DKL and their decoupled counterparts SVGP- DCDKL. and PPGPR-DCDKL. on UCT datasets, and we choose to learn inducing points in the input space for all models. Despite the convention of learning inducing points in the feature space, we found that for baseline models SVGP-DKL and PPGPR-DKL, learning inducing points in the input space performs better on the UCI datasets, see Table 10 and Table
B.2 Ablation Study $2
In addition to the ablation study results DCPPGPR in the main paper, here we provide more ablation study results on DCSVGP and DCPPGPR with varying 92 values on all datasets. Together with the representative results shown in the main paper, We obtain the same conclusion that
18

---

Table 11: Test RMSE and NLL on 10 univariate regression datasets (lower is better) using PPGPR- DKL. We compare learning inducing points in the feature space with learning inducing points in the input space. Results are averaged over 10 random train/validation/les splits. Best ones with statistical significance are bold. See App. B.1.6 for more details.
Test RMSE
Pol
Elevators
Bike
Kin40k
Protein
Keggdir
Slice
Keggundir
3Droad
Song
feature space
0.1614
0.346 0.343
0.096 0.030
0.099 0.082
0.620 0.597
0.0953 0.0871
0.0213 0.0176
0.124
0.481 0.375
0.769
input space
0.0847
0.121
0.771
Test NLL
Pol
Elevators
Bike
Kin40k
Protein
Keggdir
Slice
Keggundir
3Droad
Song
feature space
-2.169
0.303
-3.054
-1.754
0.764
-1.574
-2.118
-1.956 -1.951
0.106
1.100
input space
-2.324
0.303
-3.09
-1.739
0.712
-16627
-2.625
-0.232
1.098
Table 12: Test RMSE and NLL on 10 regression datasets (lower is better). We compare DCSVGP with different B2 values. Results are averaged over 10 random train/vllidationttest splits. Best ones with statistical significance are bold. We observe that $2 = 0.001 or even B3 = 0 are good choices.
See App. B.2 for more details.
Test RMSE
Pol
Elevators
Bike
Kin40k
Protein
Keggdir
Slice
Keggundir
3Droad
Song
& 1.0
0.255
0.379
0.290
0.158
0.640
0.087
0.039
0.121
0.452
0.780
& 0.001
0.156
0.379
0,286
0,150
0,604
0.086
0.039
0.121
0.434
0.777
&
0.155
0.379
0.286
0.151
0.605
0.087
0.039
0.121
0,433
0.778
Test NLL
Pol
Elevators
Bike
Kin40k
Protein
Keggdir
Slice
Keggundir
3Droad
Song
8. 1.0
0.120
0.450
0.184
-0.421
0.974
-1.04
-1.274
-0.695
0.632
1.170
B 0.001
-0.373
0.450
0.165
-0,502
0.919
-1047
-1,293
-0.697
0.586
1.166
8.
-0.370
0.450
0.165
-0.497
0.920
-1.039
-1.293
-0.698
0.582
1.167
1) 32 0.001 or even 32 = 0 are good choices for both DCSVGP and DCPPGPR (see Table
and Table 13
2) on rare cases (only on the Pol and Elevators dataset, see Table B3 DCPPGPR could overfit and B2 = 0.1 or 82 = 1.0 would fix the issue and improve the prediction performance.
More detailed results and discussions follows.
ID Example Using the same ID toy example from the main paper, here in Figure show ablation study on this 1D example. Consistently, DCSVGP outperforms SVGP with 10 inducing points, and DCSVGP keeps improving as 32 decreases. We also observe that the decoupled lengthscales meaa and covar differ more and more as 32 decreases and thus allowing the model to better fit mean and covariance differently.
DCSVGP From Table 12 we observe that, DCSVGP favors small B2 values and B3 0.001 or even 32 are good choices. The general trend is that DCSVGP performs better as B2 decreases. To clearly see this trend, in Figure we provide detailed performance study on datasets: the ID synthetic dataset, the Pol dataset, and the Song dataset. We observe that both RMSE and NLL decrease 32 decreases, and the learned lengthscales differ more and more as B2 decreases to better model the mean and covariance differently.
DCPPGPR From Table we observe similar trend that DCPPGPR also favors small Bg values and 82 = 0.001 or B3 0 are still good choices. In the main paper, we observe that DCPPGPR overfits on dataset Elevators and Protein, and on Protein we show how varying 82 improves test NLL in detail (Figure 2 in the main paper). Here we also observe from Table 13 that regularization with 32 = 1.0 would fix the issue for both dataset Elevators and Protein. Figure 13 shows how the test R RMSE and NLL change with 32 on the Elevators dataset we observe that RMSE does not change much as 32 varies, but NI.. decreases as 3n increases, showing how regularization improves test NLL.
19

---

(()
(b) NLJ.4)54, Amsunn 0.06, Jonvarr 0.08
2
DCSVGP
(c) Test NLL.0.17, Amnon: 0.34, Teowar:: 0.52
(d) Tese NLL:-0.33, finea:: 0.28, lcovar: 0.71
2-
DESVUP (suns) 0.
DOSVGP (suns)
22
Latent Punetioo
*
Thiining data
Inducing points
Figure 7: We compare model fit on ID latent function using 100 training samples and 10 inducing points. Solid curves with shading area depict the predictive mean and 95% confidence interval. In subplot (a), SVGP underfits the latent function with large lengthscale = 1.13. From subplot (b) to (d), DCSVGP fits better and better with decreasing 832 (1.0, 0.1, 0.01), We also observe that the decoupled lengthscales mmean and lewarr differ more and more as B2 decreases and thus allowing the model better fit mean and covariance differently. See App. B.2 for more details.
Table 13: Test RMSE and NLL on 10 regression datasets (lower is better). We compare DCPPGPR with different 32 values. Results are averaged over 10 random train/validation/tess splits. Best ones with statistical significance are bold. We observe that DCPPGPR generally favors small 82 values, and regularization would fix rare overfiting (on Elevators and Protein). See App. B.2 for more details.
Test RMSE
Pol
Flevators
Bike
Kin40k
Protein
Keggdir
Slice
Keggundir
3Droad
Song
8g
1.0
0.258
0.393
0.366
0.233
0.645
0.090
0.044
0.124
0.546
0.778
82
0.001
0.178
0.395
0.348
0.226
0.632
0.089
0.042
0.124
0.543
0.779
f/2
0.177
0.393
0.345
0.227
0.634
0.090
0.043
0.123
0.544
0.779
Test NLL
Pol
Elevators
Bike
Kin40k
Protcin
Keggdir
Slice
Keggundir
3Droad
Song
82
-0.287
-0.815
-0.987
0,790
-1.616
-1.524
-1,859
0.237
1.108
82 0.001
-0.588
0.403
-0.901
-1.067
0.854
-1.648
-1.574
-1,904
0.227
1.109
8
-0.616
0.399
-0,899
-1.065
0.855
-16335
-1.538
-1.906
0.225
1.110
Table 14: Test RMSE on 10 regression datasets (lower is better). We compare NNSVGP to DCSVGP and SVGP and compare NNPPGPR to DCPPGPR and PPGPR. Results are averaged over 10 random train/validation/tes splits. Best ones with statistical significance are bold. NNSVGP and NNPPGPR both yield the best RMSE on 9 of 10 datasets.
Pol
Elevators
Bike
Kin40k
Protein
Keggdir
Slice
Keggundir
3Droad
Song
SVGP
0.313
0.380
0.294
0.186
0.662
0.089
0.131
0.122
0.511
0.797
DCSVGP
1.56
0.379
0.286
0.150
0.604
0.086
0.039
0.121
0.434
0.777
NNSVGP
0.069
0.353
0.057
0.103
0,590
0.084
0.019
0.117
0.523
0.778
PPGPR
0.306
0.392
0.377
0.282
0.659
0.091
0.205
0.125
0.552
0.780
DCPPGPR 0.178
0.395
0.348
0.226
0.632
0.089
0.042
0.124
0.543
0.779
NNPPGPR
0.069
0.352
0.057
0.118
0.604
0.085
0.021
0.118
0.623
0.776
B.3 Plausible Extension
RMSE results of NNSVGP We provide the RMSE results of NNSVGP and NNPPGPR here in Table 14 support the observation that they yield the best test RMSE on 9 out of 10 datasets.
20

---

Test IMSR
Toxt NL..
Amsan
0144
211
15
060
0.4
3.5
11:00
0.2
3.
8.005
25
0.4
0.375
1.5
-164
0.790
1.300
22
0.770
Refommne
8.778
1.358
1.8
0.30
13306
0.776
Figure 8: We evaluate DCSVGP with varying 32 the ID dataset (first row), dataset Pol (second row) and Song (third row). Solid blue lines show DCSVGP with nonzero 32 and green dashed lines show DCSVGP with B2 O. First two columns contain test metrics, lower the better. Last two columns show the learned decoupled lengthscales Imean and Leovar We observe that both RMSE and NLL decrease 3a decreases. See App. B.2 for more details.
0.43
Figure 9: We evaluate DCPPGPR with varying 32 dataset Elevators, similar to Figure We observe that test RMSE does not change much as Bg varies, but NLL decreases as B2 increases, showing how regularization improves test NLL. See App. B.2 for more details.
21


Reviews:
[{"review_id": "ING9sJWjIp", "reviewer": "Reviewer_u1uu", "summary": "This paper considers the problem of increasing the expressivity of sparse Gaussian processes without increasing the number of inducing points by considering a decoupled ELBO. In their setup, the gram matrices appearing in the predictive mean and covariance have different parameterizations. They consider two such parameterizations, the first corresponding to learning different length-scales in a non-ARD RBF kernel, and the second using different neural networks in a deep kernel learning setup. An ELBO is derived in this setup, and a version is given for the PPGPR analogue. Whitening is considered with mean-whitening being favored. Numerical examples are given on UCI regression and Bayesian optimization tasks. The effects of differing levels of regularization are examined. Extensions parameterized with further neural networks are considered.", "questions": "How do the DKL models compare to SVGP?\nWhy are only non-ARD kernels considered?\nWhy is a model that utilizes both whitening matrices not considered?\nCan you report performance in a way that considers the tradeoff between accuracy and uncertainty?", "limitations": "These are discussed above from my end. No limitations of the study are listed in the work, however the risk of harm or civilisation collapse based on this work is minimal.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well structured and written. Numerical examples are given and their methods appear to perform well. The new parameter that's introduced in their derivation is studied.", "weaknesses": "There are typos/grammatical errors, as well as inconsistencies in use of abbreviations (see e.g. 5.2 vs the previous paragraph), so this should be thoroughly checked by the authors. The non-DKL baselines would be illuminating in the DKL experiments. Only non-ARD kernels are considered when considering different length-scales without justification - it would seem natural to consider different length scales in each dimension as is common practice. The authors give choice between one of two different whitening matrices, however no justification is given to why both are not used. Since the model separates the posterior mean and covariance, a metric that measures both and their tradeoff would be useful, or at least discussion on how this can be inferred from the reported results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We will address each of your questions below.\n\n> Typos/grammatical errors and inconsistencies in use of abbreviations.\n\nThank you for pointing this out. We will make sure to resolve all inconsistencies, for example changing DKL-DCPPGPR to PPGPR-DCDKL in 5.2. \n\n> The non-DKL baselines would be illuminating in the DKL experiments. How do the DKL models compare to SVGP?\n\nThe difference between non-DKL and DKL is only in the use of a feature extractor. If you wish to compare DKL to non-DKL performance, note that we run all methods on the same datasets using the same setup, so the values in Table 2 and 3 are already comparable. \n\n> Only non-ARD kernels are considered when considering different length-scales without justification.\n\nWe do consider ARD kernels, and decouple the multi-dimensional length scales in the same way. We mentioned that results with ARD kernels are in the supplement (see l.179-l.181 in the paper and Table 2 in the supplement). Decoupled ARD kernels performs similarly well as decoupled non-ARD kernels. We will draw more attention to this.\n\n> The authors give choice between one of two different whitening matrices, however no justification is given to why both are not used.\n\nWe can only choose to use one of the whitening schemes (see l.81-l.84 and l.146-l.150 in the paper). If one uses both whitening matrices, for example whitening the variational mean $\\mathbb{m}$ with $K_{mm}$ and whitening the variational covariance $\\mathbb{S}$ with $Q_{mm}$, then neither the predictive distribution nor the KL divergence will be simplified.\n\nWe choose to use Q matrices and provide the intuition of our choice. We also provide empirical results to compare the two choices in the supplement (see l.179-l.181 in the paper, and Table 3&4 in section 2.1.5 in the supplement). We will draw more attention to this.\n\n> Since the model separates the posterior mean and covariance, a metric that measures both and their tradeoff would be useful, or at least discussion on how this can be inferred from the reported results.\n\nWe did not separate the posterior mean and covariance directly. We provided a decoupled conditional probability, which still leads to a unified ELBO for a single probabilistic model. We already report both RMSE and NLL, with RMSE measuring only mean performance and NLL measuring probabilistic performance. Reporting these two metrics to address this concern is common practice in the GP literature (e.g. Hensman et al., 2017, Havasi et al., 2018, Salimbeni et al., 2019, Jankowiak et al., 2020).\n\n> Can you report performance in a way that considers the tradeoff between accuracy and uncertainty?\n\nA GP provides a natural tradeoff between accuracy and uncertainty since predictive mean and covariance are analytically computed. We evaluate two metrics - RMSE measuring the predictive mean (accuracy) and NLL measuring both the predictive mean (accuracy) and predictive variance (uncertainty).\n\nIf our response answers you questions, would you kindly update your rating? Please let us know if you have further questions.\n\n\n\n\n### References\n\nHensman, James, Nicolas Durrande, and Arno Solin. \"Variational Fourier Features for Gaussian Processes.\" J. Mach. Learn. Res. 18.1 (2017): 5537-5588.\n\nHavasi, Marton, José Miguel Hernández-Lobato, and Juan José Murillo-Fuentes. \"Deep Gaussian processes with decoupled inducing inputs.\" arXiv preprint arXiv:1801.02939 (2018).\n\nSalimbeni, Hugh, et al. \"Orthogonally decoupled variational Gaussian processes.\" Advances in neural information processing systems 31 (2018).\n\nJankowiak, Martin, Geoff Pleiss, and Jacob Gardner. \"Parametric gaussian process regressors.\" International Conference on Machine Learning. PMLR, 2020."}, {"review_id": "uqAv9DHeTv", "reviewer": "Reviewer_2Aqn", "summary": "This paper develops a variational approximation for learning sparse Gaussian Processes. The central idea is to \"decouple\" the mean and covariance parameters. From my read, this decoupling simply means formulating two different kernel matrices (called $Q$ and $K$ in the paper), and then working those new parameters through the inference and learning mechanisms: to that end, the paper provides appropriately-detailed derivations, including for some empirical concerns (such as whitening). The paper formulates the proposed approach in two cases, evaluating on 10 benchmark regression tasks. Overall, the proposed approach yields improved benchmark tasks. Ablation studies are also performed, which help provide insight into the approach.", "questions": "Q1: Please provide a few paragraphs that would form the basis of an error analysis & extended discussion section. \n\nQ2: Please provide a limitation discussion (or if I missed it in the paper, please point me to it).", "limitations": "No: no obvious limitations section was provided in the main paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Overall, this paper achieves an acceptable balance of mathematical derivation vs. explanation. However, please see \"Clarity\" in weakness for qualifications.\n\nExperiments: Experiments were averaged across 10 runs, and sufficient details of standard error & significance were included. There's an appropriate amount of rigor, and ablation studies run.", "weaknesses": "Clarity: Key summary statements, to act as milestones, throughout would definitely help. This is especially true of Table 1, section 3.2 (Eqn 3), and section 3.3: broadly, the equations are presented, but without a lot of discussion contextualizing the meaning and importance of those equations. \n\nLimited discussion: Given the number of experiments run, the discussion does not have a good pay-off. For example, Table 5 (one of the ablation studies) has fewer than 3 lines of prose dedicated to discussing those results in the main paper. \n\nLack of error analysis: The battery of regression benchmark tasks gives some evidence of broad performance, but there's no error analysis or insight into the numbers and performance. Currently, the results read as \"just numbers\": it's not clear, e.g., what a reduction from 0.321 to 0.156 on \"Pol\" means.\n\nThis is more of a minor point, but not without impact: splitting related work into sections 2.1 and 4 was not effective for me. It made 2.1 very much a slog, and section 4 seem unimportant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback. We reply to each of your suggestions and questions below.\n\n> Clarity: Key summary statements, to act as milestones, throughout would definitely help. This is especially true of Table 1, section 3.2 (Eqn 3), and section 3.3: broadly, the equations are presented, but without a lot of discussion contextualizing the meaning and importance of those equations.\n\nThank you for pointing out areas we could focus on for adding additional clarification. We found space to be fairly tight to include our reasonably extensive experimental evaluation, and plan to use the additional content page to add additional summary and discussions. \n\n> Limited discussion: Given the number of experiments run, the discussion does not have a good pay-off. For example, Table 5 (one of the ablation studies) has fewer than 3 lines of prose dedicated to discussing those results in the main paper.\n\nWe will make sure to add more discussion of experimental results, especially for Table 4 and 5. For example, in Table 5, our SVGP-DCDKL performs the best in both mean performance (RMSE) and probabilistic performance (NLL). This suggests that it is more beneficial to have different models (decoupled feature extractors in SVGP-DCDKL) for conditional mean and covariance rather than only modeling the conditional mean (SVGP-MeanDKL) or having same models (SVGP-DKL). Moreover, it is surprising that SVGP-MeanDKL outperforms baseline SVGP-DKL, which re-emphasizes that a *decoupled* (*different*) or even a much simpler model for the covariance is more beneficial than using the same model for conditional mean and covariance.  For the PPGPR base model, PPGPR-DCDKL also performs the best and we can draw the same conclusion. However, PPGPR-MeanDKL does not outperform the baseline PPGPR-DKL. This suggests that for the PPGPR base model, it is vital to decouple the feature extractor (PPGPR-DCDKL) rather than having a simpler model for the covariance (PPGPR-MeanDKL) since PPGPR treats the predictive variance differently in the learning objective.\n\n> Lack of error analysis: The battery of regression benchmark tasks gives some evidence of broad performance, but there's no error analysis or insight into the numbers and performance. Currently, the results read as \"just numbers\": it's not clear, e.g., what a reduction from 0.321 to 0.156 on \"Pol\" means.\n\nWe use standard regression benchmark datasets that are commonly used in GP regression papers (e.g. Wenger et al., 2022, Jankowiak et al., 2020, Wang et al., 2019), and the labels are standardized to mean 0 and variance 1. We will do our best to better explain the meaning of improvement in numbers. For example, the reduction from 0.321 to 0.156 on \"Pol\" generally indicates a large performance improvement.\n\n> This is more of a minor point, but not without impact: splitting related work into sections 2.1 and 4 was not effective for me. It made 2.1 very much a slog, and section 4 seem unimportant.\n\nWe will do our best to reorganize these related work. Section 2.1 is intended to contain necessary background and prior work needed to understand our method directly. Section 4 is intended to contain broader related work and provide context. Such a section is commonly included and we will do our best to make it more coherent.\n\n> Please provide a few paragraphs that would form the basis of an error analysis & extended discussion section. \nPlease provide a limitation discussion (or if I missed it in the paper, please point me to it).\n\nAs stated above, we will provide more summary statements, more discussion of experiment results, as well as error analysis. We will also add a limitation discussion -- for example, we only applied our decoupling framework to regression tasks and BO applications, and we have not applied it to classification tasks.\n\nThank you for your positive and constructive feedbacks! Please let us know if you have further questions. \n\n\n### References\n\nWenger, Jonathan, et al. \"Preconditioning for scalable Gaussian process hyperparameter optimization.\" International Conference on Machine Learning. PMLR, 2022.\n\nJankowiak, Martin, Geoff Pleiss, and Jacob Gardner. \"Parametric gaussian process regressors.\" International Conference on Machine Learning. PMLR, 2020.\n\nWang, Ke, et al. \"Exact Gaussian processes on a million data points.\" Advances in neural information processing systems 32 (2019)."}, {"review_id": "z1tpgE1rYY", "reviewer": "Reviewer_3gEr", "summary": "This paper studies sparse Gaussian processes which allow different kernel hyperparameters such as length scale to be used, for instance, for the variational posterior mean and covariance. They derive a variational approximation that enables training of models of this kind. This extra scalable approximation flexibility is shown to lead to better predictions on UCI datasets, as well as a few benchmark-style Bayesian optimization problems.\n\nEDIT: As indicated below, I've acknowledged the authors' rebuttal and previously updated my review and score appropriately.", "questions": "Is there a way to view the inducing point approximation you propose as a random function via pathwise conditioning? By this, I mean that, since the true posterior satisfies $(f|y)(.) = f(.) + K_{(.)x} (K_{xx} + \\sigma^2 I)^{-1} (y - f(x) - \\epsilon)$ where $f \\sim GP(0,k)$ and $\\epsilon \\sim N(0,\\sigma^2 I)$, and the approximation of Titsias satisfies $(f|u)(.) = f(.) + K_{(.)z} K_{zz}^{-1} K_{zx} (K_{xz} K_{zz}^{-1} K_{zx} + \\sigma^2 I)^{-1} (y - g(x) - \\epsilon)$ where $g(x) \\sim N(0, K_{xz} K_{zz}^{-1} K_{zx})$ and $\\epsilon \\sim N(0,\\sigma^2 I)$, your posterior approximation might admit a similar formula, and its structure should give insight into the approximation's behavior. Do you know what the respective formula is?\n\nIs your variational optimization problem consistent in the limit of infinitely many test points? By this, I mean, does it lead to a valid Kullback-Leibler divergence minimization problem between stochastic processes, where the true posterior and variational approximation are absolutely continuous so that the KL is well-defined?", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is technically sound, generally well-written, and includes all of the content and evaluation I would expect for this topic. The authors have done a reasonable and comprehensive job here. They evaluate on both modeling tasks (UCI dataset regression) and decision-making tasks (Bayesian optimization), as well as include an ablation study for one of the key non-obvious hyperparameters they introduce (the \\beta regularization term which controls how the test conditional component enters the objective).\n\nMy most common reaction while reading this paper was \"that makes sense\" - which is actually very good, but also means that I don't have too many comments on what to add since the paper was largely clear, and therefore this review will be a bit short. The authors should not be discouraged by this - the shortness in this case should be understood as a sign they wrote the paper well.", "weaknesses": "This paper is studying what I view as a relatively old-school topic, and as a result much of the value here is from good evaluation and execution of ideas, rather than from the ideas themselves. Inducing point approximations have been studied very extensively, and at this stage there are so many variations out there that it isn't clear why we need to write more papers about them, since these methods are mature enough that readers can probably figure out new ones themselves when they are needed for applied work.\n\nThe need to add $\\beta$ regularization parameters is a bit suspicious, since this suggests there might sometimes be a need to downscale the extra term coming from different conditionals because it starts behaving badly in the limit of many test points. It also forces the user to consider even more hyperparameters, and in most cases there are already enough of those out there. As a result, I'm curious for what choices of conditionals the respective infinite-dimensional variational approximation problem between stochastic processes is well-defined. See questions.\n\nThe empirical evaluation is overall good, but it focuses too much on RMSE tables on UCI datasets. These take up a lot of space and are not super informative, so it would be better to have just one and move extra results into the appendix, creating more room for describing the experiments. I would be particularly interested to see more figures which evaluate the technique's produced uncertainty, particularly since inducing points tend to blow up uncertainty and over-smooth the data in situations where there are not enough inducing points, and it would be much more interesting to see how the authors' technique behaves in this situation, rather than more RMSE tables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback. We will address each of your questions below. \n\n> \"Could move some RMSE tables into appendix and describe more about experiments.\"\n\nThanks for the suggestion. We plan to use the additional page of content to add additional description for the experiments. We agree that the RMSE results are generally less important than the NLL results given the probabilistic nature of the model, and are happy to move them to create additional space as well.\n\n> \" ... more figures which evaluate the technique's produced uncertainty, particularly since inducing points tend to blow up uncertainty and over-smooth the data in situations where there are not enough inducing points, and it would be much more interesting to see how the authors' technique behaves in this situation, rather than more RMSE tables.\"\n\nWe agree that uncertainty evaluation is very important. We do at least provide the NLL metric that evaluates both accuracy and uncertainty. For visualization, with synthetic data, we see that when there are not enough inducing points, uncertainty blows up and over-smoothed data (Figure 1a) and our method resolves this issue (Figure 1b). On real datasets, generally our setting does not have enough inducing points due to large number of trainig data, and we provide model calibration results in terms of z-score distribution figures in the supplement (Figure 3 in section 2.1.3 in the supplement). It shows that our method has a better calibrated predictive uncertainty. We will draw more attention to this using the additional page. Please let us know if you would like to see more visualization of the uncertainty evaluation.\n\n> \"View the inducing point approximation you propose as a random function via pathwise conditioning. ... Your posterior approximation might admit a similar formula, and its structure should give insight into the approximation's behavior. Do you know what the respective formula is?\"\n\nYes. We can derive a similar formula following Titsias. For standard variational GP, it requires computing the optimal variational distribution $\\phi^*$ by differentiating the ELBO. Our ELBO for the decoupled model is formed similarly with an additional expected KL divergence term. If we set the regularization parameter $\\beta_2=0$ (removing the additional expected KL divergence term), we could follow Titsias and find a similar optimal variational mean $ \\mathbb{m}^*=\\sigma^{-2} Q_{zz}\\Sigma^{-1} Q_{zx} y$, where $\\Sigma = Q_{zz} K_{zz}^{-1} Q_{zz} + \\sigma^{-2} Q_{zx} Q_{xz}$. With $\\mathbb{m}^*$, we can then derive the predictive mean $ \\mu(\\cdot) = Q_{(\\cdot) z} Q_{zz}^{-1} \\mathbb{m}^* $. \n\nFinally, the random function expression of the decoupled model is: $ (f\\|y)(\\cdot) = f(\\cdot) + Q_{(\\cdot) z} P_{zz}^{-1} Q_{zx} (Q_{xz} P_{zz}^{-1} Q_{zx} + \\sigma^2 I)^{-1} (y - g(x) - \\epsilon)$, where $P_{zz} = Q_{zz} K_{zz}^{-1} Q_{zz}$ and $g(x) \\sim \\mathcal{N}(0, Q_{xz} P_{zz}^{-1} Q_{zx})$. \n\nSo the differences between the standard variational GP formula and ours are (1) our formula mostly involves the Q kernel matrices from decoupled mean and (2) the $K_{zz}$ matrix in the standard formula is replaced by $P_{zz}$ which involves the interaction of $K_{zz}$ and $Q_{zz}$.\n\nHowever, in the more general case where $\\beta_2 \\ne 0$, it is harder to solve for the optimal variational distribution and therefore trickier to find such a formula. With that said, we note (admittedly loosely) that as $\\beta_2 \\to \\infty $, $K_{zz} \\to Q_{zz}$ and therefore $P_{zz} \\to K_{zz}=Q_{zz}$, which would recover the original derivation. Thus the closed form looks like–but may not actually be–a strict generalization.\n\n\n> \"Is your variational optimization problem consistent in the limit of infinitely many test points? By this, I mean, does it lead to a valid KL divergence minimization problem between stochastic processes, where the true posterior and variational approximation are absolutely continuous so that the KL is well-defined?\"\n\nNote that the modification we made to derive our model is not directly made to the SVGP ELBO, but to the GP prior itself. Thus, our derivation results in simply a standard variational inference problem for a slightly different probabilistic model. Our method is therefore best interpreted not as a “modified SVGP” but rather as a “modified GP” to which we are applying variational inference, and there is indeed still a true posterior being approximated by our KL minimization. \n\nThank you for your insightful comments and please let us know if you have further questions."}, {"review_id": "TFJnWC6dXO", "reviewer": "Reviewer_TCkp", "summary": "This paper addresses enhancing the performance of sparse approximate Gaussian processes, which has scalability with the dataset size but causes performance degradation.\nThe authors proposed a novel parametrization of conditionals as decoupled forms for training and testing ones, where lengthscales in kernel representations in their means and covariances can have different values.\nExperimental results on multiple public datasets demonstrate that the proposed method performs better than existing methods.", "questions": "NA", "limitations": "There is a study of performances over different hyperparameters.", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- Experimental results of the proposed method showed promising performance.", "weaknesses": "### Originality:\n- Originality is limited to the proposal of Eq.3.\n\n### Clarity:\n- The important concept of how to select or generate inducing points is not described in the main text. For example, inducing points are different between the left and right in Figure 1, but it is not described why. \n- It is better to provide the definition of SVGP in Introduction.\n- In the equation under l.53, some variables, such as mu, are not defined. The notation for x* is inconsistent\n- In the description of Eq.1, it is better to define Kmm explicitly. Accordingly, the difference between Kmms for the proposed method and the existing methods is not clear.\n- It is better to introduce p(f∗|fm) with more intuition.\n- It is unclear how to determine the regularization parameter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We will address each of your questions below. \n\n> Originality is limited to the proposal of Eq.3.\n\nWe would greatly appreciate it if you can **concretely** point out the limitations on our novelty. We believe that referring to Eq (3) as our full novelty is reductive, as it describes only the modifications we make to the prior and not the resulting considerations when deriving an inference scheme or implementation details like whitening. Arguably, you could be this reductive about other prior published work just as easily, and in our opinion just as wrongly (e.g., by this logic, Salimbeni et al., 2019 “reduces” to equation 12, Jankowiak et al., 2020 “reduces” to equation 18, and so on).\n\nTo be concrete about our novelty from our end, as far as we are aware, ours is not the first work to consider modifications  to the conditionals used in inducing point methods, but *is* the first to consider *more flexible* conditionals. This has significant payoff, reducing error by as much as half on several commonly used benchmark datasets in the GP literature. By being the first to do so, our method is arguably “novel.”\n\n> The important concept of how to select or generate inducing points is not described in the main text. For example, inducing points are different between the left and right in Figure 1, but it is not described why.\n\nIn all results, the inducing points are learned automatically by maximizing the ELBO, which is (1) standard in practice, and (2) the default approach to selecting inducing points in software (e.g., both in GPFlow and in GPyTorch). This allows us to select the “best” inducing points, assuming the numerical optimization of the ELBO is done properly.\n \nIt is natural and expected to have different inducing points in the left and right in Figure 1. The inducing points are learned parameters of the model, and the left and right figures depict different trained models. We will revise the paper to make this more clear.\n\n> It is better to provide the definition of SVGP in Introduction.\n\nSVGP, stochastic variational Gaussian process, is a widely used method in the literature and we will add the definition of SVGP in the introduction to make it more clear.\n\n> In the equation under l.53, some variables, such as mu, are not defined. The notation for x* is inconsistent.\n\nWe will carefully go through the paper for better consistency. In the equation under l.53, $\\mu(\\cdot)$ is the prior mean function. We will write the variable $\\mathbf{x}^*$ in l.53 in bold for consistency. Thanks for spotting the typo.\n\n> In the description of Eq.1, it is better to define Kmm explicitly. Accordingly, the difference between Kmms for the proposed method and the existing methods is not clear.\n\n$K_{mm} \\in \\mathbb{R}^{m\\times m}$ is the kernel matrix formed by kernel function evaluated at inducing points: $K_{mm}(i, j) = k(u_i, u_j)$ where $u_i, u_j$ are inducing points, which is a standard expression in GP literature and we will make it more clear in the text. \n\nThe difference between Kmms for our method and the existing methods lies in the decoupling of the kernel hyperparameters and we provide explicit examples in section 3.2 (example 1 and 2).\n\n> It is better to introduce $p(f^∗|f_m)$ with more intuition.\n\n$p(f^* \\mid f_m)$ is the distribution of testing labels $f^*$ conditioned on the inducing values $f_m$ induced directly by having a GP prior. It would be the same distribution as conditioning a GP on the inducing points and the noise-free labels $f_m$ and making predictions at the test points.\n\n> It is unclear how to determine the regularization parameter.\n\nWe thoroughly evaluate the effect of $\\beta_2$ in Figure 2 and section 5.2 on two of our test datasets, with additional similar figures on most datasets in the supplementary materials (see l.207-l.209 in the paper and Figure 5&6 and Table 7&8 in the supplement). In practice, the lessons learned in this study seem to be general, where our method prefers lower values of $\\beta_2$ but is insensitive otherwise. To set the regularization parameter even more precisely on a real world application of our method, consider doing cross validation.\n\nPlease let us know if you have further questions and we would appreciate it if you would raise your score given our response.\n\n### References \n\nSalimbeni, Hugh, et al. \"Orthogonally decoupled variational Gaussian processes.\" Advances in neural information processing systems 31 (2018).\n\nJankowiak, Martin, Geoff Pleiss, and Jacob Gardner. \"Parametric gaussian process regressors.\" International Conference on Machine Learning. PMLR, 2020.\n"}]

Author rebuttal:

Meta-review: The paper proposes a variational approximation for sparse GP by using the decoupled kernel parameters for posterior mean and covariance.  Improved performance is demonstrated on 10 benchmark regression tasks with ablation and on a Bayesian optimization task.

Reviewers acknowledged the technically sound contributions that bring good performance compared to baselines, while they raised major concerns: 

- Only non-ARD kernels are considered as baselines. (actually some in supplementary)
- No justification for some design choices
- Clarity issues
- lack of error analysis
- Limited originality

The authors addressed all major concerns, and all reviewers (except an unengaged one) agree to accept the paper.
Decision: Accept (poster)