Localizing and Amortizing: Efficient Inference for Gaussian Processes
Authors: Linfeng Liu
Liping Liu
Venue:  (2020)
Keywords: Gaussian Processes
Variational Inference
Amortized Inference
Nearest Neighbors

Abstract: The inference of Gaussian Processes concerns the distribution of the underlying function given observed data points. GP inference based on local ranges of data points is able to capture fine-scale correlations and allow fine-grained decomposition of the computation. Following this direction, we propose a new inference model that considers the correlations and observations of the K nearest neighbors for the inference at a data point. Compared with previous works, we also eliminate the data ordering prerequisite to simplify the inference process. Additionally, the inference task is decomposed to small subtasks with several technique innovations, making our model well suits the stochastic optimization. Since the decomposed small subtasks have the same structure, we further speed up the inference procedure with amortized inference. Our model runs efficiently and achieves good performances on several benchmark tasks.

Under review as a conference paper at ICLR 2020
LOCALIZING AND AMORTIZING: EFFICIENT INFERENCE FOR GAUSSIAN PROCESSES
Anonymous authors Paper under double-blind review
ABSTRACT
The inference of Gaussian Processes concerns the distribution of the underlying function given observed data points. GP inference based on local ranges of data points is able to capture fine-scale correlations and allow fine-grained decompo- sition of the computation. Following this direction, we propose a new inference model that considers the correlations and observations of the K nearest neigh- bors for the inference at a data point. Compared with previous works, we also eliminate the data ordering prerequisite simplify the inference process. Addi- tionally, the inference task is decomposed to small subtasks with several technique innovations, making model well suits the stochastic optimization. Since the decomposed small subtasks have the same structure, we further speed up the infer- ence procedure with amortized inference. Our model runs efficiently and achieves good performances on several benchmark tasks.
INTRODUCTION
Gaussian processes (GP) (Rasmussen & Williams 2006) are flexible non-parametric models with wide range of applications. GP poses Gaussian prior function values f and assumes ob- servations are generated independently given f. GP inference considers the calculation of the posterior of these function values (Matthews al. 2016) given observations, namely p(fly). Direct computation of the posterior is often intractable on large datasets, motivating people to consider its approximations. Variational inference (Jordan et 1999; Blei al., (017) for GP (Rasmussen & Williams 2006) has achieved great successes recently. Variational inference constructs variational distribution, which usually a multivariate Gaussian distribution, to approximate the posterior. The approximation is done by minimizing the KI. divergence from the posterior to the variational dis- tribution Blei et al. 2017). The variational distribution is often constructed with some special structures to reduce the number of variational parameters and speed up the computation.
Titsias 2009 Hensman et al, Inducing- point methods (Quinoner-Candella Rasmussen] 2005
2013 2015) define variational distributions on small number M of inducing points and then derive the distribution of non-inducing points conditioned on these inducing points. Inducing points sum- marize the entire posterior distribution, and their number M balances the computational cost and the quality of the approximation. Inducing-poit methods are further improved in several directions, such as generic inference for non-Gaussian likelihoods (Sheth al. 2015 Dezfouli & Bonilla [0055] Krauth et al. 20166] Hensman et al.) 2015). inter-domain and subspace inducing points (Hensman al. 2017 Panos et al. [2018), and decoupled approximation with two different sets of inducing points [Cheng & Boots| Salimben et al.) 2018). Burt et al. (2019) provide theoretical analysis to show that relatively small M is sufficient to produce reliable variational approximation when the input dimension is low.
While inducing-poin methods capture global correlations among data points through inducing points, inference methods based on local neighbors focus more on correlation structures at local scales. These methods consider only local-range dependencies to save computation because local- range correlations are often much stronger than distant ones. Nguyen-Tuong
al. (2009); Park & Apley (2018) partition the input space into subregions, fit local models over subregions and then stitch local models into one. Other works examine neighbors of each data point directly. Gramacy & Apley (2013) investigate the properties of GP predictive equation and construct a local predictive approximator. Covariance tapering (Furrer et al. 2006 Kaufman et al. 2008) gains computational

---

Under review as a conference paper at ICLR 2020
efficiency by constructing sparse correlation matrix with zero correlations between distant data
points. Methods based on Vecchia's 's approximation (Vecchia 1988 Datta et al., 2016; Liu & Liu 2019 Finley et al. 2019) decompose the joint probability of data points into conditionals according to a data ordering and then neglect far data points that are conditioned on.
Recently, Liu & Liu (2019) propose the AIGP method, which extends the idea of local inference to GP models with non- Gaussian likelihoods. They use directed graphical models to approximate both the prior and the posterior. With this construction, the inference task decomposes into local inference subtasks, then they introduce amortized inference and use inference networks to identify solutions to these subtasks (Kingma & Welling, 2013] Dai et al. 2013 Miao et al. 2016). Amortization reduces the number of optimization parameters and greatly speeds up the inference procedure. However, this method has two drawbacks. First, the inference at a data point considers a few of its nearest neighbors but not all of them; therefore, it may lose some important correlations. Second, it depends bad ordering often deteriorates the performance, but it is hard to guard against
on a data ordering. such a bad situation. There are no easy fixes of the two issues, because all these designs in AIGP serve the purpose of decomposition.
In this work, we propose new GP inference method, Localized and Amortized Inference based on Nearest neighbors (LAIN) LAIN considers K nearest neighbors for the inference at each data point. Particularly, LAIN uses variational distribution whose covariance is parameterized by sparse decomposition. The decomposition focuses on the correlations between every data point and its K nearest neighbors LAIN also eliminates the need for data ordering. These nice properties come after several technical innovations. First, the new distribution does not admit decomposable entropy calculation. We overcome this difficulty by using decomposable lower bound of the en- tropy (Ranganath et al.] [2016] Louizos & Welling 2017). Second, to decompose the logarithm of the prior, [AIGP] and previous methods use a directed graphical model as an approximation of the prior. We follow this idea, but we consider all possible orderings of data points and collapse them to local combinations, making the computation manageable. With these techniques, [LAIN] still de- composes the inference task into subtasks, so amortized inference can apply. It is worthing noting that subtasks in LAIN are generated from the same mechanism while those in [AIGP] are not. We argue that subtasks sharing the same "distribution" are more appropriate for amortization.
Our empirical evaluations show that the LAIN method outperforms baseline methods including AIGP in several learning tasks. Our investigation also indicates that LAIN can achieve decent per- formance even only a few neighbors are considered.
2 BACKGROUND
Gaussian Processes. Suppose we have a dataset containing a feature matrix X = (x and observations y = (2)) We assume there is a latent function f that generates y; from x; for each i. Particularly, each yt is generated by likelihood model p(yalf)) with f, = fxx).. Denote f = () then p(y[f) f II ff.) The likelihood p(vsJft) can be very general here we only assume that log pggp fA) is differentiable with respect to ft. This mild assumption allows a wide range of data distributions. For example, y. is binary, f(y)lf)) is a Bernoulli distribution with fa
as the logit.
We put GP prior with a mean function v(-) and a kernel function x(..)) over the latent function f. The kernel function encodes the prior knowledge of the smoothness of f. One commonly used kernel function is the Radial Basis Function (RBF) kernel, n(x, x,) = 72 exp( 00.5 - x, -x,10 11//a)) with and parameters. With this prior, function values in f follow multivariate Gaussian, f ~ ((, Σ), with the mean v = (v(x.)) and the covariance matrix Σ with Σi, = xx(, x,) Vi
GP inference concerns the calculation of the posterior yf (Matthews et al. 2016). from which can infer the function value f. for any new input x, with integral J, P(J.(f)) (fl)) The posterior p(fly) is generally not tractable, so we appeal to approximate inference.
Variational Inference for GP. Variational inference approximates the posterior pffly) with a vari- ational distribution (f), which is defined as a multivariate Gaussian distribution, (f)) ~ N(u, V).
We use the term "nearest neighbors" for easy reference, but actually consider K most correlated neigh- hors in the prior. When the kernel is hased on some distance metrics, then they are K nearest neighbors.
2

---

Under review as a conference paper at ICLR 2020
GCNN
f2
Ms
63
Aup
A-B
fa
prior kernel
observations
white noise
variational mean
variational samples
Figure The structure of variational distribution. The left hox shows the amortization, which fits ja-8 and Rag-s from their related prior kernel and observations. The right part shows the generation process of fi-s.
The inference is carried out by maximizing the Evidence Lower BOund (ELBO) with respect to ((f) Blei et al. 2017).
log p X) 2 max Byg [log(yyf)) [(gg(()) Rgg [log (())
(1)
g(f)
Leut
Lerous
Leni
Here we name the three terms in the ELBO for easy reference later. Typically the ELBO is maxi- mized by gradient-based optimization, preferably stochastic gradient optimization when N is large. Direct optimization of the ELBO is challenging, since the kernel matrix E and the variational co- variance V are both large and have size N x N.
Inducing- point methods define ((f) Jr, (pf())))()) df,, where g(fr) is the distribution over inducing points I, and p( derived from the prior. The computation is reduced mainly because only the small distribution (fy)) is optimized, while the conditional p(flf,) is fixed when the prior
is given.
AIGP parameterizes V by Cholesky decomposition, V LL Here L is sparse lower trian- gular matrix, and each row of has at most non-zero entries. AIGP uses a triangular L for easy entropy computation. It also approximates log with directed graphical model. Both the lower triangular matrix L and the directed graph require an ordering of data points.
3
METHOD
3.1 THE VARIATIONAL DISTRIBUTION
Following previous works, we also define variational distribution g(f) to be a multivariate Gaus- dianN(re V). We parameterize V R.R. + 8 with R being sparse matrix and 8 being a small constant. Note that we do not require R to be triangular. The sparse pattern of R decided by the nearest neighbors: Raj * only when j n(i). Here n(i) is the neighbor set containing data points that have the largest covariance with in the prior (by definition n()) includes i). this work, we fix the size of ni) to be K, though our derivation works for varied sizes of nt).. The row R; can be viewed as a representation of fi in the variational distribution: R; informs ft's correlation with other function values, just like word embedding informs its relation with other words (Mikolov et al. 2013).
Efficient sampling from the marginal is critical for the decomposition the ELBO later. Owing to the sparse decomposition of the covariance matrix, we can cheaply draw marginal samples for an fi from qff) with linear transformation of white noise. The sampling scheme shown in (2) and pictured in the right part of Figure
f. = + R, + 6c μ4s + Ri,(()) Ex(() + 8E, e~ N(O.), & ~~ 1).
(2)
The constructed distribution (f)) well approximates the strong correlations in the prior. From fi and f, correlate in (f) ) by sharing noise entries in n(i) (n( when the intersection is not empty. In this case, either fo neighbors fz. f, neighbors fi, or fe, share common neighbors. When the neighbor sets are large enough, most strong correlations will be approximated by some non-zero entries in V.
(
3

---

Under review as a conference paper at ICLR 2020
3.2 OPTIMIZATION OF THE ELBO
We optimize the ELBO in (I) to find good q(f) to approximate the GP posterior. To apply stochas- tic optimization, will decompose the three terms in the ELBO. We mainly consider the decom- position of Leross and Lene, as the decomposition of Lelt is easy.
We first decompose the cross entropy Leross. By convention, the GP prior has a zero mean. Though there is a flosed-form calculation of Loross with both q(f) and p(f) being multivariate Gaussian, it involves expensive calculations of det and Σ-1 Previous works approximate the prior with Vecchia's s method for easy decomposition and good approximation (Vecchia 1988 Stein et al. 2004 Datta al., 2016; Liu & Liu
2019 Finley et al., 2019). The idea to build directed graphical model and approximate p(f) II fa(s) with a(i) being a small parent set of i. In the original work. Vecchia 1988) first set an order to data points and then choose a(i)) as the K nearest parents of i. But it not easy to guarantee good ordering of data points (Banerjee et al. 2014] Guinness 2018). Here we consider all possible orderings and take the average of approximations to address the data ordering concern.
We estimate Leross as follows. First, we randomly sample a parent set ri(i) C. r((i) with i & n'() (2) for each i. Then we approximate the log-prior by log)(f) = Σn log p(filfffa))) with the conditional distribution p(filnrtt)) (i((t)) derived from the joint Gaussian p(fi, (1) the prior. Then Leroea is estimated by random batch of terms. The complete calculation is given as
N
Leross Re Leross
Σ
Ellhnfar)) [logp(filfw(o)) random set n (t) C cn)...
(3)
SS|
iES
Here S a random batch of data points.
Now we justify that this is an average over all data orderings. Suppose there is data order (-),, such that we can define a directed graphical model over p(f) by assigning every i parent set n', (i) (j j E n((), (i) < {i)]]. Denote IT as all permutations of N data points, with each permutation inducing a graphical model. The average the log densities all graphical models can be collapsed to the average computed from local neighborhoods. Denote lln(s) permutations of indices in the set n(i). then we have
Bottriing (n) (ogg(((lfwg.))
=
Σ K! 1=1
Σ
Bultingg (n)
1ogP(Ailf2(1))) (4)
N! RETI 1=11
xEllaC))
Here we only need to consider permutations of data points within n2i) for each i. Then we obtain by estimating the inner summation by single random permutation of ni)) and the outer summation by a random batch S.
We then decompose the entropy Tent- The entropy of q(f) requires the expensive computation of det(V), To circumvent this difficulty, we find a decomposable lower bound of the entropy by using an auxiliary distribution (Ranganath al. 2016 Louizos & Welling (0177). Note that we always prefer a lower bound of the objective in this maximization problem. With an arbitrary distribution r(e(f), lower bound of Lent
Lent = EE [log (ff) 2 EEy(F,.) [log q(fle) + log q(E) log (ogr()
(5)
The bound is tight when r<e(f) matches q(e(f). In this work, we try to let r(c(f) match g(((f).
Particularly, we set r(c(f) = II, (())(fn())) where the conditional q(c, (f.()) is derived from the joint Gaussian distribution likelihoods and are decomposable over data points. We can then reach the estimation of the entropy lower bound with batch of data points.
(n()). Then all terms in the lower bound in (5) are Gaussian log-
Lent 2 Lent
N
log (1 (I-R, (Rno,.BTaaa)
Baas)ss) + const.
(6)
2
S
ÈES
We finally decompose the likelihood Leu- The likelihood term l(y((y naturally decomposes because yi-S are conditionally independent given se-s.
N
Lell
Bacn log p(ll.f)). Lall
log.(vilfi)
|S|
ie
4

---

Under review as a conference paper at ICLR 2020
Here for each term in the summation, the expectation is estimated by Monte Carlo sample f; from (fi)). The gradients of variational parameters propagated through fa via reparameterizattion Kingma& & Welling 2013).
Finally, the ELBO has a decomposable approximation Lell + Leross + Lent to enable efficient stochastic optimization. From the derivations above, we the objective can be decomposed by data points. The computation for a data point only involves itself and its nearest neighbors. Therefore, each stochastic gradient calculation takes time only ((K3). There are N(K + 1) parameters in ja and R to optimize, so the optimization takes at least O(N) time. We further reduce the number of parameters by amortizing the cost through a shared inference model, taking advantage of the fact that the inference for each data point i only needs its K nearest neighbors.
3.3 AMORTIZED INFERENCE
Following AIGP, we also apply amortized inference to GP inference. Particularly, we train an infer- ence network to identify variational parameters (1, and (i)) for each data point i. Since node correlations at a neighborhood can be easily treated as weighted graph, we use Graph Convolu- tional Networks (GCN)) (Kipf & Welling, 2017) as our inference network.
A GCN takes an adjacency matrix A E R of graph and the node features H E Rxxo as the input and then makes predictions for all graph nodes. Let A be the normalized adjacency matrix, A D- AD- -2, with D being the diagonal degree matrix. A GCN layer & with the input H(-1)) defined by (e) gr (H A) := (H((-))((()) Here w()) -Rd is the weight matrix of the layer f. (.) is the activation function. An L-layer GCN computes its output by H ( A) == 9r (-.-999 (Ho, A). We use twoGCN) for the inference task, genn for the calculation of μ4; and genz for R.,()))
#4 = a gen ([yn(1)) e.), En())())) Ri,n()) = genz (Iyn(i), e.], En(i))(t))
Here we use (() as the adjacency matrix and stack the observation Yn(i) and the one-hot vector e, the input feature. The vector e, indicates the element i for which the inference is running for. We choose the activation (-) to be ReLU for intermediate layers and identity for the last layer. The last layer of each GCN has size 1 to output K vector. is an averaging vector with all K elements as K. The dashed box in Figure shows the amortization.
LAIN defines an inference subtask a data point and its nearest neighbors, while AIGP defines a subtask on a data point and its parents. Due to this difference, LAIN has two advantages. First, the inference network of LAIN uses the observations from all the K nearest neighbors, while the inference network of AIGP uses observations from parents only but not children. Second, inference subtasks of LAIN are generated with the same mechanism because the nearest-neighhor relationship is homogeneous across all data points. However, the parent-child relationship in AIGP depends on the ordering of data points (e.g. the first one in the order does not have parents). As a learning model, the inference network prefers subtasks from the same "distribution".
The computational cost of GCN is O(?2) by treating the network size as constant. The complex- ity of one gradient calculation is O(K3). The optimization procedure converges fast since it only optimizes a constant number of variational parameters. In practice, we often observe that the op- timization procedure converges in less than epoch, which is not possible for methods without amortization. Finding nearest neighbors is the only step with running time bounds to the data size, but it only needs one and is often fast medium to large data sizes. If the data has a very large size, we can use k-d trees for low-dimensional data and approximate algorithms (Arya el al. Datar al. 2004) for high dimensional data.
1998
3.4 PREDICTION
For a new data point x, with its K nearest neighbors n(*) in the prior, the predictive distribution is
plu-1x,. X,y) ~ (olll
Rs
1
Σ
p(u-lf-)-
(9)
f.er
5

---

Under review as a conference paper at ICLR 2020
The first two plots compare predictive distributions of full GP and LAIN with K = 10. The right three plots show how SVGP AIGP and LAIN perform with a very small number of inducing points/neighbors. Data points blue circles are not well fitted.
Figure 2:
Here ((s X n (n(()) = Senc. p(f- P(LIla())))(((()) (d) is Gaussian with parameters,
μ4 = bx(l(()))
0f = - [.,(,)) b." + b, (Rn(.)) (()) )b
(10)
with b. = (.) n(+))((+)) F is a set of Monte Carlo samples from g( X Yn(s)) Monte Carlo estimation is accurate since the integral is one-dimensional.
The
EXPERIMENT
We compare our method with five state-of-the-art methods: SVGP Hensman al. 2015), SAVIGP
(Dezfouli & Bonilla 2015). DGP (Cheng & Boots 2017). VFF Hensman et al. (017), 7), and AIGP (Liu & Liu 2019p. The first three methods are based on inducing points, VFF uses domain inducing points, and AIGP uses local neighbors. Through all experiments,
use RBF as the
default kernel, except for VFF we use Mafern- kernel (the code does not provide RBF kernel). We the implementation of SVGGPPfron from GPFlow (Matthews et al., (017), the implementation of DGP 2018), and implementations of all other algorithms from their authors.
from Faust
vary the number of inducing points, M {200, 1000, 2000). to check their performances. DGP has separate inducing points for mean approximation and those for variance approximation. We use 256 inducing points for variance approximation and vary the number of inducing points for mean approximation from 200 to 2000. We vary the number of neighbors, K {10, 20, 40), for AIGP and LAIN GCN used in these two methods have three hidden layers with dimensions [20, IO, TJ. We randomly split each dataset into training (75%) and testing (25%) and report both the predictive performance on the test set and the inference running time. To save the space, we report results from two settings for each competing method: one setting M = 200 or = 10, with which all methods have their fastest speed (marked by f), and another setting giving the best predictive performance (marked by v).
For SVGP SAVIGP and VFF
4.1 TOY EXAMPLE
In this section, test different methods one-dimensional toy example studied in (Snelson & Ghahramani 2006). The dataset contains 200 data points, shown as black dots in figure 2 We assume Gaussian likelihood in this experiment and run exact inference as the baseline. A smaller GCN (hidden dimensions [10, 5, 1]) is used in this task.
The predictive mean and variance from the exact inference and LAIN with K 10 are shown in the first two plots of Figure 2 The result LAIN]ss is very similar to that of exact inference, except that the mean curve of LAIN less smooth, which does not really hurt the predictive performance.
We test different methods with very small M and K and observe how they behave. We are likely to face this situation when we work on large datasets in high-dimensional spaces. The last three plots of Figure
predictive distributions of SVGG with M = 2 inducing points, ATGP with K = parents, and LAIN with K 2 nearest neighbors. When there are not enough inducing points, SVGP over-smooths the prediction and performs poorly for good fraction of data points. AIGP does not have a good predictive mean either, because under a random ordering the directed graph constructed by canno well capture neighboring relations. The predictive mean of LAIN does not deviate far from the ground-trui in the area with training instances, though the curve is rugged due to local variations.

---

Under review as a conference paper at ICLR 2020
Table 1: Comparison the eBird dataset.
Table 2: Comparison the precipitation dataset.
Method
Contgg
Pred NLL
Time
Method
Config
Pred NLL
Time
M=200
1.90
107s
M=200
1.572.03
2.5ks
SVGP
SVGP
M=1000
1.88+
4.5ks
M=2000 <
1.28+.03
42ks
SAVIGP
M=200 $
2.04
167s
SAVIGP
M=200 4
1.70+.02
2.8ks
M=2000 /
1.99+.03
50ks
M=2000 <
+.58+.02
50ks
M=200
1.91 +.22
1.3ks
M=200 4
1.54+.03
9.1ks
VFF
VFF
M=2000
1.91+.02
13ks
M=2000 <
1,53+.03
32ks
M=200
1.82+.02
96s
M=200
1.07+.05
402s
DGP
DGP
M=2000 V
1.80+.02
21338
M=2000 V
1.00+.05
889s
K=10
1.79+.05
458
K=10
0.96+.03
155s
AIGP
AIGP
K=20
1.71
125s
K=10
0,96.1033
155s
K=10
1.69+.03
55s
K=10
0.74+.05
129s
LAIN
K-20
1.651.033
384s
LAIN
K=20
0.72.1.05
903s
K=40
1.60.1.02
1.3ks
K=40
0.69.1044
2.3ks
Table 3: Comparison on the MNIST dataset.
164
Method
Config
Pred NLL
Accuracy
Time
M=200
0.053:..00 0004
98.4
623s
LAIN
SVGP
LAIN wio inf. nec
M=1000
0.051#.004
98.5
23ks
Negative EL.BO
SAVIGP
M=200
0.339+ .008
51.7
6.5ks
M-200 V
0.33911008
51.7
6.5ks
M=200 $
0.0591.005
98.1
292s
DGP
M=2000 V
0.052+.005
98.3
2.1ss
K=10
0.293110002
98.0
3.9ks
AIGP
K=40
0,215++0033
98.2
24ks
K=10
0.053+1.003
98.9
128s
LAIN
K=20
0.050:1.033
99.0
632s
2 Number eppoch
3 45
10
20
50 100
K-40
+,51
99.1
2.9ks
K=9
N.A.
98.6
24s
Figure 3: ELBO trajectories of LAIN with and without inference networks.
KNN
K=19
N.A.
98.3
26s
K=39
N.A.
97.6
28s
4.2 BIRD ABUNDANCE ESTIMATION
In this experiment, we estimate the spatial abundance of a bird species (Savannah Sparrow) using eBird dataset (Munson et al.) 2015). The dataset has 14,393 observations, each of which is reported bird count at a GPS location. We model the observed counts with GPS locations as the input. We set the likelihood to be Poisson distribution, with rate given by X; exp(fx).
We compare different inference methods in terms of Negative predictive Log- Likeeliidd (NLL, the smaller the better predictive performance). Table I shows the results. We see that LAIN achieves the best predictive performance at K = 40. Methods based on inducing points generally perform worse. In this dataset, observations have strong correlations in local areas, but inducing points are not efficient to capture the posterior at such a fine scale. In terms of running speed, LAIN comparable to AIGP and DGP but faster than other methods. In our experiment, we have also tried to increase inducing points for DGP but it does not improve its performance.
In this experiment, we also investigate whether inference networks work correctly. We run LAIN without inference networks and optimize fe and L for the variational distribution directly. Then we compare [AIN models with and without inference networks by checking their optimization procedure. In this task, we fix hyperparameters, so the two methods solve a pure inference problem. Figure 3 is the trace plot of the negative ELBO versus training epochs. The figure shows the ELBO of the two LAIN models eventually converge to very similar values, though the ELBO without inference networks is slightly better after 50 epochs (likely due to the amortization gap). LAIN with inference networks significantly reduces the number of optimization epochs - the inference networks are well trained after only 0.01 epoch (about 100 iterations). In summary, the result indicates that inference networks can effectively identify the variational parameters using local information.
7

---

Under review as a conference paper at ICLR 2020
4.3 PRECIPITATION LEVEL ESTIMATION
In this task, we evaluate LAIN on rainfall dataset. We process the precipitation dataset (Climate Data Online) and obtain the average precipitation level in May at 8,832 stations that are spatially distributed in the US. The GP inputs are GPS locations of these stations, and the observations are the average precipitation levels. We use the log-normal distribution as the likelihood, with its mean as function value f from GP and variance hyperparameter learned from the data.
SableeSsmmartt the experimental results. LAIN has better predictive performance, and its running speed comparable to or faster than other methods.
We also analyze the goodness of our prior approximation since we can compute the exact Leross on this dataset. We compute Leross with the optimized g(f) distribution as well as Loross. The true value Leross and the approximation Leross are: 5,465 versus 5,396 when K = 10. 9,905 versus 8,490 when K 20, and 10,086 versus 9,009 when K = 40. This result indicates that the approximation Tecross is relatively accurate. Furthermore, Leross tends to be smaller than the true value and can be considered lower bound in such cases.
4.4 HAND-WRITTEN DIGIT CLASSIFICATION
In this experiment, we explore a high-dimensional inference problem, GP classification of MNIST digits (LeCun & Cortes 2010). We consider a binary classification handwritten images of and 8. To make performance values of different methods more differentiable, we randomly choose a subset of size 7,858 from the original dataset. Pixel values normalized [0,1] in the preprocessing step. In the results, we also report the accuracy obtained by different methods.
The results are shown in Table 3 accuracy. Its predictive NLL and running speed also overperform competing methods, though not very significant. We also observe that /AGGP makes less confident predictions than other methods, which accounts for its worse predictive NLL but high accuracy. We do not report results from VFF due to memory issues.
We see that LAIN performs the best in terms of classification
We also examine KNN in this experiment. From the results, we notice that small number of neighbors are often sufficient for KNN time of KNN] we also see that the time of finding nearest neighbors is only small fraction of the total inference time on this dataset. There are slight differences regarding the test accuracy between KNN and LAIN presumably due to different weighting schemes: LAIN weights different nearest neighbors according to their correlations, while KNN treats all nearest neighbors uniformly.
models to perform well. By checking the running
CONCLUSION
In this work, we propose a novel approach for GP inference. We construct variational distribution that has sparse decomposition on its covariance matrix. With this distribution, function value at a data point inferred from nearest neighbors, encouraging the inference efficiently focuses on approximating strong correlations posed by the prior. The proposed variational distribution is ex- pressive to approximate the GP posterior and also provides a decent structure for efficient ELBO optimization. We further decompose the ELBO into homogeneous subtasks and therefore enable stochastic optimization. Finally, we devise inference networks to perform these subtasks and signif- icantly reduce the number of variational parameters. Our proposed method performs well in terms of predictive performance and running speed on a series of benchmark tasks.
REFERENCES
Sunil Arya, David M Mount, Nathan Netanyahu, Ruth Silverman, and Angela Wu. An optimal algorithm for approximate nearest neighbor searching fixed dimensions. Journal of the ACM (ACCM),5(6)))) 923, 1998.
Sudipto Banerjee, Bradley P Carlin, and Alan E Gelfand. Hierarchical modeling and analysis for spatial data. Chapman and Hall/CRC, 2014.
8

---

Under review as a conference paper at ICLR 2020
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisti- cians. Journal of the American Statistical Association, 112(518):859-8777 877, 2017.
David Burt, Carl Edward Rasmussen, and Mark Van Der Wilk. Rates convergence for sparse vari- ational Gaussian process regression. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceed- ings of Machine Learning Research, pp. 662-8711 Long Beach, Califormia, USA, 09-15 Jun 2019. PMLR.
Ching-An Cheng and Byron Boots. Variational inference for gaussian process models with linear complexity. In Advances in Neural Information Proceaatiiinnoooooiii Systems, pp. 1190-5200, 2017.
Climate Data Online. http:s//www.nndd..naa....oo/
Zhenwen Dai, Andreas Damianou, Javier González, and Neil Lawrence. Variational auto-encoded deep gaussian processes. arXiv preprint arXiv: 11X1::5511.00555.. 2015.
Mayur Datar, Nicole Immorlica, Piotr Indyk, and Vahab Mirrokni. Locality- hashing scheme based p-stable distributions. In Proceedings of the twentieth annual symposiun Computational geometry, pp. 253-262. ACM, 2004.
Abhirup Datta, Sudipto Banerjee, Andrew O Finley, and Alan Gelfand. Hierarchical nearest- neighbor gaussian process models for large geostatistical datasets. Journal of the American Sta- tistical Association, 111 (1(514))800-81,, 2016.
Amir Dezfouli and Edwin V Bonilla. Scalable inference for gaussian process models with black- box likelihoods. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 1414-1422. Curran Associates, Inc., 2015.
Aleksandra Faust. Decoupled gaussian process (github implementation).
ccomggoo) e/decoupleecccee weeeeeepoeeeeeeeeeeeeeeeeeeeeeee leeeeddussiaa___eeeeeeee propeessspprcssssse_ee____ 2018.
Andrew O Finley, Abhirup Datta, Bruce Cook, Douglas C Morton, Hans E Andersen, and Sudipto Banerjee. Efficient algorithms for bayesian nearest neighbor gaussian processes. Journal of Computational and Graphical Statistics, pp. 1-14, 2019.
Reinhard Furrer, Mare G Genton, and Douglas Nychka. Covariance tapering for interpolation of large spatial datasets. Journal of Computational and Graphical Statistics, 15(3)5002-523 2006.
Robert B Gramacy and Daniel W Apley. Local gaussian process approximation for large computer experiments. Journal of Computational and Graphical Statistics, 24(2):561-578, 2015.
Joseph Guinness. Permutation and grouping methods for sharpening gaussian process approxima- tions. Technometrics, 60(4):415-429) 2018.
James Hensman, Nicolo Fusi, and Neil D Lawrence. Gaussian processes for big data. arXiv preprint arXiv: 1309.685, 2013.
James Hensman, Alexander G de G Matthews, and Zoubin Ghahramani. Scalable variational Gaus- sian process classification. In Proceedings of the Fighteenth International Conference on Artificial Intelligence and Statistics, 2015.
James Hensman, Nicolas Durrande, and Amo Solin. Variational fourier features for gaussian pro- cesses. The Journal of Machine Learning Research, 18(1):5537-5588 2017.
Michael I. Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. An introduction to variational methods for graphical models. Machine Learning, 37(2):183-233, 1999.
Cari G Kaufman, Mark Schervish, and Douglas Nychka. Covariance tapering for likelihood- based estimation in large spatial data sets. Journal of the American Statistical Association, 103 (484):1545-1555 2008.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference Learning Representations (ICLR), number 2014, 2013.

---

Under review as a conference paper at ICLR 2020
Thomas N. Kipf and Max Welling. Semi- classification with graph convolutional net- works. In International Conference on Learning Representations (ICLR). 2017.
K. Krauth, E. V. Bonilla, K. Cutajar, and M. Filippone. AutoGP: Exploring the Capabilities and Limitations of Gaussian Process Models. ArXiv e-prints, October 2016.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL URLhttp://yann:
lecun com/ wwtm/exxb///eeee /
Linfeng Liu and Liping Liu, Amortized variational inference with graph convolutional networks for gaussian processes. In The 22nd International Conference on Artificial Intelligncce and Statistics, pp. 2291-2300, 2019.
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational bayesian neural networks. In Proceedings of the 34th International Conference on Machine Learning-Vons Volume 70, pp. 2218-2227. JMLR. org, 2017.
Alexander G de G Matthews, James Hensman, Richard Turner, and Zoubin Ghahramani. On sparse variational methods and the kullback-leible divergence between stochastic processes. In Artificial Intelligence and Statisties, pp. 231-221122 231-239, 2016.
Alexander G. de G. Matthews, Mark van der Wilk, Tom Nickson, Keisuke. Fujii, Alexis Bouk- ouvalas, Pablo León-Villagrá, Zoubin Ghahramani, and James Hensman. GPflow: A Gaussian process library using TensorFlow. Journal of Machine Learning Research, 18(40):1-6, apr 2017.
URL httlhhtt://// /mrr orq. www.oo/aaws////////ee/
.tml
Yishu Miao, Lei Yu, and Phil Blunsom. Neural variational inference for text processing. In Proceed- ings of the 33rd International Conference International Conference Machine Learning, pp- 1727-1736, 2016.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations words and phrases and their compositionality. In Advances neural information pro- cessing systems, pp. 3111-3119, 2013.
M. A. Munson, K. Webb, Sheldon, D. Fink, W. M. Hochachka, M. Iliff, M. Riedewald, D. Sorokina, B. Sullivan, C. Wood, and S. Kelling. The eBird reference dataset, version 2014, 2015.
Duy Nguyen-Tuong, Jan R Peters, and Matthias Seeger. Local gaussian process regression for real time online model learning. In Advances in Neural Information Processing Systems, pp. 1193- 1200, 2009.
A. Panos, P. Dellaportas, and M. K. Titsias. Fully Scalable Gaussian Processes using Subspace Inducing Inputs. ArXiv e-prints, July 2018.
Chiwoo Park and Daniel Apley. Patchwork kriging for large-scale gaussian process regression. The Journal of Machine Learning Research, 19(1):269-31, 2018
Joaquin Quitonero-Candela and Carl Edward Rasmussen. A unifying view of sparse approximate gaussian process regression. Journal of Machine Learning Research, 6(Dec): 1939-1999, 2005.
Rajesh Ranganath, Dustin Tran, and David Blei. Hierarchical variational models. In International Conference on Machine Learning, pp. 324-333 224-333, 2016.
Carl Edward Rasmussen and Christopher K. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006.
H. Salimbeni, C.-A. Cheng, B. Boots, and M. Deisenroth. Orthogonally Decoupled Variational Gaussian Processes. ArXiv e-prints, September 2018.
Rishit Sheth, Yuyang Wang, and Roni Khardon. Sparse variational inference for generalized gaus- sian process models. Proceedings of the 32Nd International Conference on International Con- ference Machine Learning, ICML pp. 1302-1311. JMLR.org, 2015.
10

---

Under review as a conference paper at ICLR 2020
Edward Snelson and Zoubin Ghahramani. Sparse gaussian processes using pseudo-inputs. In Ad- vances in neural information processing systems, pp. 1257-1264, 2006.
Michael L Stein, Zhiyi Chi, and Leah J Welty. Approximating likelihoods for large spatial data sets. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 66(2):275-296, 2004.
Michalis K. Titsias. Variational learning of inducing variables in sparse gaussian processes. In Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics, pp- 2009.
Aldo V Vecchia. Estimation and model identification for continuous spatial processes. Journal of the Royal Statistical Society: Series B (Methodological)) 50(2):297-312 1988.


Reviews:
[{"review_id": "SkldBNWptS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a family of variational distributions in which the variational covariance matrix is parameterized as RR', with R_ij being nonzero only when j is a neighbor of i as defined by prior covariance. In other words, data points are only allowed to have nonzero posterior covariance if they are highly correlated a priori. This results in a sparse factor of the covariance matrix which can be used for efficient computation. Rather than being parameterized directly, the variational parameters \\mu_{i} and R_{i, n(i)} are parameterized using a GCN: the labels and prior covariance information are supplied to the GCN, which produces the mean for a data point x_i and the |n(i)| nonzero elements of the covariance factor R.\n\nOverall, the idea is interesting in the sense that a variational family with enforced sparsity likely leads to a reasonable probabilistic model. However, I have a few concerns about the execution.\n\nFirst, as a minor point, in my opinion, the approximations eqns. 3-7 are not sufficiently motivated, and are essential to the method as they allow for stochastic optimization. It would be useful to see an empirical analysis of the tightness of the additional approximations, as well as a generally expanded discussion in this section. Beyond this, the method is highly engineered but the only ablation study done of the various components is Figure 3, which merely offers an analysis of convergence speed, but not on final model performance. Both ideas introduced in the paper (localization and amortization) can be used independently of the other.\n\nMore importantly, I believe the experimental evaluation should be substantially broadened. At present, three datasets are considered and on one of them (MNIST) three methods considered are within the error bars of each other: the bolding in Table 3 is inappropriate. Many popularly used benchmark datasets for sparse GP methods are widely available, and it seems particularly essential to include datasets larger than the ones considered here, since exact GPs can trivially be trained on these datasets in a matter of seconds (at least for the regression tasks). Again ignoring the single classification task, Variational GP methods are usually only considered for regression for much larger datasets. I suspect that, with proper hardware (e.g. a GPU) and truly large datasets much of the speed advantage enjoyed by LAIN as reported in the paper will be lost to overhead.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HkewrlOptS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "PAPER SUMMARY:\n\nThis paper proposes a fast inference method for Gaussian processes (GPs) that imposes a sparse decomposition on the VI approximation of the posterior GP (for computational efficiency) using the KNN set of each data point. This is further coupled with armortized inference for better scalability. \n\nNOVELTY & SIGNIFICANCE:\n\nThis paper adopts a different approach of characterizing the VI approximation of a GP posterior than original VI approximation that was developed in Titsias (2009): Instead of characterizing the surrogate q(f_I) of p(f_I | Y) for a small collection of inducing inputs, the proposed method characterize q(f) directly where q(f) = int_f_I q(f_I) p(f | f_I)df_I.\n\nThis is, however, a somewhat strange direction which, to me, seems to raise extra issues that could have been avoided if one follows the conventional VI approximation:\n\n(1) As the posterior surrogate is now directly over f instead of f_I, the number of variational parameters is now proportional to the data size which requires several (redundant) extra approximations including armortized inference & the lower-bound on the entropy term that admits a sparse decomposition.\n\n(2) This at least creates the armortized and entropy gaps that decrease the expressiveness of the original VI. While I understand that this is in exchange for the ability to encode local information (via KNN) within the surrogate posterior, it is not clear to me why do we need to incur all these computational issues to incorporate such local information.\n\n(3) For example, instead of forcing such local information in the posterior surrogate q(f), we could alternatively let it be reflected in the test conditional p(f_* | f_I, Y_n(*)) such that the test output depends on both the inducing output and a local partition of data (e.g., via K-mean), which has been previously explored in [*] and later incorporated in the conventional VI paradigm of Titsias (2009) without incurring extra intractability [**].\n\n(4) This maintains the dense correlation between data points within the same neighborhood while allowing the VI surrogate to be more concisely specified and independent of the no. of training data points. Furthermore, it also helps avoid the data-bound overhead of computing a KNN per test point. \n\n[*] Local and global sparse Gaussian process approximations (AISTAT-07)\n[**] A distributed variational inference framework for unifying parallel sparse gaussian process regression models (ICML-16)\n\nTo summarize, the practical significance of placing such a VI approximation directly on q(f) to encode such (indirectional) local information is, given the above, questionable to me.\nPlease note that I am not disputing the potential use of this VI form here, which could have been the only way to encode a different (directional) type of information. \nFor encoding KNN information, however, it only seems to create more troubles than it solves. \n\nMinor point: \n\nThe above references, especially [*], should have been cited. \n\nTECHNICAL SOUNDNESS:\n\n[A] Optimization of the ELBO:\n\n(1) The ordering of data (i.e., the directional information) was mentioned repeatedly in the paper but its importance to the fast approximation was neither explained nor discussed.\n(2) The decomposition form of Eq. (6) also raises a question: How do we know that the term inside the log is positive? There seems to be missing information on the constraint of R.\n\n[B] Amortized Inference:\n\n(1) The choice of the GCN seems arbitrary here. I am in fact not sure why GCN is necessary for the inference network & furthermore, GCN also brings to the table another heuristic choice of A.\n(2) How do we set the adjacency graph A? \n(3) How do we know what is the right complexity for the GCN?\n\n[C] Complexity: \n\nThe complexity analysis is too informal and lacking fine-grained information. \nPlease include a detailed complexity analysis of the training and inference cost in terms of the input dimension, the no. of data points, the size of the neighborhood and the batch size.\nIt is also necessary to factor in the KNN overhead (e.g., the cost of building the K-D tree for low-dimensional embedding of data & the approximation cost of projecting that information to high-dimensional data)\n\nEXPERIMENT:\n\nThe experiment results only show marginal improvement over the baselines, and the size of the dataset for regression is too small. If I read correctly, both have fewer than 20000 data points.\nSVGP in particular has been tested on a much larger datasets (AIRLINE, UK Housing) comprising millions of data points -- comparison on such dataset should have been reported. \n\nNote that the largest dataset used to evaluate the efficiency of fast approximation of GP is on the scale of 6M data points [****]. On that note, eBird and precipitation should not even be considered mid-sized.\n\nTo demonstrate the efficiency of local information encoding, comparison with [*] should be reported. There is another class of inducing-point methods that use expectation propagation\nthat should have been discussed and/or compared with:\n\n[***] A Unifying Framework for Gaussian Process Pseudo-Point Approximations using Power Expectation Propagation (JMLR-18)\n\n[****] Distributed Gaussian Processes (ICML-15)\n\nCLARITY:\n\nThe paper is clearly written. \n\nREVIEW SUMMARY:\n\nThis paper adopts a VI approximation that deviates from the conventional form of (Titsias, 2009) to encode the KNN information, which causes extra computational issues (that incurs extra approximations). I find this deviation redundant seeing that the same information could have also been accounted for using the old VI form, which is a lot more computational efficient. I also find the experiment lacking as comparison with fast approximation method such as [*] that incorporate local information is not included. There are also a few technical ambiguities that need to be clarified.\n\n------ Post-Rebuttal Update ------\n\nThank you for the rebuttal & I have read it in detail. However, it still does not address my concern, which I re-summarize here:\n\nI do not dispute the beneficial of exploiting neighborhood information but my point is we could still leverage the same amount of neighborhood information without going through the trouble of incurring extra steps of approximation due to approximating q(f) instead of q(f_I) -- I think I have elaborated this in points (1) - (4) in my original review -- which also creates the amortized gap. I am also not sure what model reuse means (in this context) and whether it is relevant since it appears somewhat noncentric to the objective of this paper. \n\nAlso, apparently, the experiment is still lacking since to me, comparing with [*] is important in substantiating the contribution claim of this paper.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJelqU8X9H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1) Summary\nThe manuscript proposes a k-nearest-neighbor (KNN) Gaussian process (GP) approximate inference scheme to render computations more scalable.\n\n2) Quality\nAlthough the application is clear and the methodology is well established, the quality of the submission can be improved by a more thourough empirical evaluation in particular a proper evaluation in terms of runtime, approximation accuracy and comparison to baseline methods.\n\n3) Clarity\nThe manuscript is reasonably well written and most of the technical and experimental content is accessible. There are some typos and some glitches in the notation. See \"Details\". There are some open issues regarding the KNN computations. See Questions.\n\n4) Originality\nThe use of a localized (in the KNN sense) set of inducing inputs to improve GP inference but the impact needs to be better quantified empirically.\n\n5) Significance\nThe proposed method is aiming at improving the established setting of GP inference. The modification is rather marginal and the empirical evaluation makes it hard to judge the relative merit of the proposal.\n\n6) Reproducibility\nThe data is from published sources (toy, ebirds, precipitation, digits) and the code for the baseline methods and for the LAIM method itself is available. However, there is no code for the experiments, which makes the results slightly tricky to exactly reproduce.\n\n7) Evaluation\nThe evaluation does not consider simple baselines like dense GPs or sparse approximations such as FITC and VFE. Also plain NN should be considered.\n\n8) Questions\n  A) How do you set the parameter delta?\n  B) How are the nearest-neighbors computed in the first place? Does it require computing the dense covariance matrix?\n  C) How accurate is the NN computation? How much of the computational effort (percentage) of the overall pipeline is required for the NN computation?\n  D) How much better is the proposed approach than directly using NN predictions?\n  E) Does \"most correlated\" in the footnote on page 2 really mean correlation or is it about covariance? The latter would involve a diagonal rescaling of the covariance matrix.\n\n9) Details\n  a) Abstract: \"Gaussian Processes\" -> \"Gaussian processes\"\n  b) Intro: \"GP poses a Gaussian prior\" -> funny sentence, \"with some special \"\n  c) Intro: \"with some special structures\" -> fix\n  d) Background: \"q(f)~N(mu,V)\" -> imprecise notation, rather \"q(f)=N(mu,V)\"\n  e) Background, footnote: \"distance metrics\" -> Are you talking about \"distance\" or \"metric\"?\n  f) \"Experiment\" -> \"Experiments\"\n  g) References: capitalization not correct e.g. Gaussian, Fourier, Bayes", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for speeding up Gaussian process inference by leveraging locality information through k-nearest neighbours. 

The key idea is well-motivated intuitively, however the way in which it is implemented seems to introduce new complications. One such issue is KNN overhead in high dimensions, but R1 outlines other potential issues too. Moreover, the method's merit is not demonstrated in a convincing way through the experiments. The authors have provided a rebuttal for those issues, but it does not seem to solve the concerns entirely.

Decision: Reject