Scalable Gaussian Processes with Grid-Structured Eigenfunctions (GP-GRIEF)

We introduce a kernel approximation strategy that enables Gaussian process training and inference in Opdnpq time and Opdnq storage for a d-dimensional dataset of size n. Our GRIEF kernel consists of p eigenfunctions approximated on a dense Cartesian tensor product grid of inducing points. We show that by exploiting algebraic properties of Kronecker and Khatri-Rao tensor products, computational complexity of the training procedure can be independent of the number of inducing points, allowing us to use arbitrarily many to achieve a globally accurate kernel approximation. We benchmark our algorithms on real-world datasets with as many as two-million training points and up to 10 inducing points. Gaussian process (GP) modelling is a powerful Bayesian approach for classification and regression, however, it is restricted to modestly sized datasets since training and inference require Opn3q time and Opn2q storage, where n is the number of training points [1]. This has motivated the development of approximate GP methods that use a set of m (! nq inducing points to reduce the computational cost and memory requirements to Opm2nq and Opmnq, respectively [2–5]. However, such techniques perform poorly if too few inducing points are used, and any computational savings are lost on complex datasets that require m to be large. Wilson and Nickisch [6] exploited the structure of inducing points placed on a Cartesian product grid, allowing for m ¡ n while dramatically reducing computational demands over an exact GP. This inducing point structure enables significant performance gains in low-dimensions, however, time and storage complexities scale exponentially with the dataset dimensionality, rendering the technique intractable for general learning problems unless a dimensionality reduction procedure is applied. In the present work, a Cartesian product grid of inducing points is also considered, however, we show that these computational bottlenecks can be eliminated by identifying and exploiting further structure of the resulting matrices. The proposed approach leads to a highly scalable algorithm that performs exact training and inference with a non-degenerate GP in Opdnpq time and Opdnq memory, where d is the dimensionality of the dataset, and p is the number of eigenfunctions that we will describe next. We emphasize that our complexity is independent of m, which can be set arbitrarily high. Our central idea is to approximate the exact kernel as a finite sum of eigenfunctions that can be found efficiently using a Nyström approximation [5]. This approach is similar to the “Nyström method” of Williams and Seeger [4], however, our final approximation is a valid probabilistic model. In other words, our model is sparse in the kernel eigenfunctions rather than the number of inducing points, which can greatly exceed the size of the training set due to the structure we introduce. This is attractive since it is well-known that eigenfunctions produce the most compact representation among orthogonal basis functions. Although our eigenfunctions are approximate, they convergence in the limit of large n [7]. Additionally, our ability to consider a huge value of m allows us to fill out the input space with inducing points, thereby enabling accurate global approximations of the eigenfunctions, even far from the training data. These basis functions also live in a reproducing kernel Hilbert space, unlike other sparse GPs whose bases have a pre-specified form (e.g. Lázaro-Gredilla 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA. et al. [8]). In addition, we show how to recover a non-degenerate kernel by including a correction term that further improves our approximation without affecting computational complexity. We begin by outlining our approximate kernel for a general distribution of inducing points. We then place the inducing points on a grid and discuss how our method remains computationally attractive even in high-dimensional problems with m " n. The discussion will focus on solving a linear system with the kernel covariance matrix which is the most expensive part of GP training and inference. 1 Approximate Kernel Given a kernel k : R R Ñ R, we consider its approximation in terms of p eigenfunctions (obtained using the Nyström approximation) written in the following form [5]