Conditioning Sparse Variational Gaussian Processes for Online Decision-making

1 Introduction . Intelligent systems should be able to quickly and efficiently adapt to new data , adjusting their prior beliefs in response to the most recent events . These characteristics are desirable whether the system in question is controlling the actuators of a robot , tuning the power output of a laser , or monitoring the changing preferences of users on an online platform . What these applications share in common is a constant stream of new information . In this paper , we are interested in efficient conditioning , meaning that we wish to efficiently update a posterior distribution after receiving new data . The ability of Gaussian process ( GP ) regression models to condition on new data in closed form has made them a popular choice for Bayesian optimization ( BO ) , active learning , and control [ 24 ] . All of these settings share similar characteristics : there is an “ outer loop ” , where new data is acquired from the real world ( e.g . an expensive simulator ) , interleaved with an “ inner loop ” , which chooses where to collect data . In BO , for example , the “ inner loop ” is the optimization of an acquisition function evaluated using a surrogate model of the true objective . Simple acquisition functions , e.g . expected improvement ( EI ) , consider only the current state of the surrogate , while more sophisticated acquisition functions “ look ahead ” to consider the effect of hypothetical observations on future surrogate states . One such acquisition function , batch knowledge gradient ( qKG ) , defines the one-step Bayes-optimal data batch as the batch that maximizes the expected surrogate maximum after the batch has been acquired [ 2 , 84 ] . Advanced acquisition functions like qKG require the surrogate to have both efficient posterior sampling and efficient conditioning on new data . GP regression has two major limitations that have prevented its large scale deployment for online decision-making . First , the computational and memory consumption of exact GPs grows at least quadratically with the amount of data [ 25 , 65 ] , generally limiting their usage to BO problems with fewer than 1 , 000 function evaluations [ 24 , 2 , 79 ] . Second , they are limited to applications that have continuous real-valued responses , enabling modelling with solely a Gaussian likelihood . Stochastic 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . responses ( b ) and non-Gaussian models ( c ) such as the Gaussian copula volatility model [ 82 ] . variational Gaussian processes ( SVGPs ) [ 33 ] have constant computational and memory footprints and are applicable to non-Gaussian likelihoods , but they sacrifice closed form expressions for updated posteriors on receiving new data . The SVGP posterior is optimized through the evidence lower bound ( ELBO ) . In the online setting , training with the ELBO has two primary difficulties : the need to specify a fixed number of observations to properly scale the ELBO gradient [ 8 ] and the need to adjust the inducing points without looking at past data [ 3 , 9 ] . Thus , we are presented with a choice between the simplicity and analytic tractability of exact GPs and the scalability and flexibility of SVGPs . In this work , we develop Online Variational Conditioning ( OVC ) to allow SVGPs to be conditioned on-the-fly , as shown in Figure 1 . In the top row of each subplot , we fit the data points shown in red , shifting to another batch of data points in the bottom row . We use an exact GP in Figure 1a , with exact conditioning shown in the bottom panel . The SVGP emulates the exact GP very well before seeing the new data and again after conditioning on the new data by using OVC ( Figure 1b ) . In Figure 1c , we consider a non-Gaussian data model ( a Gaussian copula volatility model [ 82 ] ) , where we can not use exact GPs ; the SVGP is still able to update its posterior over the latent volatility in response to new data without “ forgetting ” old observations . OVC is inspired by a new , simple rederivation of streaming sparse GPs ( O-SGPR ) , originally proposed by Bui et al . [ 9 ] . OVC makes SVGPs truly compelling models for online decision-making , augmenting their existing strengths with efficient , closed-form conditioning on new data points . In short , our contributions are : • The development of OVC , a novel method to condition SVGPs on new data without reoptimizing the variational posterior through an evidence lower bound . • OVC provides both stable inducing point initialization for SVGPs while enabling the inducing points and variational parameters to update in response to the new data . • Enabling the effective application of SVGPs , through OVC , in look-ahead acquisitions in BO for black-box optimization , controlling dynamical systems , and active learning . Please see Appendix A for discussion of the limitations and broader impacts of our work . Our code is available at https : //github.com/wjmaddox/online_vargp . 2 Preliminaries and Related Work . In this section , we first review exact inference with Gaussian processes before reviewing variational sparse Gaussian processes , introducing sparse Gaussian process regression ( SGPR ) , sparse variational Gaussian processes ( SVGP ) , and streaming sparse GPs ( O-SGPR ) . Please see Appendix B for further background and Appendix B.1 specifically for further related work . 2.1 Gaussian Processes and Exact Inference . For a full introduction to GPs , see Rasmussen and Williams [ 65 ] . We begin by reviewing GP regression , with inputs x 2 X and responses y 2 R. GP regression assumes y ⇠ N ( f , 2 ) , where f ⇠ GP ( 0 , k✓ ( x , x0 ) ) is a GP characterized by the kernel function k✓ and corresponding hyperparameters ✓ . Given Xtrain = [ x1 , . . . , xn ] > and Xtest = [ x01 , . . . , x0k ] > , we denote the corresponding function values as v = [ f ( x1 ) , . . . , f ( xn ) ] > and w = [ f ( x01 ) , . . . , f ( x0k ) ] > , respectively . Note that v and w are random variables with prior covariance Cov [ v , w ] = Kvw : = k✓ ( Xtrain , Xtest ) . A well-known identity for multivariate conditional Gaussians allows us to compute the GP predictive posterior p ( w|Xtest , D , ✓ ) = N ( µ⇤w|D , ⌃⇤w|D ) as follows : µ⇤w|D = Kwv ( Kvv + 2I ) 1y , ⌃⇤w|D = Kww Kwv ( Kvv + ⌃y ) 1Kvw , ( 1 ) where D : = ( Xtrain , y ) and ⌃y = 2I . Naïvely , computations with Kvv cost O ( n2 ) space and O ( n3 ) operations . When repeatedly computing p ( w|Xtest , D , ✓ ) at different sets of query points Xtest , it is more efficient to cache the terms which depend only on the training data.1 Specifically , we store a : = ( Kvv + ⌃y ) 1y ( the predictive mean cache ) and RR > : = ( Kvv + ⌃y ) 1 ( the predictive covariance cache ) , resulting in simplified forms for the predictive distribution , µ⇤w|D = Kwva and ⌃ ⇤ w|D = Kww KwvRR > Kvw . Adding a new observation is equivalent to adding a single row and column to Kvv and an entry to y , which enables efficient low-rank updates to the predictive caches [ 58 , 25 , 64 , 39 ] .2 2.2 Variationally Sparse Gaussian Processes . Variational sparse GPs reduce the computational burden of GP inference through sparse approximations of the kernel matrix . For further reference , see Matthews [ 50 ] and Van der Wilk [ 77 ] . These methods define a variational distribution over the inducing point values u = [ f ( z1 ) , . . . , f ( zp ) ] > , defined at inducing point locations Z = [ z1 , · · · , zp ] > , where zi 2 X . ( u ) is parameterized as a Gaussian with variational mean and covariance mu and Su . These methods assume the latent function values f ( x ) , f ( x0 ) are conditionally independent given u and x , x0 /2 Z , so as to cheaply approximate the predictive posterior p ( w|Xtest , D , ✓ ) ⇡ q ( w ) = N ( µ⇤w| , ⌃⇤w| ) . Like exact GP regression , we can compute q ( w ) in closed form ( given mu , Su ) , µ⇤w| : = KwuK 1 uumu , ⌃ ⇤ w| : = Kww KwuK 1uu ( Kuu Su ) K 1uuKuw . ( 2 ) Similarly the predictive mean and covariance caches are given by a = K 1uumu and RR > = K 1uu ( Kuu Su ) K 1uu , reducing the complexity of inference from O ( n3 ) to O ( np2 ) , which is a significant improvement if p ⌧ n. There are two common approaches to finding optimal variational parameters mu and Su . In seminal work , Titsias [ 73 ] proposed sparse GP regression ( SGPR ) , which optimizes mu and Su in closed form , resulting in a `` collapsed '' evidence lower bound3 ( ELBO ) that only depends on ✓ and Z . The computational cost of each gradient update to the remaining model parameters is still linear in n , and like exact GP regression , SGPR requires a Gaussian likelihood . Stochastic variational GPs ( SVGPs ) remedies both these limitations by using gradient-based optimization to learn mu and Su alongside Z and ✓ [ 33 , 34 ] . The SVGP objective is an “ uncollapsed ” ELBO which decomposes additively across the training examples , allowing gradients to be estimated from minibatches of data , reducing the complexity of each gradient update to O ( bp2 + p3 ) , where b is the minibatch size . We emphasize the distinction between constant-time minibatch gradients , and constant-time conditioning . Given an SVGP already trained on some existing data , conditioning jointly on both the old and new data requires storing all the data and making multiple gradient updates to the variational parameters . As the size of the dataset grows , so does the number of gradient steps needed . In contrast by constant-time conditioning we mean a procedure that takes a posterior conditioned on existing data and produces a new posterior conditioned jointly on the old and new data with a fixed amount of compute and memory , regardless of the number of past observations . One example of constant-time conditioning is found in Bui et al . [ 9 ] , who proposed streaming sparse GPs ( which we call online SGPR , or O-SGPR , to distinguish from sparse spectrum GPs [ 44 ] ) for 1Cached expressions will be written in orange 2What we use as the SGPR covariance cache is slightly different from the implementation in the prediction strategy in GPyTorch , which stores R = KvuK 1/2uu . However , they reduce to the same strategy . 3A lower bound of the true GP marginal log-likelihood Algorithm 1 Online Variational Conditioning ( OVC ) Input : Data batch ( Xbatch , y ) , SVGP with inducing points Z 0 and ( u0 ) = N ( mu0 , Su0 ) . 1 . Compute c0 , C 0 ( Eq . 5 ) . 2 . Compute ŷ = K 0u0u0C 0 1c0 and ⌃ŷ = K 0 1u0u0C 0K 0 1u0u0 ( Eq . 8 ) . 3 . Construct GP with D = ( [ Xbatch Z 0 ] , [ y ŷ ] ) and ⌃ = blkdiag ( ⌃y , ⌃ŷ ) . 4 . Compute predictive mean and covariance caches , a and RR > as in Section 2.1 . 5 . Use caches to compute conditioned GP posterior on test points , Xtest . incremental learning . We extend their work , providing an alternative , simpler derivation of their model that highlights the connection with SGPR [ 73 ] . Furthermore , our perspective enables us to construct a principled approach to updating inducing point locations as new data arrives , that prevents the “ forgetting ” of old data induced by the resampling heuristic used by Bui et al . [ 9 ] .

Summary: The authors are interested in the problem of online learning with applications to online Decision making. The focus is on (sparse variational) Gaussian Processes and Bayesian optimization.  The authors 1) propose an alternative formulation of the online learning algorithm of Bui et al, which allow to ‘preserve’ what has been learned from the previous data 2) propose a way to update the posterior when adding observations in the conditioning set for non Gaussian likelihood 3) Thoroughly evaluate the resulting algorithm on various BO tasks.