Deep Recurrent Gaussian Process with Variational Sparse Spectrum Approximation

1 INTRODUCTION . Modeling sequential data for simulation tasks in the context of machine learning is hard for several reasons . Their internal structure poses the problem of modeling short term behavior and long term behavior together for different types of data variables , where the data variables themselves might differ in the information gain in the chosen time frequency . Recurrent models ( Hochreiter & Schmidhuber , 1997 ; Nelles , 2013 ; Pascanu et al. , 2013 ) have proven to perform well on these tasks . They consist of output-data and input-data structured sequentially for shifted discrete time steps . The general form of a recurrent model is given by hi = ζ ( hi−1 , . . . , hi−H , xi−1 , . . . , xi−H ) + h i , yi = ψ ( hi , . . . , hi−H ) + y i , ( 1 ) ( 2 ) where xi is an external input , yi is an output observation , hi is a latent hidden representation or state at time i = H + 1 , . . . , N , where N ∈ N is the number of data samples , H ∈ N is the chosen time horizon , ζ , ψ are non-linear functions modeling transition and observation and hi , y i are transition and observation noise , which are adjusted for the specific problem ( details on dimensions and ranges will be specified in upcoming sections ) . In control and dynamical system identification previous work on Bayesian recurrent approaches for modeling sequential data usually makes use of ( non- ) linear auto-regressive with exogenous inputs models ( ( N ) ARX ) and state-space models ( SSM ) , for both see Nelles ( 2013 ) . The general recurrent model given in Equation ( 1 ) and ( 2 ) represents both cases . This can be recognized by its general recurrent and hierarchical structure . This work deals with deep learning in a recurrent fashion for modeling sequential data in a Bayesian non-parametric approach by using GPs . To make a connection to the general recurrent model , the deep structure arises by defining ζ in Equation ( 1 ) in a deep manner ( Pascanu et al. , 2013 , Section 3 ) . To achieve scalability , GPs normally make use of sparse approximations for the covariance function . This paper proposes DRGP models based on ( V ) SS approximations , denoted by DRGP- ( V ) SS . For reproducibility of the experimental results , we provide the code online1 . Therefore , we follow the same deep recurrent structure as introduced in Mattos et al . ( 2016 ) . To summarize , the contributions of this paper are the following : • Extension of the sparse GP based on the SS approximation introduced by Lázaro-Gredilla et al . ( 2010 ) and the improved VSS approximation by Gal & Turner ( 2015 ) to DRGPs ; • Improvement of regularization properties of the variational bounds through the combination of the ( V ) SS approximations with the inducing-point ( IP ) regularization of Titsias & Lawrence ( 2010 ) ; • Propagation of uncertainty through the hidden layers of our DGPs by variationally integrating out the input-space ; • The existence of an optimal variational distribution in the sense of a functional local optimum of the variational bounds of our DRGPs models is established . The DRGP of Mattos et al . ( 2016 ) is limited to a small class of deterministic covariance functions , because the covariance functions variational expectation has to be analytically tractable . Using the ( V ) SS approximations instead , we can derive a valid approximation for every stationary covariance function , because the basis functions expectation is always tractable . We show that this approach improves over state of the art approaches in prediction accuracy on several cases of the experimental data-sets used in Mattos et al . ( 2016 ) ; Svensson et al . ( 2016 ) ; Al-Shedivat et al . ( 2017 ) ; Salimbeni & Deisenroth ( 2017 ) ; Dörr et al . ( 2018 ) in a simulation setting . For scalability , Distributed Variational Inference ( DVI ) ( Gal et al. , 2014 ) is recommended and can lower the complexity from O ( NM2Qmax ( L+ 1 ) ) down toO ( M3 ) for N ≤M , M the sparsity parameter , ( L+ 1 ) the amount of GPs and Qmax is the maximum over all input dimensions used in our defined deep structure for ζ and ψ . Therefore , the number of cores must scale suitably with the number of training-data . 2 RELATED WORK TO GAUSSIAN PROCESSES WITH SSM AND DGPS . An SSM with GPs ( GP-SSM ) for transition and observation functions is used by Wang et al . ( 2005 ) , where the uncertainty in the latent states is not accounted for , which can lead to overconfidence . Turner et al . ( 2010 ) solved this problem , but they have complicated approximate training and inference stages and the model is hard to scale . Frigola et al . ( 2014 ) used a GP for transition , while the observation is parametric . Svensson et al . ( 2016 ) used an approximation of the spectral representation by Bochner ’ s theorem in a particular form and with a reduced rank structure for the transition function . They realize inference in a fully Bayesian approach over the amplitudes and the noise parameters . The construction of Eleftheriadis et al . ( 2017 ) involves a variational posterior that follows the same Markov properties as the true state with rich representational capacity and which has a simple , linear , time-varying structure . Dörr et al . ( 2018 ) introduced a GP-SSM with scalable training based on doubly stochastic variational inference for robust training . Our models extend the GP-SSM framework by defining the transition as a DRGP based on our newly derived ( V ) SS approximations in the Sections 3.3 , 3.4 , where the latent ( non-observed ) output-data is learned as a hidden state . We refer to the report Föll et al . ( 2017 ) 2 for a detailed but preliminary formulation of our models and experiments . Following Damianou & Lawrence ( 2013 ) , a Deep Gaussian Process ( DGP ) is a model assuming yi = f ( L+1 ) ( f ( L ) ( f ( L−1 ) ( . . . ( f ( 1 ) ( xi ) + h ( 1 ) i ) . . . ) + h ( L−1 ) i ) + h ( L ) i ) + y i , where the index i = 1 , . . . , N is not necessarily the time and where we define h ( 1 ) i def = f ( 1 ) ( xi ) + h ( 1 ) i , h ( l+1 ) i def = f ( l ) ( h ( l ) i ) + h ( l ) i , for l = 2 . . . , L − 1 , where L ∈ N is the number of hidden layers . The noise h ( l ) i , y i is assumed Gaussian and the functions f ( l ) are modeled with GPs for l = 1 , . . . , L+ 1 . To obtain computational tractability , in most cases variational approximation and inference is used . Damianou & Lawrence ( 2013 ) introduced these kind of DGPs based on the sparse variational approximation following Titsias ( 2009 ) ; Titsias & Lawrence ( 2010 ) . Based on this , 1DRGP- ( V ) SS code available from http : //github.com/RomanFoell/DRGP-VSS . 2Available on the website https : //arxiv.org/ . Dai et al . ( 2016 ) introduced a DGP with a variationally auto-encoded inference mechanism and which scales on larger data-sets . Cutajar et al . ( 2016 ) introduced a DGP for the so called Random Fourier Features ( RFF ) approach ( Rahimi & Recht , 2008 ) , where the variational weights for each GPs are optimized along with the hyperparameters . This approach does not variationally integrate out the latent inputs to carry through the uncertainty and no existence of an optimal variational distribution for the weights is proven to reduce the amount of parameters to optimize in training . Furthermore , Salimbeni & Deisenroth ( 2017 ) introduced an approximation framework for DGPs , which is similar to the single GP of Hensman & Lawrence ( 2014 ) , but does not force independence between the GP layers and which scales to billions of data . Two state of the art approaches for DRGPs have been introduced by Mattos et al . ( 2016 ) , the RGP , which we call DRGP-Nyström , based on the variational sparse Nyström/inducing-point approximation introduced by Titsias ( 2009 ) ; Titsias & Lawrence ( 2010 ) , as well as Al-Shedivat et al . ( 2017 ) , which we call GP-LSTM , based on deep kernels via a Long-short term memory ( LSTM ) network ( Hochreiter & Schmidhuber , 1997 ) , a special type of Recurrent Neural Network ( RNN ) . DRGP-Nyström uses a recurrent construction , where the auto-regressive structure is not realized directly with the observed output-data , but with the GPs latent output-data and uses a Variational Inference ( VI ) framework , named Recurrent Variational Bayes ( REVARB ) . The structure acts like a standard RNN , where every parametric layer is a GP . So additionally uncertainty information can be carried through the hidden layers . GP-LSTM is a combination of GPs and LSTMs . LSTMs have proven to perform well on modeling sequential data . LSTMs try to overcome vanishing gradients by placing a memory cell into each hidden unit . GP-LSTM uses special update rules for the hidden representations and the hyperparameters through a semi-stochastic optimization scheme . It combines a GP with the advantages of LSTMs by defining structured recurrent deep covariance functions , also called deep kernels , which reduces the time and memory complexities of the linear algebraic operations ( Wilson et al. , 2016 ) . 3 GAUSSIAN PROCESSES AND VARIATIONAL SPARSE SPECTRUM GP . Loosely speaking , a GP can be seen as a Gaussian distribution over functions . We will first introduce GPs and GP regression and then recall the SSGP by ( Lázaro-Gredilla et al. , 2010 ) and its improved version VSSGP by ( Gal & Turner , 2015 ) . Based on these , we derive new variational approximations . We use the notation a , fx , y , ( italic ) for random variables , a , f ( x ) , y ( upright ) for realizations/samples and data . 3.1 GAUSSIAN PROCESSES . A stochastic process f is a GP if and only if any finite collection of random variables fX def = [ fx1 , . . . , fxN ] T forms a Gaussian random vector ( Rasmussen , 2006 ) . A GP is completely defined by its mean function m : RQ → R , x 7→ m ( x ) , Q ∈ N the input-dimension , and covariance function k : RQ × RQ → R , ( x , x′ ) 7→ k ( x , x′ ) ( Kallenberg , 2006 , Lemma 11.1 ) , where m ( x ) def = E [ fx ] , k ( x , x ′ ) def = cov ( fx , fx′ ) = E [ ( fx −m ( x ) ) ( fx′ −m ( x′ ) ) ] , and the GP will be written as f ∼ GP ( m , k ) . Be aware of that a valid covariance function must produce a positive definite matrixKNN def = ( k ( xi , xj ) ) N i , j=1 ∈ RN×N , when filling in combinations of data-input points xi , i = 1 , . . . , N . Let y def= [ y1 , . . . , yN ] T ∈ RN , X def = [ x1 , . . . , xN ] T ∈ RN×Q and we assume yi = f ( xi ) + yi , where yi ∼ N ( 0 , σ2noise ) , for i = 1 , . . . , N , and our aim is to model any set of function values f def = [ f ( x1 ) , . . . , f ( xN ) ] T ∈ RN at X as samples from a random vector fX . Moreover , we assume the prior fX|X ∼ N ( 0 , KNN ) , meaning that any set of function values f given X are jointly Gaussian distributed with mean 0 ∈ RN and a covariance matrix KNN . The predictive distribution pfx∗ |x∗ , X , y for a test point x ∗ ∈ RQ , where KN∗ def = ( k ( xi , x ∗ ) ) Ni=1 ∈ RN , and analogously K∗∗ , K∗N , can be derived through the joint probability model and conditioning as fx∗ |x∗ , X , y ∼ N ( K∗N ( KNN + σ2noiseIN ) −1y , K∗∗ −K∗N ( KNN + σ2noiseIN ) −1KN∗ ) . In preview of our experiments in Section 5 and the following sections , we choose a specific covariance function , the spectral mixture ( SM ) covariance function ( Wilson & Adams , 2013 ) k ( x , x′ ) = σ2power ( ∏Q q=1 exp ( −1 2 l−2q ( xq − x′q ) 2 ) ) cos ( 2π ∑Q q=1 p−1q ( xq − x′q ) ) , ( 3 ) with an amplitude σ2power and length scales lq , pq ∈ R , q = 1 , . . . , Q . As pq →∞ , this corresponds to the squared exponential ( SE ) covariance function in the limit ( Gal & Turner , 2015 ) .

Summary: This paper addresses the problem of modeling sequential data based on one of the deep recurrent Gaussian process (DRGP) structures proposed by Mattos et al (2016). This structure acts like a recurrent neural net where every layer is defined as a GP. One of the main limitations of the original method proposed by Mattos et al (2016) is that it is limited to a small set of covariance functions, as the variational expectations over these have to be analytically tractable.