Learning Nonparametric Volterra Kernels with Gaussian Processes

1 Introduction . Gaussian processes ( GPs ) constitute a general method for placing prior distributions over functions , with the properties of samples from the distribution being controlled primarily by the form of the covariance function [ 22 ] . Process convolutions ( PCs ) are one powerful method for building such covariance functions [ 4 , 13 , 3 ] . In the PC framework , the function we wish to model is assumed to be generated by the application of some convolution operator to a base GP with some simple covariance , and since linear operators applied to GPs result in GPs , the result is another GP with a covariance we deem desirable . PCs allow models for multiple correlated output functions to be built with ease , by assuming each output is generated by a different operator applied to the same base function , or set of functions [ 30 , 13 ] . The PC framework unifies a number of different ideas in the GP literature . Latent force models ( LFMs ) [ 1 ] use PCs to include physics based inductive biases in multiple output GP ( MOGP ) models by using the Green ’ s function of a linear differential operator as the kernel of the convolution . This leads to the interpretation of each output as having been generated by inputting a random latent force into a linear system , with physical properties described by the differential operator . The Gaussian process convolution model ( GPCM ) of Tobar et al . [ 29 ] treats the convolution kernel itself as an unknown function to be inferred from data , and places a GP prior over it . Linear systems are entirely 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . described by their Green ’ s function , so one can interpret the GPCM as a nonparametric , linear LFM , in which the form of the system itself is inferred from data . In the physical world , nonlinear systems are the norm , and linearity is an approximation . As a consequence it is desirable when dealing with physical data to have models that can incorporate nonlinearity naturally . Often however , given a certain set of data , it is not clear exactly what form this nonlinearity takes , and so introducing specific parametric nonlinear operators can be overly restrictive . Álvarez et al . [ 2 ] present a model known as the nonlinear convolved MOGP ( NCMOGP ) which introduces nonlinearity to MOGPs via the Volterra series [ 8 ] , a nonlinear series expansion used widely for systems identification , whose properties are controlled by a set of square integrable functions of increasing dimensionality known as the Volterra kernels ( VKs ) . The NCMOGP assumes these functions are both separable and homogeneous , and of parametric Gaussian form , it also approximates the outputs as GPs in order to make inference tractable . The present work introduces a new model which drops the separability and homogeneity assumptions on the VKs , allows their form to be learned directly from data , and makes no approximation on the distribution of the outputs . We refer to it as the nonparametric Volterra kernels model ( NVKM ) . We develop a fast sampling method for the NVKM which leverages the recent results of Wilson et al . [ 35 ] on the sampling of explicit functions from GPs to analytically map function realisations through the Volterra series , avoiding the need for computationally expensive and inaccurate high dimensional numerical integration . Fast sampling allows for the application of doubly stochastic variational inference ( DSVI ) [ 28 ] for scalable learning . The NVKM is well suited to both single and multiple output regression problems , and can be thought of as an extension of the GPCM to both nonlinear systems and multiple outputs . The NVKM can also be interpreted as a nonlinear LFM in which the operator is learned directly from data . We additionally present a variation of the NVKM that can be used for Bayesian systems identification , where the task is to learn operator mappings between observed input and output data , and show that it allows for considerably better quantification of uncertainty than competing methods which use recurrence [ 18 ] . 2 Background . In this section we give a brief introduction to the mathematical background of PCs and the Volterra series . Process convolutions In the PC framework , the set of output functions { fd ( t ) } Dd=1 , with D being the number of outputs , is generated by the application of some set of linear operators , specifically convolution operators , to a latent function u represented by a Gaussian process , fd ( t ) = ∫ T Gd ( t− τ ) u ( τ ) dτ , where T is the domain of integration , and the function Gd is known variously as the convolutional kernel , smoothing kernel , impulse response or Green ’ s function , depending on context . Assuming that the input u is bounded , the function Gd must be square integrable to ensure the output is finite . A linear operator acting on a GP produces another GP [ 22 ] , and so we obtain D distinct GPs . Since the latent function u is shared across the outputs , these D GPs are correlated , allowing joint variations to be captured , whilst the convolution with Gd adapts u to each output . Álvarez et al . [ 3 ] show that many MOGP models can be recast in terms of the PC framework by particular choices of Gd and u . In LFMs , Gd is taken to be the Green ’ s function of some differential operator . We can then interpret each output as resulting from a shared random force being fed into a distinct linear system , represented by the differential operator . The smoothing kernels are usually taken to have parametric , often Gaussian form . Tobar et al . [ 29 ] use the PC framework for a single output , and make the smoothing kernel itself a GP , giving rise to the GPCM . Bruinsma [ 7 ] extends the GPCM to the multiple output case , although the model was not applied to data . Volterra Series The PC framework can be extended to represent a broader class of output functions by instead considering the outputs { fd ( t ) } Dd=1 as being the result of some nonlinear system , acting on the latent function u . The Volterra series is a series approximation for nonlinear systems that is widely used in the field of system identification [ 8 ] . It is given by , f ( t ) = C∑ c=1 ∫ T Gc ( t− τ1 , . . . , t− τc ) c∏ j=1 u ( τj ) dτj , ( 1 ) where f is the system output , Gc is the cth order VK , u is the system input , and C is the order of the approximation . We can think of the Volterra series as an extension of the well known Taylor expansion , that allows f to have memory of the past values of u , that is to say f depends on u at all values t ∈ T . The Volterra series can approximate a broad class of nonlinear operators , however it is unable to represent certain properties of general nonlinear systems , for example chaos . Álvarez et al . [ 2 ] use Equation ( 1 ) to construct the NCMOGP , applying the Volterra series with d distinct sets of VKs { Gd , c } Cc=1 to a shared latent GP input u , to produce outputs { fd ( t ) } Dd=1 . Since the Volterra series is a nonlinear operator , the output becomes an intractable , non-Gaussian process . The authors perform inference by approximating the outputs as GPs and using the first and second moments of the output process to form its mean and covariance function . To enable to computation of these moments , the authors restrict the set of VKs to those which are both separable and homogeneous , i.e . Gd , c ( t1 , . . . , tc ) = ∏c i=1Gd ( ti ) . Additionally , since the moment computation requires analytically solving a number of non-trivial convolution integrals , the authors only consider a Gaussian form for the VKs . 3 The nonparametric Volterra kernels model . The NVKM relaxes the restrictions of separability and homogeneity which are placed on the VKs in the NCMOGP , and represents these kernels as independent GPs , allowing their form and uncertainty to be inferred directly from data . The generative process for the NVKM can be stated as u ( t ) ∼ GP ( 0 , k ( u ) ( t , t′ ) ) , Gd , c ( t ) ∼ GP ( 0 , k ( Gd , c ) ( t , t′ ) ) , t ∈ Rc , ∀c ∈ 1 , . . . , C , ∀d ∈ 1 , . . . , D , fd ( t ) = C∑ c=1 ∫ ∞ −∞ Gd , c ( t− τ1 , . . . , t− τc ) c∏ j=1 u ( τj ) dτj , ( 2 ) where k ( u ) ( t , t′ ) is the covariance function for the input process , and k ( Gd , c ) ( t , t′ ) is the covariance function for the cth VK of the dth output . We follow Tobar et al . [ 29 ] in using the decaying square exponential ( DSE ) covariance for the VKs , which is a modification to the ubiquitous square exponential ( SE ) covariance that ensures the samples have finite energy.1 The DSE covariance has the form kDSE ( t , t ′ ) = σ2 exp ( −α ( ‖t‖2 + ‖t′‖2 ) − γ‖t− t′‖2 ) , ( 3 ) where ‖ · ‖ is the or ` 2 norm , σ is the amplitude , α controls the rate at which the samples decay away from the origin , and γ is related to the length scale l of the samples by γ = 1l2 . A diagram of the generative process for the model is shown in Figure 1 . Obtaining an exact distribution over the outputs fd is intractable , since it involves integration over nonlinear combinations of infinite dimensional stochastic processes . In order to sample from the model , and perform inference , approximations must be introduced . In particular , we employ the results of Wilson et al . [ 35 ] to sample in linear time , which enables efficient learning through the use of variational inducing points [ 27 ] with doubly stochastic variational inference ( DSVI ) [ 28 ] . 3.1 Sampling . One could sample from the model by drawing from the input and filter GPs at some finite set of locations , and then using the samples with some method for numerical integration to find the output . As the dimensionality of the filters increases , however , many points would be needed to obtain an accurate answer , this quickly becomes computationally intractable , since sampling exactly from a GP has cubic time complexity with respect to the number of points requested . We can sidestep this problem , and avoid the need for any numerical integration , by representing samples from the GPs explicitly as functions . Using the results from Wilson et al . [ 35 ] , and following their notation , we can write a sample from a GP f : Rc −→ R , with covariance function k ( t , t′ ) , given M inducing variables v ∈ RM with corresponding inducing inputs { zj } Mj=1 , with zj ∈ Rc , as ( f |v ) ( t ) = Nb∑ i=1 wiφi ( t ) + M∑ j=1 qjk ( t , zj ) , ( 4 ) 1This is shown in Appendix A of the supplemental material . where { φi } Nbi=1 is a random Fourier basis , with Nb being the number of basis functions in the approximation , w ∈ RNb with entries wi ∼ N ( 0 , 1 ) , and qj are the entries of the vector q = K−1 ( v − Φw ) , where K ∈ RM×M , with elements Km , n = k ( zn , zm ) , is the covariance matrix of the inducing points , and Φ ∈ RM×Nb is a feature matrix , with each basis function being evaluated at each inducing location . The random Fourier basis is obtained by first sampling βi ∼ U ( 0 , 2π ) , where U is the uniform distribution , and then sampling θi ∼ FT ( k ) , where FT is the Fourier transform of the covariance function , which is the spectral density of the process . The basis functions are then given by φi ( t ) = √ 2/Nb cos ( θ > i t + βi ) . We can see Equation ( 4 ) as consisting of an approximate GP prior , using a random Fourier features approximation [ 21 ] , with a correction term which uses Matheron ’ s update rule to account for the inducing points . By using Equation ( 4 ) , samples can be obtained in linear time with respect to the number of requested points . It should be noted that Equation ( 4 ) only applies to GPs with stationary kernels , however , the DSE covariance required for Gc , d , is non-stationary . It can be shown that the process exp ( −α||t||2 ) G′c , d ( t ) has the DSE covariance if G′c , d has the SE covariance . We can then write a sample from the output of the NVKM as , ( fd| { vGd , c } Cc=1 , vu ) ( t ) = C∑ c=1 ∫ ∞ −∞ e−α ∑c i=1 ( t−τi ) 2 ( G′d , c|vGd , c ) ( t− τ1 , . . . , t− τc ) c∏ j=1 ( u|vu ) ( τj ) dτj , ( 5 ) which can be computed analytically , assuming that u also has the SE covariance , by representing the Fourier basis in complex form , and factorising the integrals , leading to combinations of sums and products of single dimensional integrals of the form ∫∞ −∞ exp ( −ax 2 + bx ) dx = √ π/a exp ( b2/4a ) . See Appendix B in the supplemental material for details of the computation .

Summary: The paper places GP priors over the kernels of a Volterra series which is driven by an input process drawn from a GP, which may optionally have associated observations (the IO variant of the model). Sampling and mean field variational inference procedures are described. The method is demonstrated on two real data sets.