Learning Nonparametric Volterra Kernels with Gaussian Processes

1 Introduction . Gaussian processes ( GPs ) constitute a general method for placing prior distributions over functions , with the properties of samples from the distribution being controlled primarily by the form of the covariance function [ 22 ] . Process convolutions ( PCs ) are one powerful method for building such covariance functions [ 4 , 13 , 3 ] . In the PC framework , the function we wish to model is assumed to be generated by the application of some convolution operator to a base GP with some simple covariance , and since linear operators applied to GPs result in GPs , the result is another GP with a covariance we deem desirable . PCs allow models for multiple correlated output functions to be built with ease , by assuming each output is generated by a different operator applied to the same base function , or set of functions [ 30 , 13 ] . The PC framework unifies a number of different ideas in the GP literature . Latent force models ( LFMs ) [ 1 ] use PCs to include physics based inductive biases in multiple output GP ( MOGP ) models by using the Green ’ s function of a linear differential operator as the kernel of the convolution . This leads to the interpretation of each output as having been generated by inputting a random latent force into a linear system , with physical properties described by the differential operator . The Gaussian process convolution model ( GPCM ) of Tobar et al . [ 29 ] treats the convolution kernel itself as an unknown function to be inferred from data , and places a GP prior over it . Linear systems are entirely 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . described by their Green ’ s function , so one can interpret the GPCM as a nonparametric , linear LFM , in which the form of the system itself is inferred from data . In the physical world , nonlinear systems are the norm , and linearity is an approximation . As a consequence it is desirable when dealing with physical data to have models that can incorporate nonlinearity naturally . Often however , given a certain set of data , it is not clear exactly what form this nonlinearity takes , and so introducing specific parametric nonlinear operators can be overly restrictive . Álvarez et al . [ 2 ] present a model known as the nonlinear convolved MOGP ( NCMOGP ) which introduces nonlinearity to MOGPs via the Volterra series [ 8 ] , a nonlinear series expansion used widely for systems identification , whose properties are controlled by a set of square integrable functions of increasing dimensionality known as the Volterra kernels ( VKs ) . The NCMOGP assumes these functions are both separable and homogeneous , and of parametric Gaussian form , it also approximates the outputs as GPs in order to make inference tractable . The present work introduces a new model which drops the separability and homogeneity assumptions on the VKs , allows their form to be learned directly from data , and makes no approximation on the distribution of the outputs . We refer to it as the nonparametric Volterra kernels model ( NVKM ) . We develop a fast sampling method for the NVKM which leverages the recent results of Wilson et al . [ 35 ] on the sampling of explicit functions from GPs to analytically map function realisations through the Volterra series , avoiding the need for computationally expensive and inaccurate high dimensional numerical integration . Fast sampling allows for the application of doubly stochastic variational inference ( DSVI ) [ 28 ] for scalable learning . The NVKM is well suited to both single and multiple output regression problems , and can be thought of as an extension of the GPCM to both nonlinear systems and multiple outputs . The NVKM can also be interpreted as a nonlinear LFM in which the operator is learned directly from data . We additionally present a variation of the NVKM that can be used for Bayesian systems identification , where the task is to learn operator mappings between observed input and output data , and show that it allows for considerably better quantification of uncertainty than competing methods which use recurrence [ 18 ] . 2 Background . In this section we give a brief introduction to the mathematical background of PCs and the Volterra series . Process convolutions In the PC framework , the set of output functions { fd ( t ) } Dd=1 , with D being the number of outputs , is generated by the application of some set of linear operators , specifically convolution operators , to a latent function u represented by a Gaussian process , fd ( t ) = ∫ T Gd ( t− τ ) u ( τ ) dτ , where T is the domain of integration , and the function Gd is known variously as the convolutional kernel , smoothing kernel , impulse response or Green ’ s function , depending on context . Assuming that the input u is bounded , the function Gd must be square integrable to ensure the output is finite . A linear operator acting on a GP produces another GP [ 22 ] , and so we obtain D distinct GPs . Since the latent function u is shared across the outputs , these D GPs are correlated , allowing joint variations to be captured , whilst the convolution with Gd adapts u to each output . Álvarez et al . [ 3 ] show that many MOGP models can be recast in terms of the PC framework by particular choices of Gd and u . In LFMs , Gd is taken to be the Green ’ s function of some differential operator . We can then interpret each output as resulting from a shared random force being fed into a distinct linear system , represented by the differential operator . The smoothing kernels are usually taken to have parametric , often Gaussian form . Tobar et al . [ 29 ] use the PC framework for a single output , and make the smoothing kernel itself a GP , giving rise to the GPCM . Bruinsma [ 7 ] extends the GPCM to the multiple output case , although the model was not applied to data . Volterra Series The PC framework can be extended to represent a broader class of output functions by instead considering the outputs { fd ( t ) } Dd=1 as being the result of some nonlinear system , acting on the latent function u . The Volterra series is a series approximation for nonlinear systems that is widely used in the field of system identification [ 8 ] . It is given by , f ( t ) = C∑ c=1 ∫ T Gc ( t− τ1 , . . . , t− τc ) c∏ j=1 u ( τj ) dτj , ( 1 ) where f is the system output , Gc is the cth order VK , u is the system input , and C is the order of the approximation . We can think of the Volterra series as an extension of the well known Taylor expansion , that allows f to have memory of the past values of u , that is to say f depends on u at all values t ∈ T . The Volterra series can approximate a broad class of nonlinear operators , however it is unable to represent certain properties of general nonlinear systems , for example chaos . Álvarez et al . [ 2 ] use Equation ( 1 ) to construct the NCMOGP , applying the Volterra series with d distinct sets of VKs { Gd , c } Cc=1 to a shared latent GP input u , to produce outputs { fd ( t ) } Dd=1 . Since the Volterra series is a nonlinear operator , the output becomes an intractable , non-Gaussian process . The authors perform inference by approximating the outputs as GPs and using the first and second moments of the output process to form its mean and covariance function . To enable to computation of these moments , the authors restrict the set of VKs to those which are both separable and homogeneous , i.e . Gd , c ( t1 , . . . , tc ) = ∏c i=1Gd ( ti ) . Additionally , since the moment computation requires analytically solving a number of non-trivial convolution integrals , the authors only consider a Gaussian form for the VKs . 3 The nonparametric Volterra kernels model . The NVKM relaxes the restrictions of separability and homogeneity which are placed on the VKs in the NCMOGP , and represents these kernels as independent GPs , allowing their form and uncertainty to be inferred directly from data . The generative process for the NVKM can be stated as u ( t ) ∼ GP ( 0 , k ( u ) ( t , t′ ) ) , Gd , c ( t ) ∼ GP ( 0 , k ( Gd , c ) ( t , t′ ) ) , t ∈ Rc , ∀c ∈ 1 , . . . , C , ∀d ∈ 1 , . . . , D , fd ( t ) = C∑ c=1 ∫ ∞ −∞ Gd , c ( t− τ1 , . . . , t− τc ) c∏ j=1 u ( τj ) dτj , ( 2 ) where k ( u ) ( t , t′ ) is the covariance function for the input process , and k ( Gd , c ) ( t , t′ ) is the covariance function for the cth VK of the dth output . We follow Tobar et al . [ 29 ] in using the decaying square exponential ( DSE ) covariance for the VKs , which is a modification to the ubiquitous square exponential ( SE ) covariance that ensures the samples have finite energy.1 The DSE covariance has the form kDSE ( t , t ′ ) = σ2 exp ( −α ( ‖t‖2 + ‖t′‖2 ) − γ‖t− t′‖2 ) , ( 3 ) where ‖ · ‖ is the or ` 2 norm , σ is the amplitude , α controls the rate at which the samples decay away from the origin , and γ is related to the length scale l of the samples by γ = 1l2 . A diagram of the generative process for the model is shown in Figure 1 . Obtaining an exact distribution over the outputs fd is intractable , since it involves integration over nonlinear combinations of infinite dimensional stochastic processes . In order to sample from the model , and perform inference , approximations must be introduced . In particular , we employ the results of Wilson et al . [ 35 ] to sample in linear time , which enables efficient learning through the use of variational inducing points [ 27 ] with doubly stochastic variational inference ( DSVI ) [ 28 ] . 3.1 Sampling . One could sample from the model by drawing from the input and filter GPs at some finite set of locations , and then using the samples with some method for numerical integration to find the output . As the dimensionality of the filters increases , however , many points would be needed to obtain an accurate answer , this quickly becomes computationally intractable , since sampling exactly from a GP has cubic time complexity with respect to the number of points requested . We can sidestep this problem , and avoid the need for any numerical integration , by representing samples from the GPs explicitly as functions . Using the results from Wilson et al . [ 35 ] , and following their notation , we can write a sample from a GP f : Rc −→ R , with covariance function k ( t , t′ ) , given M inducing variables v ∈ RM with corresponding inducing inputs { zj } Mj=1 , with zj ∈ Rc , as ( f |v ) ( t ) = Nb∑ i=1 wiφi ( t ) + M∑ j=1 qjk ( t , zj ) , ( 4 ) 1This is shown in Appendix A of the supplemental material . where { φi } Nbi=1 is a random Fourier basis , with Nb being the number of basis functions in the approximation , w ∈ RNb with entries wi ∼ N ( 0 , 1 ) , and qj are the entries of the vector q = K−1 ( v − Φw ) , where K ∈ RM×M , with elements Km , n = k ( zn , zm ) , is the covariance matrix of the inducing points , and Φ ∈ RM×Nb is a feature matrix , with each basis function being evaluated at each inducing location . The random Fourier basis is obtained by first sampling βi ∼ U ( 0 , 2π ) , where U is the uniform distribution , and then sampling θi ∼ FT ( k ) , where FT is the Fourier transform of the covariance function , which is the spectral density of the process . The basis functions are then given by φi ( t ) = √ 2/Nb cos ( θ > i t + βi ) . We can see Equation ( 4 ) as consisting of an approximate GP prior , using a random Fourier features approximation [ 21 ] , with a correction term which uses Matheron ’ s update rule to account for the inducing points . By using Equation ( 4 ) , samples can be obtained in linear time with respect to the number of requested points . It should be noted that Equation ( 4 ) only applies to GPs with stationary kernels , however , the DSE covariance required for Gc , d , is non-stationary . It can be shown that the process exp ( −α||t||2 ) G′c , d ( t ) has the DSE covariance if G′c , d has the SE covariance . We can then write a sample from the output of the NVKM as , ( fd| { vGd , c } Cc=1 , vu ) ( t ) = C∑ c=1 ∫ ∞ −∞ e−α ∑c i=1 ( t−τi ) 2 ( G′d , c|vGd , c ) ( t− τ1 , . . . , t− τc ) c∏ j=1 ( u|vu ) ( τj ) dτj , ( 5 ) which can be computed analytically , assuming that u also has the SE covariance , by representing the Fourier basis in complex form , and factorising the integrals , leading to combinations of sums and products of single dimensional integrals of the form ∫∞ −∞ exp ( −ax 2 + bx ) dx = √ π/a exp ( b2/4a ) . See Appendix B in the supplemental material for details of the computation .

Summary: The article proposes a methodology for designing covariances kernels in a nonparametric way using Volterra series. The ideas proposed build on existing works on convolution models, Volterra kernels, SysID and MOGP. The paper substantially differ from previous works and the contribution is validated empirically.