Compositional Modeling of Nonlinear Dynamical Systems with ODE-based Random Features

1 Introduction . Dynamical systems are ubiquitous across the natural sciences , with many physical and biological processes being driven on a fundamental level by differential equations . Inferring ordinary differential equation ( ODE ) parameters using observational data from such systems is an active area of research [ Meeds et al. , 2019 , Ghosh et al. , 2021 ] , however , in particularly complex systems it is often infeasible to characterise all of the individual processes present and the interactions between them . Rather than attempt to fully describe a complex system , latent force models ( LFMs ) [ Alvarez et al. , 2009 ] specify a simplified mechanistic model of the system which captures salient features of the dynamics present . This leads to a model which is able to readily extrapolate beyond the training input space , thereby retaining one of the key advantages of mechanistic modeling over purely data-driven techniques . Modeling nonlinear dynamical systems presents an additional challenge , and whilst nonlinear differential equations have been incorporated into LFMs [ Ward et al. , 2020 ] , shallow models such as LFMs are generally less capable than deep models of modeling the non-stationarities often present in nonlinear systems . Deep model architectures have greater representational power than shallow models as a result of their hierarchical structure , which typically consists of compositions of functions [ LeCun et al. , 2015 ] . Deep probabilistic models such as deep Gaussian processes ( DGPs ) [ Damianou and Lawrence , 2013 ] are able to leverage this representational power , with the additional advantage of being able to accurately quantify uncertainty . In this paper , we aim to model nonlinear dynamics by constructing a DGP from compositions of physics-informed random Fourier features , with the motivation behind this approach being that many real-world systems are compositional hierarchies [ LeCun et al. , 2015 ] . Whilst a number of recent works have incorporated random Fourier features into deep models [ Cutajar et al. , 2017 , Mehrkanoon and Suykens , 2018 ] , the only work we are aware of which does so in the context of a physics-informed DGP is that of Lorenzi and Filippone [ 2018 ] . The authors impose physical structure on a DGP with 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . random features by constraining the function values within the model as a means of performing ODE parameter inference . Rather than following the approach of Lorenzi and Filippone [ 2018 ] and placing constraints on function values whilst using an exponentiated quadratic ( EQ ) kernel , we instead integrate a physics-informed prior into the structure of the DGP by utilising a kernel based on an ODE . Our main contribution in this paper is the introduction of a novel approach to incorporating physical structure into a deep probabilistic model , whilst providing a sound quantification of uncertainty . We achieve this through derivation of physics-informed random Fourier features via the convolution of an EQ GP prior with the Green ’ s function associated with a first order ODE . These features are then incorporated into each layer of a DGP , as shown in Figure 1 . To ensure the scalability of our model to large datasets , stochastic variational inference is employed as a method for approximate Bayesian inference . We provide experimental evidence that our modeling framework is capable of capturing highly nonlinear dynamics effectively in both toy examples and real world data , whilst also being applicable to more general regression problems . 2 Background . In this section , we review the theory behind using random Fourier features for deep Gaussian processes . We also review the idea of building covariance functions for GPs that encode the dynamics of ODEs and how to compute those covariances using random Fourier features . 2.1 Deep Gaussian Processes with Random Feature Expansions . Gaussian processes ( GPs ) are non-parametric probabilistic models which offer a great degree of flexibility in terms of the data they can represent , however this flexibility is not unbounded . The hierarchical networks of GPs now widely known as deep Gaussian processes ( DGPs ) were first formalised by Damianou and Lawrence [ 2013 ] , with the motivating factor behind their creation being the ability of deep networks to reuse features and allow for higher levels of abstraction in said features [ Bengio et al. , 2013 ] , which results in such models having more representational power and flexibility than shallow models such as GPs . DGPs are effectively a composite function , where the input to the model is transformed to the output by being passed through a series of latent mappings ( i.e . multivariate GPs ) . If we consider a supervised learning problem with inputs denoted by X = { xn } Nn=1 and targets denoted by y = { yn } Nn=1 , we can write the analytic form of the marginal likelihood for a DGP with Nh hidden layers as p ( y|X , θ ) = ∫ p ( y|F ( Nh ) ) p ( F ( Nh ) |F ( Nh−1 ) , θ ( Nh−1 ) ) ... p ( F ( 1 ) |X , θ ( 0 ) ) dF ( Nh ) ... dF ( 1 ) , where F ( ` ) and θ ( ` ) represent the latent values and covariance parameters respectively at the ` -th layer , where ` = 0 , ... , Nh . However , due to the need to propagate densities through non-linear GP covariance functions within the model , this integral is intractable [ Damianou , 2015 ] . As this precludes us from employing exact Bayesian inference in these models , various techniques for approximate inference have been applied to DGPs in recent years , with most approaches broadly based upon either variational inference [ Salimbeni and Deisenroth , 2017 , Salimbeni et al. , 2019 , Yu et al. , 2019 ] or Monte Carlo methods [ Havasi et al. , 2018 ] . Cutajar et al . [ 2017 ] outline an alternative approach to tackling this problem , which involves replacing the GPs present at each layer of the network with their two layer weight-space approximation , forming a Bayesian neural network which acts as an approximation to a DGP . If the ` -th layer of such a DGP ( consisting of zero-mean GPs with exponentiated quadratic kernels ) receives an input F ( ` ) ( where F ( 0 ) = X ) , the random features for this layer are denoted by Φ ( ` ) ∈ RN×2N ( ` ) RF ( where N ( ` ) RF denotes the number of random features used ) , and are given by , Φ ( ` ) = √ ( σ2 ) ( ` ) N ( ` ) RF [ cos ( F ( ` ) Ω ( ` ) ) , sin ( F ( ` ) Ω ( ` ) ) ] , ( 1 ) where ( σ2 ) ( ` ) is the marginal variance kernel hyperparameter , N ( ` ) RF is the number of random features used and Ω ( ` ) ∈ RDF ( ` ) ×N ( ` ) RF is the matrix of spectral frequencies used to determine the random features . This matrix is assigned a prior p ( Ω ( ` ) d ) = N ( Ω ( ` ) d | 0 , ( l ( ` ) ) −2 ) where l ( ` ) is the lengthscale kernel hyperparameter , DF ( ` ) is the number of GPs within the layer , and d = 1 , ... , DF ( ` ) . The random features then undergo the linear transformation , F ( ` +1 ) = Φ ( ` ) W ( ` ) , where W ( ` ) ∈ R2N ( ` ) RF×DF ( ` +1 ) is a weight matrix with each column assigned a standard normal prior . Training is achieved via stochastic variational inference , which involves establishing a tractable lower bound for the marginal likelihood and optimising said bound with respect to the mean and variance of the variational distributions over the weights and spectral frequencies across all layers of the network . The bound is also optimised with respect to the kernel hyperparameters across all layers . 2.2 Random Feature Approximations for Latent Force Models . In this paper , we take a similar approach to Cutajar et al . [ 2017 ] , but we utilise physically-inspired random features . Physics-inspired covariance functions for GPs have been proposed by [ Alvarez et al. , 2009 ] under the name of latent force models , and more recently , the authors of Guarnizo and Álvarez [ 2018 ] studied the construction of random features related to such covariance functions . Latent force models ( LFMs ) [ Alvarez et al. , 2009 ] are GPs which incorporate a physically-inspired kernel function , which typically encodes the behaviour described by a specific form of differential equation . Instead of taking a fully mechanistic approach and specifying all the interactions within a physical system , the kernel describes a simplified form of the system in which the behaviour is determined byQ latent forces . Given an input data-point t and a set ofD output variables { fd ( t ) } Dd=1 , a LFM expresses each output as fd ( t ) = ∑Q q=1 Sd , q ∫ t 0 Gd ( t− τ ) uq ( τ ) dτ , where Gd ( · ) represents the Green ’ s function for a certain form of linear differential equation , uq ( t ) ∼ GP ( 0 , kq ( t , t′ ) ) represents the GP prior over the the q-th latent force , and Sd , q is a sensitivity parameter weighting the influence of the q-th latent force on the d-th output . The general expression for the covariance of a LFM is given by kfd , fd′ ( t , t ′ ) = ∑Q q=1 Sd , qSd′ , q ∫ t 0 Gd ( t− τ ) ∫ t′ 0 Gd′ ( t ′ − τ ′ ) kq ( τ , τ ′ ) dτ ′dτ . Typically , an EQ form is assumed for the kernel governing the latent forces , kq ( · ) . Due to the linear nature of the convolution used to compute fd ( t ) , the outputs can also be described by a GP . Exact inference in LFMs is tractable , however as with GPs , it scales with O ( N3 ) complexity [ Rasmussen and Williams , 2006 ] . Furthermore , computing the double integration for kfd , fd′ ( t , t ′ ) leads to terms that include the numerical approximation of complex functions which burden the computation of the kernel function . By providing a random Fourier feature representation for the EQ kernel kq ( t , t′ ) , Guarnizo and Álvarez [ 2018 ] were able to reduce this cubic dependence on the number of data-points to a linear dependence . This representation arises from Bochner ’ s theorem , which states , kq ( τ , τ ′ ) = e − ( τ−τ′ ) 2 ` 2q = ∫ p ( ω ) ej ( τ−τ ′ ) ωdω ≈ 1 NRF NRF∑ s=1 ejωsτe−jωsτ ′ , ( 2 ) where ` q is the lengthscale of the EQ kernel , NRF is the number of random features used in the approximation and ωs ∼ p ( ω ) = N ( ω|0 , 2/ ` 2q ) . Substituting this form of the EQ kernel into kfd , fd′ ( t , t ′ ) leads to a fast approximation to the LFM covariance kfd , fd′ ( t , t ′ ) ≈ ∑Q q=1 Sd , qSd′ , q NRF [ ∑NRF s=1 φd ( t , θd , ωs ) φ ∗ d′ ( t ′ , θd′ , ωs ) ] , where φd ( t , θd , ω ) = ∫ t 0 Gd ( t− τ ) ejωτdτ , ( 3 ) with θd representing the Green ’ s function parameters . When the Green ’ s function is a real function , φ∗d′ ( t ′ , θd′ , ω ) = φd′ ( t ′ , θd′ , −ω ) . Guarnizo and Álvarez [ 2018 ] refer to φd ( t , θd , ω ) as random Fourier response features ( RFRFs ) .

Summary: This manuscript describes Deep Latent Force Models (DLFM), a nonlinear dynamical system with nicely calibrated uncertainty quantification. Similar to deep Gaussian processes (DGPs), the presented framework stacks layers of GP layers with physics-inspired covariance functions following ordinary differential equations (ODEs). The inference is based on the weight space approximation of GPs that maintains variational distributions over the weights and spectral frequencies. The model outperforms (deep) GPs on time series prediction tasks and performs slightly better in regression tasks.