Neural graphical modelling in continuous-time: consistency guarantees and algorithms

1 INTRODUCTION . This paper deals with learning directed graphs from a combination of temporal data and assumptions on the parameterization of the underlying structural dynamical system . Graphical models can offer a parsimonious , interpretable representation of the dynamics of stochastic processes , and have proven to be especially useful in problems involving complex systems , non-linear associations and chaotic behaviour that are characteristic in a wide array of applications in biology ( Trapnell et al. , 2014 ; Qiu et al. , 2017 ; Bracco et al. , 2018 ; Raia , 2008 ; Qian et al. , 2020 ) , neuroscience ( Friston et al. , 2003 ; Friston , 2009 ) and climate science ( Runge , 2018 ; Runge et al. , 2019 ) . In these contexts , inferring graphical models from temporal data subject to practical limitations as to how finely and regularly each variable can be measured over time is a longstanding challenge . Time series data is often assumed to be a sequence of observations from an underlying process evolving continuously in time . This underlying representation is fundamental to define the semantics of dependencies between sequences . Time defines an asymmetry between dependencies in dynamical systems , distinguishing between local , direct dependencies that occur over infinitesimal time intervals not mediated by other variables in the system and indirect dependencies that necessarily occur over longer time frames . In many applications , the underlying structural model is formalized as the state of random variables ( e.g . x ( t ) ∈ Rd ) contemporaneously influencing the rate of change of the same or other variables ( e.g . dx ( t ) ) , dx ( t ) = f ( x ( t ) ) dt+ dw ( t ) , x ( 0 ) = x0 , t ∈ [ 0 , T ] , ( 1 ) ∗Work primarily conducted while at the University of Cambridge and at the Alan Turing Institute . where w ( t ) a d-dimensional standard Brownian motion and x0 is a Gaussian random variable independent of w ( t ) . The functional dependence structure of the vector field f defines a directed graph G and associated adjacency matrix G ∈ { 0 , 1 } d×d , i.e. , Gij = 1 if and only if xj appears as an argument of fi = [ f ] i . The problem of structure learning is to search over the space of graphs compatible with the data , but the pattern of observation in dynamical systems emphasize a number of differences with respect to classical graphical modelling with static data or explicitly discrete-time stochastic process . • Observed data is sampled at a sequence of ( often irregular ) time points ( t1 , . . . , tn ) and is systematically subsampled . Most work on graphical modelling with time series data assume a fundamentally discrete parameterization of the underlying structural model ( e.g . based on vector autoregression models ) . Associations in discrete-time in general do not correspond to the structure of the underlying dynamical system and are highly dependent on the interval between observations . The same subsampled discrete model may disaggregate to several continuous models , which are observationally equivalent at the subsampled frequency , see e.g . ( Runge , 2018 ; Gong et al. , 2015 ; Danks & Plis , 2013 ) and a worked example in Appendix A . The realm of problems that involve irregularlysampled data are fundamentally out of scope in discrete-time in general . We complement this point in Figure 1 with an illustration of our performance results comparing a state of the art discrete-time graphical modelling method with our proposed continuous-time counterpart that is shown to be more accurate and more robust to the frequency and irregularity of sampling . • Discrete samples are not independent which can ( and does ) increase the sample complexity . An increasing sample frequency will produce an increasing number of distinct samples . However , samples become more dependent , and intuitively one expects that there is limited information to be harnessed from a given time interval [ 0 , T ] . Learning performance depends on the number of independent samples which is a function both of the number of samples n and the length of the observed interval T . • Non-parametric graphical modelling in dynamical systems is relatively unexplored . Existing approaches rely on specific model assumptions ( e.g . linearity , additivity ) to establish the consistency of structure recovery even though flexible model families , such as neural networks , are increasingly used in related problems such as feature selection and graphical modelling with static data . In addition , consistent derivative approximations are typically required for consistency arguments which in practice involve choices on the smoothness of the interpolated curve and makes two-step strategies far from automatic and applicable in general dynamical systems . Contributions . This paper establishes the consistency of score-based recovery of G when an analytic deep neural network model is imposed for f ( such as feed-forward networks with multiple hidden layers and convolutional neural networks ) under general observation patterns including irregular sampling . In particular , we consider penalized optimization problems of the form , arg min fθ 1 n n∑ i=1 ||x ( ti ) − x̂ ( ti ) ) ||22 , subject to ρn , T ( fθ ) ≤ η and dx̂ ( t ) = fθ ( x̂ ( t ) ) dt , ( 2 ) where the observation process ( x ( t1 ) , . . . , x ( tn ) ) is given by an irregular sequence of time points 0 ≤ t1 < · · · < tn ≤ T . ρn , T ( fθ ) is an adaptive group lasso constraint on the parameter space of fθ . We analyze this problem with fixed dimension d and increasing sample size n and horizon T – the sample complexity of this problem depending both on the frequency of sampling n as well as on the time horizon T . A second contribution is to propose an instantiation of this method using differential equations with vector fields parameterized by neural networks ( Chen et al. , 2018 ) to model the mean process of ( 1 ) with the advantage of implicitly inferring variable derivatives instead of involving a separate approximation step ( that is common in the dynamical systems literature ) . This construction shows that , empirically , graphical models in continuous-time can be inferred accurately in a large range of settings despite irregularly-sampled multivariate time series data and non-linear underlying dependencies . Code associated with this work may be found at https : //github.com/alexisbellot and at https : //github.com/vanderschaarlab/mlforhealthlabpub . 2 RELATED WORK . A substantial amount of work devoted to graphical modelling has considered the analysis of penalized least squares and its variants , most prominently in the high-dimensional regression literature with i.i.d data , see e.g . ( Friedman et al. , 2008 ; Zou , 2006 ; Zhao & Yu , 2006 ) . Closely related to our results are a number of extensions that have considered parameter identification in neural networks , using for instance a sparse one-to-one linear layers ( Li et al. , 2016 ) , group lasso constraints of the input layer of parameters ( Zhang et al. , 2019 ) and input to output residual connections ( Lemhadri et al. , 2021 ) . For a large class of neural networks Dinh & Ho ( 2020 ) proved the consistency of adaptive regularization methods . The distinction with our formalism in ( 2 ) is that the observations are not corrupted by i.i.d . noise ( since successive samples are correlated ) and therefore standard concentration inequalities are not sufficient . Learning graphical models with dependent noise terms is also a topic of significant literature in the context of Granger causality , proposed by Granger ( 1969 ) and also popularized by Sims ( 1980 ) within autoregressive models . Various authors have considered the consistency of penalized vector autoregression models and proposed tests of Granger causality using parameter estimates in these models , see e.g . ( Nardi & Rinaldo , 2011 ; Kock & Callot , 2015 ; Adamek et al. , 2020 ; Chernozhukov et al. , 2019 ) , and extended some of these approaches to models of neural networks , see e.g . ( Tank et al. , 2018 ; Khanna & Tan , 2019 ; Marcinkevičs & Vogt , 2021 ) ( without however proving consistency of inference ) . Methods exist also using conditional independence tests such as those given by Runge et al . ( 2017 ) and transfer entropy principles originating in Schreiber ( 2000 ) . The conceptual and statistical contrasts between discrete and continuous accounts of the underlying structural model are substantial and are discussed in the Appendix A . In the context of differential equations , penalized regression has been explored using two-stage collocation methods , first proposed by Varah ( 1982 ) , by which derivatives are estimated on smoothed data and subsequently regressed on observed samples for inference . The consistency of parameter estimates has been established for linear models in parameters , as done for example in ( Ramsay et al. , 2007 ; Chen et al. , 2017 ; Wu et al. , 2014 ; Brunton et al. , 2016 ) . From a modelling perspective , our approach in contrast is end-to-end , coupling the estimation of the underlying paths x and the vector field f . Graphical modelling has also been considered for linear stochastic differential equations by Bento et al . ( 2010 ) . Similarly to the discrete-time literature , proposals exist for recovering non-linear vector fields via neural networks ( see e.g . ( Raissi et al. , 2017 ; Bellot & van der Schaar , 2021 ) ) and Gaussian processes ( see e.g . ( Heinonen et al. , 2018 ; Wenk et al. , 2020 ) ) but we are not aware of any identifiability guarantees . 3 GRAPHICAL MODELLING IN CONTINUOUS-TIME . We consider the underlying structure of an evolving process to be described by a multivariate dynamical system of d distinct stochastic processes x = ( x1 , . . . , xd ) : [ 0 , T ] → X d with each instantiation in time xj ( t ) for j = 1 , . . . , d and t > 0 defined in a bounded open set X ⊂ R. Definition 1 ( Neural Dynamic Structural Model ( NDSM ) ) . We say that x = ( x1 , . . . , xd ) : [ 0 , T ] → X d follows a Neural Dynamic Structural Model if there exist functions f1 , . . . , fd ∈ F such that fj : X d → R and , dxj ( t ) = fj ( x ( t ) ) dt+ dwj ( t ) , x ( t0 ) = x0 , t ∈ [ 0 , T ] , ( 3 ) with F defined as the space of analytic feed-forward neural networks with sets of parameters θ ∈ Θ defined in bounded , real-valued intervals and wj ( t ) is standard Brownian motion independently generated across processes j1 . We will write fθ0 = ( f1 , . . . , fd ) for the true underlying vector field , parameterized by a set of parameter values θ0 . It will be useful to define each layer of each network precisely . Let A j 1 ∈ Rd×h denote the d × h weight matrix ( we omit biases for clarity ) in the input layer of fj , j = 1 , . . . , d. Let Ajm ∈ Rh×h , for m = 2 , . . . , M − 1 , denote the weight matrix of each hidden layer , and let AjM ∈ Rh×1 be the h× 1 dimensional output layers of each sub-network such that , fj ( X ) : = φ ( · · ·φ ( φ ( XAj1 ) A j 2 ) · · · ) A j M , j = 1 , . . . , d , ( 4 ) where φ ( · ) is an analytic activation function ( e.g . tanh , sigmoid , arctan , softplus , etc . ) and X ∈ Rn×d is the sequence of n d-dimensional instantiations of x . Assumption 1 ( Observation process ) . The data in practice , is a partial sequence of observations of x at n time points ( t1 , . . . , tn ) sampled from a temporal point process with positive intensity such that , ( x1 , . . . , xn ) ∼ N ( µ , Σn ) , ( 5 ) with a dependency structure encoded in Σn ∈ Rn×n . The closer in time two observations are the more closely correlated we can expect them to be . We assume the data to be normalized , i.e . diagonal elements of Σn to be equal to 1. µ are the instantiations of the mean process that can be described by an system of ordinary differential equations dx ( t ) = f ( x ( t ) ) dt , x ( 0 ) = x0 , t ∈ [ 0 , T ] . Time points at which observations are made are thus themselves assumed stochastic , driven by an independent temporal point process with intensity limdt→0 Pr ( Observation in [ t , t + dt ] |Ht ) > 0 for any t > 0 with respect to a filtrationHt that denotes sigma algebras generated by any sequence of prior observations . Perfectly homogeneous and systematic subsampling has measure zero under this probability model . This is important because it will enable , in principle , to infer local conditional independencies ( defined below ) arbitrarily well with increasing sample size .

Summary: The paper proposes to learn Jacobian-sparse neural network ODEs from irregular trajectories of a dynamical system. The main contribution is the sparsity of the ODE Jacobian, which results in learning of differential covariate causalities. Learning the differential structure is an important real-world problem. The proposed method is elegant, simple and effective, although incremental.