Neural graphical modelling in continuous-time: consistency guarantees and algorithms

1 INTRODUCTION . This paper deals with learning directed graphs from a combination of temporal data and assumptions on the parameterization of the underlying structural dynamical system . Graphical models can offer a parsimonious , interpretable representation of the dynamics of stochastic processes , and have proven to be especially useful in problems involving complex systems , non-linear associations and chaotic behaviour that are characteristic in a wide array of applications in biology ( Trapnell et al. , 2014 ; Qiu et al. , 2017 ; Bracco et al. , 2018 ; Raia , 2008 ; Qian et al. , 2020 ) , neuroscience ( Friston et al. , 2003 ; Friston , 2009 ) and climate science ( Runge , 2018 ; Runge et al. , 2019 ) . In these contexts , inferring graphical models from temporal data subject to practical limitations as to how finely and regularly each variable can be measured over time is a longstanding challenge . Time series data is often assumed to be a sequence of observations from an underlying process evolving continuously in time . This underlying representation is fundamental to define the semantics of dependencies between sequences . Time defines an asymmetry between dependencies in dynamical systems , distinguishing between local , direct dependencies that occur over infinitesimal time intervals not mediated by other variables in the system and indirect dependencies that necessarily occur over longer time frames . In many applications , the underlying structural model is formalized as the state of random variables ( e.g . x ( t ) ∈ Rd ) contemporaneously influencing the rate of change of the same or other variables ( e.g . dx ( t ) ) , dx ( t ) = f ( x ( t ) ) dt+ dw ( t ) , x ( 0 ) = x0 , t ∈ [ 0 , T ] , ( 1 ) ∗Work primarily conducted while at the University of Cambridge and at the Alan Turing Institute . where w ( t ) a d-dimensional standard Brownian motion and x0 is a Gaussian random variable independent of w ( t ) . The functional dependence structure of the vector field f defines a directed graph G and associated adjacency matrix G ∈ { 0 , 1 } d×d , i.e. , Gij = 1 if and only if xj appears as an argument of fi = [ f ] i . The problem of structure learning is to search over the space of graphs compatible with the data , but the pattern of observation in dynamical systems emphasize a number of differences with respect to classical graphical modelling with static data or explicitly discrete-time stochastic process . • Observed data is sampled at a sequence of ( often irregular ) time points ( t1 , . . . , tn ) and is systematically subsampled . Most work on graphical modelling with time series data assume a fundamentally discrete parameterization of the underlying structural model ( e.g . based on vector autoregression models ) . Associations in discrete-time in general do not correspond to the structure of the underlying dynamical system and are highly dependent on the interval between observations . The same subsampled discrete model may disaggregate to several continuous models , which are observationally equivalent at the subsampled frequency , see e.g . ( Runge , 2018 ; Gong et al. , 2015 ; Danks & Plis , 2013 ) and a worked example in Appendix A . The realm of problems that involve irregularlysampled data are fundamentally out of scope in discrete-time in general . We complement this point in Figure 1 with an illustration of our performance results comparing a state of the art discrete-time graphical modelling method with our proposed continuous-time counterpart that is shown to be more accurate and more robust to the frequency and irregularity of sampling . • Discrete samples are not independent which can ( and does ) increase the sample complexity . An increasing sample frequency will produce an increasing number of distinct samples . However , samples become more dependent , and intuitively one expects that there is limited information to be harnessed from a given time interval [ 0 , T ] . Learning performance depends on the number of independent samples which is a function both of the number of samples n and the length of the observed interval T . • Non-parametric graphical modelling in dynamical systems is relatively unexplored . Existing approaches rely on specific model assumptions ( e.g . linearity , additivity ) to establish the consistency of structure recovery even though flexible model families , such as neural networks , are increasingly used in related problems such as feature selection and graphical modelling with static data . In addition , consistent derivative approximations are typically required for consistency arguments which in practice involve choices on the smoothness of the interpolated curve and makes two-step strategies far from automatic and applicable in general dynamical systems . Contributions . This paper establishes the consistency of score-based recovery of G when an analytic deep neural network model is imposed for f ( such as feed-forward networks with multiple hidden layers and convolutional neural networks ) under general observation patterns including irregular sampling . In particular , we consider penalized optimization problems of the form , arg min fθ 1 n n∑ i=1 ||x ( ti ) − x̂ ( ti ) ) ||22 , subject to ρn , T ( fθ ) ≤ η and dx̂ ( t ) = fθ ( x̂ ( t ) ) dt , ( 2 ) where the observation process ( x ( t1 ) , . . . , x ( tn ) ) is given by an irregular sequence of time points 0 ≤ t1 < · · · < tn ≤ T . ρn , T ( fθ ) is an adaptive group lasso constraint on the parameter space of fθ . We analyze this problem with fixed dimension d and increasing sample size n and horizon T – the sample complexity of this problem depending both on the frequency of sampling n as well as on the time horizon T . A second contribution is to propose an instantiation of this method using differential equations with vector fields parameterized by neural networks ( Chen et al. , 2018 ) to model the mean process of ( 1 ) with the advantage of implicitly inferring variable derivatives instead of involving a separate approximation step ( that is common in the dynamical systems literature ) . This construction shows that , empirically , graphical models in continuous-time can be inferred accurately in a large range of settings despite irregularly-sampled multivariate time series data and non-linear underlying dependencies . Code associated with this work may be found at https : //github.com/alexisbellot and at https : //github.com/vanderschaarlab/mlforhealthlabpub . 2 RELATED WORK . A substantial amount of work devoted to graphical modelling has considered the analysis of penalized least squares and its variants , most prominently in the high-dimensional regression literature with i.i.d data , see e.g . ( Friedman et al. , 2008 ; Zou , 2006 ; Zhao & Yu , 2006 ) . Closely related to our results are a number of extensions that have considered parameter identification in neural networks , using for instance a sparse one-to-one linear layers ( Li et al. , 2016 ) , group lasso constraints of the input layer of parameters ( Zhang et al. , 2019 ) and input to output residual connections ( Lemhadri et al. , 2021 ) . For a large class of neural networks Dinh & Ho ( 2020 ) proved the consistency of adaptive regularization methods . The distinction with our formalism in ( 2 ) is that the observations are not corrupted by i.i.d . noise ( since successive samples are correlated ) and therefore standard concentration inequalities are not sufficient . Learning graphical models with dependent noise terms is also a topic of significant literature in the context of Granger causality , proposed by Granger ( 1969 ) and also popularized by Sims ( 1980 ) within autoregressive models . Various authors have considered the consistency of penalized vector autoregression models and proposed tests of Granger causality using parameter estimates in these models , see e.g . ( Nardi & Rinaldo , 2011 ; Kock & Callot , 2015 ; Adamek et al. , 2020 ; Chernozhukov et al. , 2019 ) , and extended some of these approaches to models of neural networks , see e.g . ( Tank et al. , 2018 ; Khanna & Tan , 2019 ; Marcinkevičs & Vogt , 2021 ) ( without however proving consistency of inference ) . Methods exist also using conditional independence tests such as those given by Runge et al . ( 2017 ) and transfer entropy principles originating in Schreiber ( 2000 ) . The conceptual and statistical contrasts between discrete and continuous accounts of the underlying structural model are substantial and are discussed in the Appendix A . In the context of differential equations , penalized regression has been explored using two-stage collocation methods , first proposed by Varah ( 1982 ) , by which derivatives are estimated on smoothed data and subsequently regressed on observed samples for inference . The consistency of parameter estimates has been established for linear models in parameters , as done for example in ( Ramsay et al. , 2007 ; Chen et al. , 2017 ; Wu et al. , 2014 ; Brunton et al. , 2016 ) . From a modelling perspective , our approach in contrast is end-to-end , coupling the estimation of the underlying paths x and the vector field f . Graphical modelling has also been considered for linear stochastic differential equations by Bento et al . ( 2010 ) . Similarly to the discrete-time literature , proposals exist for recovering non-linear vector fields via neural networks ( see e.g . ( Raissi et al. , 2017 ; Bellot & van der Schaar , 2021 ) ) and Gaussian processes ( see e.g . ( Heinonen et al. , 2018 ; Wenk et al. , 2020 ) ) but we are not aware of any identifiability guarantees . 3 GRAPHICAL MODELLING IN CONTINUOUS-TIME . We consider the underlying structure of an evolving process to be described by a multivariate dynamical system of d distinct stochastic processes x = ( x1 , . . . , xd ) : [ 0 , T ] → X d with each instantiation in time xj ( t ) for j = 1 , . . . , d and t > 0 defined in a bounded open set X ⊂ R. Definition 1 ( Neural Dynamic Structural Model ( NDSM ) ) . We say that x = ( x1 , . . . , xd ) : [ 0 , T ] → X d follows a Neural Dynamic Structural Model if there exist functions f1 , . . . , fd ∈ F such that fj : X d → R and , dxj ( t ) = fj ( x ( t ) ) dt+ dwj ( t ) , x ( t0 ) = x0 , t ∈ [ 0 , T ] , ( 3 ) with F defined as the space of analytic feed-forward neural networks with sets of parameters θ ∈ Θ defined in bounded , real-valued intervals and wj ( t ) is standard Brownian motion independently generated across processes j1 . We will write fθ0 = ( f1 , . . . , fd ) for the true underlying vector field , parameterized by a set of parameter values θ0 . It will be useful to define each layer of each network precisely . Let A j 1 ∈ Rd×h denote the d × h weight matrix ( we omit biases for clarity ) in the input layer of fj , j = 1 , . . . , d. Let Ajm ∈ Rh×h , for m = 2 , . . . , M − 1 , denote the weight matrix of each hidden layer , and let AjM ∈ Rh×1 be the h× 1 dimensional output layers of each sub-network such that , fj ( X ) : = φ ( · · ·φ ( φ ( XAj1 ) A j 2 ) · · · ) A j M , j = 1 , . . . , d , ( 4 ) where φ ( · ) is an analytic activation function ( e.g . tanh , sigmoid , arctan , softplus , etc . ) and X ∈ Rn×d is the sequence of n d-dimensional instantiations of x . Assumption 1 ( Observation process ) . The data in practice , is a partial sequence of observations of x at n time points ( t1 , . . . , tn ) sampled from a temporal point process with positive intensity such that , ( x1 , . . . , xn ) ∼ N ( µ , Σn ) , ( 5 ) with a dependency structure encoded in Σn ∈ Rn×n . The closer in time two observations are the more closely correlated we can expect them to be . We assume the data to be normalized , i.e . diagonal elements of Σn to be equal to 1. µ are the instantiations of the mean process that can be described by an system of ordinary differential equations dx ( t ) = f ( x ( t ) ) dt , x ( 0 ) = x0 , t ∈ [ 0 , T ] . Time points at which observations are made are thus themselves assumed stochastic , driven by an independent temporal point process with intensity limdt→0 Pr ( Observation in [ t , t + dt ] |Ht ) > 0 for any t > 0 with respect to a filtrationHt that denotes sigma algebras generated by any sequence of prior observations . Perfectly homogeneous and systematic subsampling has measure zero under this probability model . This is important because it will enable , in principle , to infer local conditional independencies ( defined below ) arbitrarily well with increasing sample size .

Summary: This paper introduces a brand new graphical modeling framework from the perspective of neural ODEs. Traditionally structure learning involves using sampled data to learn the structure of graphs. This paper, however, looks at the graph structure learning problem from a different viewpoint, using continuous-time dynamics inspired from neural ODEs. Theoretical guarantees on parameter estimation are provided. Some experiments on benchmark time-series datasets are also conducted.