Title: Graph Neural Flows for Unveiling Systemic Interactions Among Irregularly Sampled Time Series
Venue: NeurIPS 2024
Accepted: yes
Keywords: Graph learning

neural flows

time series
Abstract: Interacting systems are prevalent in nature. It is challenging to accurately predict the dynamics of the system if its constituent components are analyzed independently. We develop a graph-based model that unveils the systemic interactions of time series observed at irregular time points, by using a directed acyclic graph to model the conditional dependencies (a form of causal notation) of the system components and learning this graph in tandem with a continuous-time model that parameterizes the solution curves of ordinary differential equations (ODEs). Our technique, a graph neural flow, leads to substantial enhancements over non-graph-based methods, as well as graph-based methods without the modeling of conditional dependencies. We validate our approach on several tasks, including time series classification and forecasting, to demonstrate its efficacy.

Graph Neural Flows for Unveiling Systemic Interactions Among Irregularly Sampled Time Series

Abstract

Interacting systems are prevalent in nature. It is challenging to accurately predict the dynamics of the system if its constituent components are analyzed independently. Wedevelop a graph-based model that unveils the systemic interactions of time series observed at irregular time points, by using a directed acyclic graph to model the conditional dependencies (a form of causal notation) of the system components and learning this graph in tandem with a continuous-time model that parameterizes the solution curves of ordinary differential equations (ODEs). Our technique, a graph neural flow, leads to substantial enhancements over non-graph-based methods, as well as graph-based methods without the modeling of conditional dependencies. We validate our approach on several tasks, including time series classification and forecasting, to demonstrate its efficacy.

1 Introduction

Real-life dynamical systems consist of a group of components interacting in a complex manner. With time series data for each component, predicting the system dynamics remains challenging, because modeling each component independently is straightforward while accounting for their interactions is hard without a priori knowledge. Sometimes, these interactions are causal. For example, 'phantom jams' in which a small disturbance (e.g., a driver hitting the brake too hard) in a heavy traffic can be amplified over a large area of the transportation network [16]. While traffic congestion often enjoys spatial proximity, outage of a power network can be propagated non-locally over the grid; i.e., a sequence of blackouts jumps across hundreds of kilometers [21]. We pose this question: What time series models best capture the interactive nature of different system dynamics?

Graph neural networks (GNNs) [49, 46] are modern tools to enhance time series models when multiple time series are interconnected by a given graph [33]. In these models, time series are encoded by using a recurrent neural network (RNN); at every time step, a GNN is used to aggregate features over the graph. When the graph is unknown, graph-structure learning approaches have been proposed; some approaches learn a single interaction graph over time [30, 41, 11] while others infer a different one at each time step [20]. These models are in general discrete-time models, suitable for regularly spaced time points [39, 40].

To handle irregular time points, we consider continuous-time models. The celebrated neural ordinary differential equation (ODE) technique [7] models a time series as the solution of an unknown ODE and optimizes the ODE parameters by using gradients computed through the adjoint. Neural ODEs were later adopted for latent variable modeling [39, 5], which introduced discontinuities at observed time points for reducing prediction errors and variances. Neural ODEs were also adopted for graphbased modeling [38, 24, 25, 26, 10, 27, 3], where the equation accounts for multiple time series and the right-hand side uses a graph to associate the different series. These models construct a graph, which could be time-dependent, in various manners, such as based on node features, co-observations within a sliding window, latent representations, or attentions.

In this work, we propose learning a graph that reveals the dependency structure of the time series. To this end, we consider a form of causal notation-the Bayesian network [36, 37]-which is a directed acyclic graph (DAG), where a node is conditionally independent of its non-descendents given its parents [34, 12, 43, 22]. Such a conditional dependence structure specifies how component dynamics depends on each other. This model has a potential for causal discovery when one interprets the learned graph unknown a priori (e.g., how blackouts cascade over the power grid, whose known topology differs from the unknown influence graph [21]). More importantly, when modeled properly, the graph can improve the performance of downstream tasks because of the capturing of systemic interactions.

Our proposed model is a graph neural flow (GNeuralFlow). A neural flow [4] is the learned solution of an unknown ODE based on irregularly sampled time series; it is advantageous over the neural ODE technique in that it models directly the ODE solution rather than the right-hand side, thus avoiding repeating calls of a numerical solver, whose cost could be expensive. We condition multiple neural flows, one for each time series, on the DAG, and we instantiate their interactions as a GNN; e.g., a graph convolutional network, GCN [31]. The graph convolution therein augments the parameterization of the ODE solution by aggregating the information of the neighboring time series at each time point, fitting a graph-conditioned ODE that models the interacting system.

Thus, GNeuralFlow is advantageous over prior graph ODE approaches (e.g., GDE [38], LG-ODE [24], CG-ODE [25], CF-GODE [26], STG-NCDE [10], MTGODE [27], RiTINI [3]) in two aspects. First, through learning, the graph reveals the conditional dependencies of the time series, offering a more intuitive structure for analysis. Second, it removes the reliance on numerical ODE solvers and gains computational efficiency. We demonstrate empirical evidence to show that GNeuralFlow outperforms graph ODE approaches in several downstream tasks, on both synthetic and real-life data.

We highlight the following contributions of this work:

We propose a novel graph-based continuous-time model GNeuralFlow for learning systemic interactions. The interactions are modeled as a Bayesian network, which can be learned in tandem with other model parameters.

We design model parameterizations by leveraging GNNs to encode the systemic interactions. These parameterizations can additionally be used in latent variable modeling.

We demonstrate the use of GNeuralFlow in regression problems and latent variable modeling and show the performance improvement in several time series classification and forecasting benchmarks.

2 Background: Neural ODE and Neural Flows

Denote by x ( t ) ∈ R d the solution of an ODE

under well-behaving conditions (e.g., a specified initial condition of x and Lipschitz continuity of f ). Neural ODE [7] is a modeling technique that allows uncovering the trajectory x ( t ) without a known right-hand side f . The technique parameterizes f by a neural network with parameters θ such that the trajectory x is a function of θ . Through matching the trajectory with observed data at a few (possibly irregular) time points by using a loss function L , the vector field f is unveiled. Given the initial condition x ( t 0 ) = x 0 , we write x ( t 0 ) , . . . , x ( t N ) = ODESolve ( f, x 0 , ( t 0 , . . . , t N )) , where the solutions at times t 1 , . . . , t N are obtained by invoking any blackbox numerical ODE solver (such as Runge-Kutta [17]). The training of the model parameters θ requires the gradient ∇ θ L , which can

be economically computed by using the adjoint a := ∇ x L rather than expensively back-propagating through the ODE solver.

Neural ODE has two far-reaching impacts. First, it is a continuous-time technique, which is a better alternative to discrete-time techniques (such as RNNs) for modeling irregularly sampled time series [39]. Second, it leads to a continuous version of the normalizing flow [35].

Other than directly modeling the observed data x , Chen et al. [7] proposed to use neural ODEs as latent variable models, which model the latents z instead; that is, ˙ z = f ( t, z ) . A straightforward idea is to build a variational autoencoder (V AE) [29], where the encoder is an RNN that evolves the hidden state h ( t ) over training time points and concludes a latent variable z 0 as the ODE initial condition. A drawback is that this approach still uses a discrete-time model (RNN) to handle irregularly sampled observations. Two approaches mitigating this drawback are ODE-RNN [39] and GRU-ODE-Bayes [5]. Both approaches demonstrate smaller prediction errors and variances compared with the vanilla neural ODE + VAE approach.

Another drawback of neural ODE is that it invokes a numerical solver, often multiple times in the adjoint computation because of multiple time intervals, which can be rather time consuming. A neural flow [4] is an alternative to neural ODE as it models the solution of (1) directly:

by using a parameterized function F that depends on the initial condition x 0 . Optimizing the parameters of F can be more efficient because ODE solvers are no longer needed. A neural flow is not to be confused with a normalizing flow. Neural flows can replace the use of neural ODEs in latent variable models ODE-RNN and GRU-ODE-Bayes.

3 DAG-Based ODE for Modeling Systemic Interactions

In this section, we motivate the form of ODE considered in this paper based on DAG modeling.

3.1 DAG Model for Systemic Interactions

̸

In probabilistic graphical models, the conditional dependence structure is a principled framework for modeling systemic interactions. Therein, a Bayesian network [36] of n random variables y 1 , . . . , y n is a DAG with these variables as the nodes. Let A ∈ R n × n be the (weighted) adjacency matrix of the DAG, where a ij = 0 means that y i is a parent of y j . A Bayesian network describes the conditional dependence structure of the variables; namely, a node is conditionally independent of its non-descendents given its parents. Therefore, the joint probability p ( y 1 , . . . , y n ) can be factorized into a much simpler form: p ( y 1 , . . . , y n ) = ∏ n j =1 p ( y j | pa ( y j )) , where pa ( y j ) = { y i : a ij = 0 } denotes the parent set of y j . The conditional dependence is a necessary condition for the causal relationship between parent y i and child y j [37].

̸

The (linear) structural equation model, SEM [45, 14], is a commonly used tool to further quantify the conditional probabilities. Without loss of generality, assume that the random variables are topologically sorted according to the partial ordering ≺ , where i ≺ j iff there exists an edge from i to j . Then, the DAG adjacency matrix A = [ a ij ] is strictly upper triangular. The SEM model reads

where the residuals ϵ 1 , . . . , ϵ n are (possibly correlated) Gaussian noises. In this model, y j depends on y i only when i < j . Importantly, some of the above a ij 's can be zero. Then, y j is independent of such y i 's given the rest.

non-interacting

interacting

non-interacting,newinit

interacting.newinit

0.6

0.4

xola(t)

x2la(t)

x1(t)

0.2

xala(t)

x²(t)

Xnew(t)

0.0

xnew(t)

xnew(t)

0.2

[o

0.5

0

[o

0.5

0

0

0

0.7

0

0

0.7

0

0

0

0

0

0

0.4

0.0

0.5

0.0

0.5

0.0

0.5

0.0

0.5

3.2 DAG-Based ODE as Continuous-Time Models

Consider an autonomous ODE ˙ x = Bx with the initial condition x (0) = x 0 ∈ R 2 . This ODE describes a 2D vector field Bx ; each solution curve x ( t ) = expm( B t ) x 0 given the initial point x 0 is a streamline instantaneously tangential to the vector field. Throughout the paper, we use expm to denote matrix exponential for matrix arguments and exp to denote element-wise exponential.

Now consider n trajectories x 1 ( t ) , . . . , x n ( t ) . Inspired by SEM, we model that (i) the vector fields that generate the n trajectories follow the same conditional dependence structure governed by A and (ii) the residual Bx j -∑ j -1 i =1 a ij Bx i gives the velocity ˙ x j for each j . Mathematically,

In other words, the solution curve x j for each j and any initial point x j 0 is a streamline instantaneously tangential to the residual field.

The behaviors of non-interacting and interacting systems are fundamentally different. Figure 1 illustrates an example. The first plot shows three independent trajectories satisfying ˙ x i = Bx i with initial conditions x i 0 for i = 1 , 2 , 3 . The second plot shows three conditionally dependent trajectories, under the same initial conditions, but interacting through the DAG adjacency matrix

Because x 1 is independent of the rest, it is the same in both plots; but in the second plot, because x 2 depends on x 1 and x 3 depends on x 2 , these two trajectories are different from their counterparts in the first plot. Moreover, in the third and fourth plots, we change the initial conditions. As long as the new initial points are along the original trajectories, the new trajectories still follow the old ones when A does not exist; however, when A exists, x 2 and x 3 deviate from the original trajectories, because of the conditional dependence.

3.3 From Linear Dependence to General

In general, the dependence among the time series may not be linear, even though structurally it is governed by the matrix A . For example, a nonlinear SEM may inspire the ODE system Bx 1 = ˙ x 1 , Bx 2 = Bx 1 + ˙ x 2 , Bx 3 = Bx 1 ⊙ Bx 2 + ˙ x 3 , where the dependence of x 3 on x 1 and x 2 is not linear. Thus, we consider general ODE systems of the form

where recall that pa( x j ) denotes the set of parents of x j . Equivalently, we write ˙ X = f ( t, X , A ) in the matrix form. Note that (2) is a special case of (3), which can be written as ˙ X = ( I -A ⊤ ) XB ⊤ . Note also that (3) is permutation equivariant and A can be any DAG matrix, not necessarily upper triangular ones.

4 GNeuralFlow: An ODE-Solver-Free Method

4.1 Problem Setup and Model Framework

Problem 1. Let A ∈ R n × n be the weighted adjacency matrix of a DAG and let X ( t ) : R → R n × d be the solution curve of the initial-value ODE system

where the right-hand side f is unknown. 2 Given data X ( t 0 ) , . . . , X ( t N ) at irregular time points, develop a model that predicts X ( t ) for any t ≥ t 0 as well as A . To account for practical use, at some time point t j , some rows of X ( t j ) may be missing.

A growing body of research addresses the problem when X has a single row (i.e., n = 1 ; hence, A is irrelevant), notably through using a neural network to parameterize f and using a numerical solver to evaluate X at t 0 , . . . , t N [7, 39, 5]. When n > 1 and the rows of the system are independent (i.e., f is identically the same function for each row), these methods straightforwardly apply through batch training. However, when the rows of the system are not independent, the problem becomes rather challenging. One may flatten (4) into an nd -dimensional problem, but such a high dimension renders approaches using numerical solvers too costly.

Instead, we use a neural network to parameterize the solution of (4) directly; i.e.,

where the solution F is a function of t but depends on the initial X 0 as well as A . The neural network parameterization cannot be entirely free. First, the solution F should satisfy the initial condition F (0 , X 0 , A ) = X 0 . Second, the fundamental theorem on flows [32, Theorem 9.12] asserts that every smooth f with an initial condition determines a unique F ( t, X 0 , A ) and for any t , F ( t, · , A ) is a diffeomorphism. Therefore, we formulate our model for Problem 1 in the following.

Solution framework. The model for X ( t ) is a neural network F ( t, X , A ) that satisfies:

F (0 , X 0 , A ) = X 0 ;

F ( t, X , A ) is invertible in X for any t and A ; equivalently, the streamline F ( t, X 0 , A ) given any X 0 and A is not self-intersecting.

4.2 Graph Encoder

We will make heavy use of a GNN to encode the DAG adjacency matrix A for parameterizing F . For simplicity, we employ the seminal architecture GCN. A (popularly used) two-layer GCN reads

where X is the input node feature matrix, ˜ X contains the transformed features, and W and U are parameters. GCN defines ̂ A as a symmetric normalization of A , but many alternatives are viable, such as a simple scaling of A . We also consider

̸

This definition is motivated by SEM, where I -A ⊤ is the operator (Section 3.3). Here, A is not symmetrized because doing so cannot distinguish edge directions. A benefit of taking (7) is that the scaling factor γ leads to a bounded spectral norm, which is an ingredient of invertibility required by the Solution framework .

Theorem 1. For any DAG adjacency matrix A , the matrix ̂ A defined in (7) admits ∥ ̂ A ∥ 2 ≤ 2 .

4.3 Parameterization of F

The neural network F in (5) can be defined in several ways by incorporating the graph encoder while satisfying the Solution framework .

ResNet flow. The first design is the ResNet architecture

which is a building block of invertible networks [1]. Here, φ ( t ) satisfies φ (0) = 0 such that the requirement F (0 , X 0 , A ) = X 0 is met. Additionally, if φ ( · ) ∈ [0 , 1] and g ( t, · , A ) is a contractive mapping, then F ( t, · , A ) is invertible.

We let φ be the tanh function and parameterize g by using two MLPs together with a GCN:

where || denotes concatenation row-wise and each MLP acts on the input matrix row-wise independently. The neural network g is generally not contractive, but bounding the spectral norm of each linear layer can theoretically guarantee contraction of an MLP [19]. Moreover, Theorem 1 indicates that bounding the spectral norm of the GCN parameters can guarantee contraction of GCN as well (because ∥ ̂ A ∥ 2 is bounded). Thus, in theory, g can be made contractive. In practice, regularization is used to encourage a small Lipschitz constant of g [19].

GRU flow. The second design mimics the GRU [9]:

where h k , k = 1 , 2 , is computed by

The base form X + φ ( t ) · h ( t, X ) , analogous to ResNet, comes from [5], who derived an ODE with the right-hand side being h through algebraic manipulation of the GRU. We extend the base form by including an analogous term h 2 that incorporates the graph encoder. It can be shown that F is invertible under a deliberate choice of α and β when the MLPs f k z , f k r , f k c and the GCN are contractive. As discussed earlier, Theorem 1 indicates that GCN can be made contractive similarly as the MLPs through bounding the spectral norm of their parameters.

Theorem 2. If f k z ( t, · ) , f k r ( t, · ) , f k c ( t, · ) , and GCN( A , · ) are contractive, the function F ( t, · , A ) defined in (10) is invertible whenever α (5 β +6) ≤ 2 .

Coupling flow. For the third design, we use normalizing flows, because they are invertible by definition. An example of the normalizing flow is the coupling flow [13]. Let { U, V } be a partitioning of the column indices 1 , . . . , d . With the graph encoder, we extend a usual coupling flow block to the following:

where

Here, φ u and φ v are two functions that have a range [0 , 1] and attain 0 at the origin. The input X is split into X U and X V and the second part goes through the GCN encoder, producing ˜ X V . Note that one cannot apply the GCN encoder on the entire X ; otherwise, the U block will have a dependency on itself in the scaling and the shift. The scaling network u and the shift network v are essentially MLPs that share initial layers; they take both X V and ˜ X V as inputs.

4.4 Learning the Graph

GNeuralFlow contains two sets of parameters: the DAG matrix A and other parameters of F (call them θ ), including the flow parameters, the graph encoder parameters, and possibly other parameters (e.g., in latent variable modeling, Appendix F). Let us use L ( A , θ ) to denote the training loss (which could be the quadratic loss in regression models, or likelihood/ELBO loss in latent variable models), making an explicit distinction between A and θ . Then, the learning problem is:

̸

The DAG constraint is combinatorial, which makes the problem NP-hard [8]. Fortunately, it is known that A is a DAG matrix iff tr(expm( A ⊙ A )) = n or tr(( I + α A ⊙ A ) n ) = n for any α = 0 [48, 47]. Hence, (12) becomes an equality-constrained problem over continuous variables, to which the augmented Lagrangian method [2] is an effective solution. We discuss the optimization details in Appendix D.

5 Experiments

We conduct a comprehensive set of experiments to demonstrate that the proposed graph-based approach effectively improves the performance of time series tasks. The experiments are done on four synthetically generated interacting systems and four real-life datasets. Details of these datasets and their tasks are given in Appendix G. In all experiments, we split the data into train, validation, and test sets. We train with early stopping by using Adam and report the results on the test set. Standard errors are obtained by performing five repetitive runs. Hyperparmeter details are given in Appendix H. All experiments are conducted on a machine with an Nvidia A100 GPU, 8 CPU cores, and 80GB main memory.

5.1 Synthetic Systems

We generate four synthetic systems with the graph size varying from 3 to 30. These systems follow the graph-based equation (4) or solution (5). They are named 'Sink,' 'Triangle,' 'Sawtooth,' and 'Square,' following those defined in [4]; but we add a graph to make the system SEM-like. For example, Triangle is generated by following F ( t, X , A ) = ( I -A ⊤ )( X + ∫ t 0 sign(sin( u )) du ) ; see Appendix G for other systems and details.

1.0

x10-2

Sink

x10-2Triangle

x10-2Sawtooth

x10-2 Square

1.5

1.5

2

Neural Flow

MSE

E

0.5

1.0

1.0

(learnedgraph)

GNeuralFlow

0.5

0.5

1

(true graph)

GNeuralFlow

0.0

10

20

30

10

20

30

10

20

30

10

20

30

#graph nodes

#graph nodes

#graph nodes

#graph nodes

The forecast results are reported in Figure 2. Two observations follow. First, across datasets and across system sizes, GNeuralFlow lowers the MSE of neural flows. This should not be surprising, since the systems are generated to be interacting and thus standard neural flows that treat the trajectories independently struggle to capture the influence of the graph. Second, GNeuralFlow performs similarly when the graph is either learned or supplied by the ground truth. This observation indicates that the learned graph sufficiently encodes the interacting nature of the trajectories.

We further compare GNeuralFlow with various baselines, including neural ODE, neural flows, graph ODEs (GDE [38], LG-ODE [24], and CF-GODE [26]), graph learning methods (NRI [31] and dNRI [20]), and a non-graph GRU variant for time series (GRU-D [6]). For graph ODEs, the groundtruth graph is used. For neural flows and GNeuralFlow, all three flow designs are experimented with. Table 1 shows that across all datasets, GNeuralFlow significantly outperforms the baselines; moreover, GNeuralFlow also significantly outperforms neural flow for each flow design. These findings indicate that our graph encoder is rather effective and the modeling of conditional dependencies (a DAG structure) is advantageous over that of other graph structures.

Truth

Noise = 0.01

Noise = 0.05

Noise = 0.1

Noise = 0.3

0.75

0.50

0.25

-0.00

Data

MSE 1.31e-02

MSE 5.63e-03

MSE 7.95e-03

MSE 2.76e-03

1.0

Sink

1.0

Triangle

1.0

Sawtooth

1.0

Square

DAG metric values

0.8

0.8

0.8

0.8

TPR

0.6-

0.6

0.6

0.6

0.4

FPR

0.4

0.4

0.4

130 SHD

0.2

0.2

0.2

0.2

0.0

0.0

0.2

0.4

°0.0

0.2

0.4

0.0

0.2

0.4

0.0

0.2

0.4

noise

noise

noise

noise

With the above encouraging results, we investigate the quality of graph learning. While the learning approach discussed in Section 4.4 works for a generally initialized A (as demonstrated by the previous plots), we perform a more in-depth investigation by initializing A through perturbing each entry of the ground truth with a zero-mean Gaussian. We evaluate graph quality by using the metrics proposed by [48]: TPR (true positive rate), FDR (false positive rate), FPR (false prediction rate), and SHD (structural Hamming distance).

Figure 3 shows that as the standard deviation of the Gaussian increases, the learned DAG is more and more different from the ground truth, with the TPR decreasing and the FDR, FPR, and SHD increasing. However, the forecast MSE remains relatively flat (in fact, the MSE from the ground truth is slightly higher). Note that with the nonzeros of the ground truth lying between 0 and 1, a Gaussian with standard deviation 0.3 as the initial guess barely carries the signal of the original graph. We have not been able to establish identifiability conditions for A from the ODE (4) as a data generation model; and we suspect that the conditions, if at all exist, may be unrealistically restrictive, given the

empirical findings that a better downstream performance can be achieved by a DAG significantly different from the ground truth. However, even though the ground truth is not recovered, a better downstream performance showcases the robust advantage of a graph-based model that intends to capture the complex interplay inside a system.

We also compare the time costs of neural ODE, neural flows, and GNeuralFlow. Table 2 indicates that GNeuralFlow is more expensive than the corresponding neural flow, because of the additional modeling of the graph. However, GNeuralFlow is more economic than neural ODE, as expected, because it does not run a numerical solver.

5.2 Latent Variable Modeling: Smoothing

Just like neural ODE and neural flows, GNeuralFlow can be used for latent variable modeling (see details in Appendix F). To illustrate the effectiveness of GNeuralFlow for this application, we first perform experiments with the smoothing approach in this subsection, by using real-life datasets Activity, Physionet, and MuJoCo [4]. For Activity, we treat each sensor as a graph node; while for the other two, we treat each feature as a node. The tasks are to reconstruct the time series, to classify the activity at each time step (Activity), and to predict the mortality of patients based on the entire time series (Physionet). We again compare our methods with baselines including neural ODE, neural flows, and graph ODEs. For graph ODEs, we construct a dynamic graph at each time step by utilizing the covariance matrix of the time series data within each batch.

From Table 3, we see that GNeuralFlow generally performs the best compared with the various baselines. Moreover, by using the same flow design, GNeuralFlow is always better than neural flows. These findings are consistent with those in the synthetic data case, demonstrating a good utility of our method for real-life applications.

5.3 Latent Variable Modeling: Filtering

We also perform experiments with the filtering approach, on the MIMIC-IV dataset [4]. We treat each feature as a node to set up a graph of 97 longitudinal features, including lab tests, outputs, and prescriptions in clinical events. This graph is the largest among all experiments in this paper.

Table 4 reports the forecast error on the next three time points and the estimated likelihood of the time series. We see that GNeuralFlow performs the best with the GRU flow design, while some graph ODE approaches come second. Moreover, GNeuralFlow consistently outperforms neural flows, across flow architecture designs and evaluation metrics. The improvement over corresponding neural flow versions is at least the sum of the standard errors of the two compared methods, which are demonstratively significant.

MSE

NLL

No

Graph

GRU-ODE-Bayes

0.379 (

±

0.005)

0.748 (

±

0.045)

Neural flow (ResNet)

0.379 (

±

0.005)

0.774 (

±

0.059)

Neural flow (GRU)

0.364 (

±

0.008)

0.734 (

±

0.054)

Neural flow (Coupling)

0.366 (

±

0.002)

0.675 (

±

0.003)

Graph

ODE

GDE

0.342 (

±

0.001)

0.657 (

±

0.007)

LG-ODE

0.349 (

±

0.002)

0.649 (

±

0.005)

CG-ODE

0.372 (

±

0.011)

0.825 (

±

0.018)

Our

Method

GNeuralFlow (ResNet)

0.356 (

±

0.0007)

0.663 (

±

0.008)

GNeuralFlow (GRU)

0.335

(

±

0.003)

0.606

(

±

0.001)

GNeuralFlow (Coupling)

0.350 (

±

0.004)

0.662 (

±

0.008)

6 Conclusions and Discussions

In this work, we address the challenge of learning the systemic interactions of time series, by proposing a graph-based model GNeuralFlow and learning the graph structure in tandem with the system dynamics. GNeuralFlow is a continuous-time model, which can be used for irregularly sampled time series. Moreover, the systemic interactions are modeled by a conditional dependence structure. We apply GNeuralFlow to latent variable modeling and demonstrate that incorporating the DAG structure improves time series classification and forecasting noticeably.

Several time-series approaches do not learn a static graph but a dynamic one [20]. Such a graph is interpreted as a latent structure, which varies depending on past data. In contrast, our approach learns an explicit structure that governs the dynamics over time. Nevertheless, our mathematical framework can be straightforwardly adapted to learning a time-varying latent graph, if desired. To achieve so, we reuse the loss calculation in (12), remove the constraint, and parameterize A as a function of X ( t ) . This way, we sacrifice the DAG interpretation of the interactions but gain a time-dependent graph.

Alimitation of the proposed model is that the number of parameters on the A part grows quadratically with the number of time series (nodes). Hence, this part of the computational cost can be cubic, because the evaluation of the DAG constraint and the gradient involves the computation of the matrix exponential. Such a scalability challenge is a common problem for DAG structure learning. While past research showcased the feasibility of learning a graph with a few hundred nodes [47], going beyond is generally believed to require either a new computational technique or a new modeling approach. One potential direction is to introduce structures into A (such as low-rankness [15]), which admit faster matrix evaluation.

Acknowledgments and Disclosure of Funding

GMacknowledges support from the Engineering and Physical Sciences Research Council (EPSRC) and the BBC under iCASE. AF is partially funded by the CRUK National Biomarker Centre, by the Manchester Experimental Cancer Medicine Centre and the NIHR Manchester Biomedical Research Centre. JC is supported by the MIT-IBM Watson AI Lab.

References

Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In ICML , 2019.

Dimitri P. Bertsekas. Nonlinear Programming . Athena Scientific, 2nd edition, 1999.

Dhananjay Bhaskar, Sumner Magruder, Edward De Brouwer, Aarthi Venkat, Frederik Wenkel, Guy Wolf, and Smita Krishnaswamy. Inferring dynamic regulatory interaction graphs from time series data with perturbations. In LoG , 2024.

Marin Biloš, Johanna Sommer, Syama Sundar Rangapuram, Tim Januschowski, and Stephan Günnemann. Neural flows: Efficient alternative to neural ODEs. In NeurIPS , 2021.

Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series. In NeurIPS , 2019.

Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific Reports , 8(6085), 2018.

Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In NeurIPS , 2018.

David Maxwell Chickering, David Heckerman, and Christopher Meek. Large-sample learning of Bayesian networks is NP-hard. Journal of Machine Learning Research , 5:1287-1330, 2004.

Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259 , 2014.

Jeongwhan Choi, Hwangyong Choi, Jeehyun Hwang, and Noseong Park. Graph neural controlled differential equations for traffic forecasting. In AAAI , 2022.

Enyan Dai and Jie Chen. Graph-augmented normalizing flows for anomaly detection of multiple time series. In ICLR , 2022.

Tristan Deleu, Mizu Nishikawa-Toomey, Jithendaraa Subramanian, Nikolay Malkin, Laurent Charlin, and Yoshua Bengio. Joint Bayesian inference of graphical structure and parameters with a single generative flow network. In NeurIPS , 2023.

Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In ICLR , 2017.

O. Duncan. Introduction to Structural Equation Models . Academic Press, New York, 1975.

Zhuangyan Fang, Shengyu Zhu, Jiji Zhang, Yue Liu, Zhitang Chen, and Yangbo He. On low rank directed acyclic graphs and causal structure learning. TNNLS , 35(4):4924-4937, 2024.

Morris R. Flynn, Aslan R. Kasimov, Jean-Christophe Nave, Rodolfo R. Rosales, and Benjamin Seibold. Self-sustained nonlinear waves in traffic flow. Physical Review E , 79(5):056113, 2009.

George E. Forsythe, Michael A. Malcolm, and Cleve B. Moler. Computer Methods for Mathematical Computations . Prentice Hall, first edition edition, 1977.

Ary L. Goldberger, Luis A. N. Amaral, Leon Glass, Jeffrey M. Hausdorff, Plamen Ch. Ivanov, Roger G. Mark, George B. Moody Joseph E. Mietus, Chung-Kang Peng, and H. Eugene Stanley. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation , 101(23):e215-e220, 2000.

Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J. Cree. Regularisation of neural networks by enforcing Lipschitz continuity. Machine Learning , 110:393-416, 2021.

Colin Graber and Alexander G. Schwing. Dynamic neural relational inference. In CVPR , 2020.

Paul D. H. Hines, Ian Dobson, and Pooya Rezaei. Cascading power outages propagate locally in an influence graph that is not the actual grid topology. IEEE Transactions on Power Systems , 32(2):958-967, 2017.

Sujai Hiremath, Jacqueline R.M.A. Maasch, Mengxiao Gao, Promit Ghosal, and Kyra Gan. Hybrid top-down global causal discovery with local search for linear and nonlinear additive noise models. Preprint arXiv:2405.14496, 2024.

Carlos Hoppen, Juan Monsalve, and Vilmar Trevisan. Spectral norm of oriented graphs. Linear Algebra and its Applications , 574:167-181, 2019.

Zijie Huang, Yizhou Sun, and Wei Wang. Learning continuous system dynamics from irregularly-sampled partial observations. In NeurIPS , 2020.

Zijie Huang, Yizhou Sun, and Wei Wang. Coupled graph ODE for learning interacting system dynamics. In SIGKDD , pages 705-715, 2021.

Song Jiang, Zijie Huang, Xiao Luo, and Yizhou Sun. CF-GODE: Continuous-time causal inference for multi-agent dynamical systems. In SIGKDD , 2023.

Ming Jin, Yu Zheng, Yuan-Fang Li, Siheng Chen, Bin Yang, and Shirui Pan. Multivariate time series forecasting with dynamic graph neural ODEs. TKDE , 35(9):9168-9180, 2023.

Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. MIMIC-IV (version 1.0). PhysioNet , 2021.

Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In ICLR , 2014.

Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In ICML , 2018.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR , 2017.

John M. Lee. Introduction to Smooth Manifolds . Springer, 2nd edition edition, 2012.

Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In ICLR , 2018.

Mizu Nishikawa-Toomey, Tristan Deleu, Jithendaraa Subramanian, Yoshua Bengio, and Laurent Charlin. Bayesian learning of causal structure and mechanisms with GFlowNets and variational Bayes. Preprint arXiv:2211.02763, 2022.

George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. JMLR , 22(57): 1-64, 2021.

Judea Pearl. Bayesian networks: A model of self-activated memory for evidential reasoning. In Proceedings of the 7th Conference of the Cognitive Science Society , 1985.

Judea Pearl. Causality: Models, Reasoning, and Inference . Cambridge University Press, 2000.

Michael Poli, Stefano Massaroli, Junyoung Park, Atsushi Yamashita, Hajime Asama, and Jinkyoo Park. Graph neural ordinary differential equations. Preprint arXiv:1911.07532, 2019.

Yulia Rubanova, Ricky T. Q. Chen, and David Duvenaud. Latent ODEs for irregularly-sampled time series. In NeurIPS , 2019.

Mona Schirmer, Mazin Eltayeb, Stefan Lessmann, and Maja Rudolph. Modeling irregular time series with continuous recurrent units. In ICML , 2022.

Chao Shang, Jie Chen, and Jinbo Bi. Discrete graph structure learning for forecasting multiple time series. In ICLR , 2021.

Ikaro Silva, George Moody, Daniel J Scott, Leo A Celi, and Roger G Mark. Predicting inhospital mortality of ICU patients: The physionet/computing in cardiology challenge 2012. Comput Cardiol , 2012(39):245-248, 2010.

Stephen Smith and Qing Zhou. Coordinated multi-neighborhood learning on a directed acyclic graph. Preprint arXiv:2405.15358, 2024.

Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. DeepMind control suite. Preprint arXiv:1801.00690, 2018.

S. Wright. Correlation and causation. Journal of Agricultural Research , 20:557-585, 1921.

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems , 32(1):4-24, 2021.

Yue Yu, Jie Chen, Tian Gao, and Mo Yu. DAG-GNN: DAG structure learning with graph neural networks. In ICML , 2019.

Xun Zheng, Bryon Aragam, Pradeep Ravikumar, and Eric P. Xing. DAGs with NO TEARS: Continuous optimization for structure learning. In NeurIPS , 2018.

Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. AI Open , 1:57-81, 2020.

A Supporting Code

Code is available at https://github.com/gmerca/GNeuralFlow .

B Proofs

Proof of Theorem 1. When A is a DAG adjacency matrix, its symmetrization B is the adjacency matrix of the corresponding undirected graph. We call the DAG an orientation of the undirected graph. The Gershgorin circle theorem asserts that the spectral radius of B , ρ ( B ) , is bounded by γ . Meanwhile, [23] show that the spectral norm ∥ A ∥ 2 ≤ ρ ( B ) . Then, ∥ ̂ A ∥ 2 ≤ 1 + γ/γ = 2 .

Proof of Theorem 2. We follow the proof of [4, Theorem 1] (see A.3 of the paper), which concludes that | h ( x ) -h ( y ) | ≤ α ( 5 4 β + 3 2 ) | x -y | . Since GCN is contractive, such an inequality applies to both h = h 1 and h = h 2 . Then, applying Eqn (14) of the paper,

Therefore, when α (5 β +6) ≤ 2 , the product h 1 h 2 is contractive and therefore F is invertible.

C Details of Example in Section 3

The matrix is and the initial conditions are

D Training Method

In this section, we briefly describe the augmented Lagrangian method for solving the equalityconstrained problem

where the constraint can either be h ( A ) = tr(expm( A ⊙ A )) -n or h ( A ) = tr(( I + α A ⊙ A ) n ) -n . Define the augmented Lagrangian

where λ and c denote the Lagrange multiplier and the penalty parameter, respectively. The general idea of the method is to gradually increase the penalty parameter to ensure that the constraint is eventually satisfied. Over iterations, λ as a dual variable will converge to the Lagrange multiplier of the original problem. The upate rule at the k th iteration reads

where η ∈ (1 , + ∞ ) and γ ∈ (0 , 1) are hyperparameters to be tuned.

The subproblem of optimizing A and θ can be solved by using the Adam optimizer. It requires the gradient of L c and hence of h . For h ( A ) = tr(expm( A ⊙ A )) -n , it can be derived that ∇ h ( A ) = expm( A ⊙ A ) ⊤ ⊙ 2 A , which can be obtained virtually for free after h has been evaluated. For h ( A ) = tr(( I + α A ⊙ A ) n ) -n , one may use automatic differentiation to obtain the gradient.

Algorithm 1 summarizes the training procedure. Note that an effective initialization of A would use an empty diagonal. Moreover, in every update of A , one may keep its diagonal zero throughout.

E Handling Missing Data

At a particular time t , some rows of the observed data X ( t ) may be empty (i.e., measurements of some time series at time t are missing). In this case, one evaluates the graph neural flow F on a subgraph of present measurements. Rather than extracting this subgraph and the corresponding rows of X , the GCN encoder (6) offers a convenient approach for evaluation: masking. This approach is particularly favorable in batching, because tensor dimensions do not change. In particular, we mask out (i.e., setting zero) the part of A corresponding to missing data. Then, for the output ˜ X , the part corresponding to present data is correctly calculated, while the part of missing data becomes zero and this condition is invariant across layers. Note that the GCN layers must not have bias terms to maintain this invariance.

F GNeuralFlow for Latent Variable Modeling

In addition to straightforwardly modeling the data space, neural flows find successful use in the latent space. Here, we discuss two popular approaches in latent variable modeling and how GNeuralFlow can be incorporated.

Of particular consideration is the role of the graph. One may straightforwardly extend [4] by using the flow in the latent/hidden space; however, this method models the interactions among the latent/hidden dimensions, which are less interpretable than those among the time series. Hence, we propose to use the flow on an augmented space, part of which carries the graph information, as a new design complementary to those proposed in Section 4.3.

. .

On a high level, smoothing [39] and filtering [5] approaches use a neural ODE or a neural flow to continuously evolve the hidden state from time t j -1 (denoted as H ( t j -1 ) ) to time t j (denoted as H ′ ( t j ) ); and then use an RNN to introduce a jump (denoted as H ( t j ) ) on observing input data X ( t j ) To model and learn the interaction graph in the data space, we use the graph encoder (6) to produce a transformed data ˜ X ( t j ) and use a second RNN to introduce the paired jump ˜ H ( t j ) given ˜ X ( t j ) Then, the pair of hidden states, H ( t j ) and ˜ H ( t j ) , are concatenated and a standard neural flow evolve the concatenated state to the next time point, the result of which is then projected to the proper hidden dimension. The smoothing and filtering approaches differ in fine details, including different uses of the RNNs and hidden states. As a result, the loss function L in (12) is also different. Details are presented in the following.

F.1 Smoothing Approach

Given observation data X ( t 0 ) , . . . , X ( t N ) , this approach produces a latent quantity Z 0 by a combined use of LSTM and neural flow, and then traces out a smooth curve Z ( t ) using another flow, taking Z ( t 0 ) = Z 0 as the initial condition. Then, the observation data X ( t j ) is recovered from Z ( t j ) .

A VAE is used to set up the training loss. The decoder p ( X ( t 0 ) , . . . , X ( t N ) | Z 0 ) is factorized as

where each Z ( t j ) is computed by running a standard neural flow: Z ( t j ) = F ( t j , Z 0 ) . The encoder, on the other hand, produces a latent Gaussian Z 0 with mean µ and diagonal covariance diag( σ ) ; that is,

where H ( t N ) is the hidden state to be elaborated soon and g is a neural network projection. 3 The ELBO loss for the VAE is

Our GNeuralFlow uses a pair of LSTMs together with another neural flow to evolve the hidden state. Specifically, we maintain a pair of states H ( t ) and ˜ H ( t ) , the latter of which includes the graph information. At time t j -1 , we concatenate H ( t j -1 ) and ˜ H ( t j -1 ) , run the neural flow F to evolve the concatenated state to time t j , and apply a projection g proj so that the net result H ′ ( t j ) remains in the same dimension as H ( t j -1 ) :

Then, we run a pair of LSTMs to obtain the paired states at time t j :

where the second LSTM is applied to the transformed observation data ˜ X ( t j ) produced by the GCN encoder (6). By doing so, the graph models the interaction inside the data X ( t ) rather than the hidden states H ( t ) .

F.2 Filtering Approach

As opposed to the preceding approach, the filtering approach uses only a decoder. Each time, it first evolves the hidden state to H ′ ( t j ) and then runs a GRU to update the hidden state to H ( t j ) . This approach maintains two Gaussians, the first one models the observation X ( t j ) :

while the second one models the jump caused by the GRU:

The training loss aims at maximizing the observation data likelihood while minimizing the KL divergence of the two Gaussians:

Our GNeuralFlow uses a pair of GRUs together with a standard neural flow to evolve the hidden state. Specifically, we maintain a pair of states H ( t ) and ˜ H ( t ) , the latter of which includes the graph

information. At time t j -1 , we concatenate H ( t j -1 ) and ˜ H ( t j -1 ) , run the neural flow F to evolve the concatenated state to time t j , and apply a projection g proj so that the net result H ′ ( t j ) remains in the same dimension as H ( t j -1 ) :

Then, we run a pair of GRUs to obtain the paired states at time t j :

where the second GRU is applied to the transformed observation data ˜ X ( t j ) produced by the GCN encoder (6). By doing so, the graph models the interaction inside the data X ( t ) rather than the hidden states H ( t ) .

G Datasets and Tasks

The datasets used in this paper include four synthetic ODE systems and four real-life datasets. Table 5 summarizes the basic information of these datasets, tasks, evaluation metrics, and learning methods.

G.1 Synthetic Datasets

We define four interacting systems based on either the ODE ˙ X = f ( t, X , A ) or the solution X ( t ) = F ( t, X 0 , A ) :

Sink (2D): f ( t, X , A ) = ( I -A ⊤ ) XB ⊤ where B = [ -4 10 -3 2 ]

Triangle (1D): F ( t, X , A ) = ( I -A ⊤ )( X + ∫ t 0 sign(sin( u )) du )

Sawtooth (1D): F ( t, X , A ) = ( I -A ⊤ )( X + t -⌊ t ⌋ )

Square (1D): F ( t, X , A ) = ( I -A ⊤ )( X +sign(sin( t )))

For Sink, X ∈ R n × 2 ; while for the other three systems, X ∈ R n × 1 , where n is the number of trajectories (graph nodes) in the system. The initial condition X 0 is uniformly sampled from [0 , 1] n × 2 for Sink, and from [ -2 , 2] n × 1 for the other three systems. The time interval is [0 , 10] and the time points are uniformly random.

The DAG adjacency matrix is generated by using the following procedure:

Generate a sparse n × n matrix A with a pre-defined density, where the nonzero locations are random and the nonzero values are uniformly random.

Keep only the strict upper triangular part of A (i.e., diagonal is zero).

Perform symmetric row/column permutation on A .

The task is to predict the trajectories X ( t ) given X 0 .

G.2 Real-Life Datasets

We use four real-life datasets preprocessed by [4].

Activity [39] contains time series recorded by four sensors, on individuals performing various activities: walking, sitting, lying, etc. The task is to classify the activities at each time point. Additionally, since the smoothing approach for latent variable modeling reconstructs the time series, we also evaluate different models on the reconstruction quality. We treat each sensor as one graph node.

Physionet [42] contains time series of patients' measurements (37 variables in total) from the first 48 hours after being admitted to ICU. The task is to predict the mortality of the patients. Additionally, since the smoothing approach for latent variable modeling reconstructs the time series, we also evaluate different models on the reconstruction quality. We treat each variable as one graph node.

MuJoCo [44] contains physics simulations by randomly sampling initial positions and velocities and letting the dynamics evolve deterministically in time. Each sequence includes 14 features. We treat each feature as one graph node. We evaluate different models on the reconstruction quality.

MIMIC-IV [18, 28] contains time series of ICU patients' measurements, including their vital signs, laboratory test results, medication, and any output data during their ICU stay (97 variables in total). The task is to predict the next three measurements in the 12 hour interval after the observation window of 36 hours. We treat each variable as one graph node.

H Hyperparameter Details

We reuse the architecture parameters and training hyperparameters in [4] and only tune the graph learning hyperparameters (see Algorithm 1 in Section D). Table 6 lists the tuned values.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The contributions of the paper are summarized in the introduction section and elaborated in the following sections.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of the work are discussed in the concluding section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Assumptions are given in the theorems and proofs are given in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Experiment information is provided in part in the main text and in part in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code will be released upon publication of the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experiment settings and details are provided in part in the main text and in part in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars are obtained by performing five repetitive runs for each method and dataset.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Compute information is given in the main text.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We confirm.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Codes and datasets used for experiments are publicly available under permissive licenses.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Code implementation of the proposed method will be released upon publication. Comprehensive documentation will be provided for reproducibility and access.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose a novel idea to learn the interactions between time series which is crucial to make reliable forecasts in interacting systems. Paper proposes a new algorithm GNeuralFlow; a new graph-based continuous time model to learn systematic interaction. A GNN is used to learn the interactions between time series, where each time series is associated with a directed acyclic graph. GNeuralFlow learns the graph structure for irregularly sampled time series.  \n\nThe experiments performed on synthetic and real data shows they GNeuralFlow outperform the other models based regression and classification tasks. The experiments to model latent variable modelling shows GNeuralFlow's robustness in learning and utilizing the underlying graph structure for improved predictive performance.\n\nSTRENGTHS:\nOriginality:  Authors identify an important problem in non-uniform time samples time series to model the interactions between them. The paper introduces the original idea (as per my knowledge) of using Graph Neural Flow (GNeuralFlow) which leverages the power of graph based modelling and continuous time technique to capture the relationship between time series.   The method has wide application in climate, finance and more where a dynamic system is crucial. \n\nQuality:   The main development of paper GNeuralFlow is built on a strong mathematical foundation and authors provide empirical and experimental results to support their claims. \n\nClarity: The paper is well structured and describes the problem clearly and then proceeds to describe GNeuralFlow as a solution to it and advance the field. The text provides a clear explanation of the idea with a complete mathematical framework to support the same.  \n\nSignificance: The idea of capturing relationship between time series using GNeural is significant improvement in the methodology. Learning the graph as a solution to ODE gives the flexibility to use it for unequal samples of time series. This has several applications in time series domain including streaming data. \n\nThe idea is to use traditional GCN to learn the interaction between time series for continuous time GNeuralFlow can handle the time series data observed at non-uniform time stamps without the need for interpolation or computing expensive pre-processing.\n\nWEAKNESSES:\nAlthough the paper address a big problem faced in steaming data application with unequal time samples using novel approach the idea comes with additional complexity. The complex nature of the solution is computationally expensive and is not scalable after a certain size.  The fact has been identified by authors as well and documented in the limitations sections.\n\nQUESTIONS:\n- The paper discusses about the increase in the complexity due to added Graphs in the model. Is there an estimate how much is the computation time increase (is it quadratic?) ? Does this time power increase (from quadratic to cubic) as # of interacting time series are increased in the experiment?\n\nLIMITATIONS:\nThe paper is well structured but the biggest strength becomes its limitation as well. The addition of graph and learning the parameters allow the measurement of interactions but at the same time it increase the complexity of the model. This would be a the main challenge to use the model in real applications and might need approximate solutions to adapt this version of paper.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-30"}

comments: SUMMARY:
This paper proposes a novel graph-based continuous-time model GNeuralFlow for learning systemic interactions and interactions are modeled as a Bayesian network. Experimental results for regression problems show that the proposed GNF achieves state-of-the-art performances on time series classification and forecasting benchmarks.

STRENGTHS:
+ The paper is well-written and easy to follow. 
+ The introduction give a nice overview and motivation for the problem.
+ The results are good and evaluation is reasonable.

WEAKNESSES:
- The proposed method lacks a theoretical analysis.
- The authors can consider using state-of-the-art GNN as backbone for graph encoder.
- Can the authors apply this method to real-world datasets?
- Computational complexity and corresponding comparisons with baselines are missing.

QUESTIONS:
Please see comments in Weaknesses.

LIMITATIONS:
Not applicable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on a graph-based model to capture dependencies in irregularly sampled time series data. The framework employs a causal prior—a directed acyclic graph—where nodes are conditionally independent of non-descendants given their parents, specifying component dynamics dependencies. The proposed model, termed a graph neural flow, learns the solution of an unknown ODE directly from irregularly sampled time series, which contrasts with neural ODEs by avoiding repeated calls to a costly numerical solver. Multiple neural flows, one for each time series, are conditioned on the DAG, with their interactions instantiated using a GNN, such as a graph convolutional network (GCN). The GCN enhances the ODE solution parameterization by aggregating neighboring time series information at each time point, effectively modeling a graph-conditioned ODE that captures system interactions.\n\nSTRENGTHS:\nThe proposed model is nicely presented, along with precise and meaningful mathematical formulations for capturing interactions in irregularly sampled time series. GNeuralFlow shows significant performance in both classification and regression problems spanning several synthetic and real-world datasets. Hyperparameters and models’ configurations are presented enabling reproducibility and fair comparisons.\n\nWEAKNESSES:\n1. *W1:* Except for the standard experiments on synthetic data representing dynamical systems, the authors explore different tasks on various datasets, slightly deviating from some setups followed in existing related works. For instance, some studies include additional experiments on interpolation/extrapolation [1] or use additional datasets (e.g., classification on MIMIC-III). Moreover, incorporating additional standard baselines in addition to the presented ones could strengthen the evaluation (recurrent/attention-based ones, e.g., mTAND, GRU-D, and others).\n2. *W2:* The proposed GNeuralFlow is primarily based on the concept of Neural Flows for efficient computation of ODE system solutions. The significant methodological extensions in the paper include the incorporation of graph-based representations and causal formulation. However, these extensions are not qualitatively evaluated by analyzing causal relationships or visualizing interactions within the learned graphs. For instance, authors in [2] present studies/experiments to quantify causal discovery in the studied datasets.\t\t\n3. *W3:* Although the approach of replacing the ODE solver offers a computational advantage over classical Neural-ODE-based methods, a comprehensive computational cost analysis comparing different methods, including simpler (e.g., sequential, non-ODE-based) yet effective models, is crucial.\n4. *W4:* The presentation of the related work is quite brief, causing some confusion, particularly regarding closely related graph-based methods like CF-GODE and LG-ODE. Including comparisons and comments on additional relevant papers could be beneficial [3,4] (please explain if not relevant).\n\n[1] Schirmer, M., Eltayeb, M., Lessmann, S., & Rudolph, M. (2022, June). Modeling irregular time series with continuous recurrent units. In International conference on machine learning (pp. 19388-19405). PMLR.\n\n[2] Löwe, S., Madras, D., Zemel, R., & Welling, M. (2022, June). Amortized causal discovery: Learning to infer causal graphs from time-series data. In Conference on Causal Learning and Reasoning (pp. 509-525). PMLR.\n\n[3] Choi, J., Choi, H., Hwang, J., & Park, N. (2022, June). Graph neural controlled differential equations for traffic forecasting. In Proceedings of the AAAI conference on artificial intelligence (Vol. 36, No. 6, pp. 6367-6374).\n\n[4] Jin, M., Zheng, Y., Li, Y. F., Chen, S., Yang, B., & Pan, S. (2022). Multivariate time series forecasting with dynamic graph neural odes. IEEE Transactions on Knowledge and Data Engineering, 35(9), 9168-9180.\n\nQUESTIONS:\nBased on the *weaknesses* above please focus on the following aspects:\n1. **Experiments:** Explain choices of datasets and baselines or extend the experimental evaluation (W1). Conduct qualitative evaluations of graph-based causal representations (W2).\n2. **Limitations:** Perform a comprehensive computational cost analysis comparing GNeuralFlow with simpler and more complex methods to highlight efficiency gains (W3).\n3. **Contribution:** Please better position the presented contribution among relevant works (W4).\n\nLIMITATIONS:\nIt would be more complete to experimentally showcase the computational limitations, demonstrating in practice the scalability of the proposed method for real-world datasets.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of multivariate time series prediction with feature interactions. It proposes learning a Directed Acyclic Graph (DAG) to model interactions, encoded by a Graph Neural Network (GNN), and using a neural flow to model the dynamics. The experiments demonstrate improvements over graph-free models and models with given graphs obtained from covariance.\n\nSTRENGTHS:\nThe combination of graph learning and neural flow, along with DAG-conditioned optimization, is original. Since flows are popular in generative modeling, the idea of converting them to the graph setting seems promising. The paper also learns graph parameters using the matrix A to learn (or relearn the DAG structure), which can be useful in cases where the graph is not previously known with certainty.\n\nWEAKNESSES:\nWeaknesses:\nFlow matching inherently has the idea of using a linear interpolant (straight line) to encode optimal transport paths in time series data. In spaces where this is not true, or where the optimal transport paths are perhaps in a latent dimension the assumptions are violated. This is why Flow matching has been more useful for generative modeling than time series. This is not noticed by the authors and the restrictive experiments they perform are insufficient to test this. \n\nI don't see the need for a DAG here. See the RITINI model [Bhaskar et al, LOG 2023], when there is time series data there can be feedback loops in the variables rather than DAG-style causality. I would appreciate trying to loosen this requirement, which would enable cyclic dependencies. \n\nThe set of comparisons featured here is rather limited. Here are several works that could be compared against: \n\nKipf et al., Neural Relational Inference for Interacting Systems, ICML, 2018.\nGraber and Schwing. Dynamic Neural Relational Inference, CVPR, 2020 (dynamic graph inference for discrete-time systems)\nNishikawa-Toomey et al., Bayesian learning of Causal Structure and Mechanisms with GFlowNets and Variational Bayes (jointly learn the DAG and the parameters of a linear-Gaussian causal model)\nDeleu et al., Joint Bayesian Inference of Graphical Structure and Parameters with a Single Generative Flow Network, NeurIPS, 2023 (jointly learn the posterior over the structure of a Bayesian Network, and also the parameters of its conditional probability distributions)\nSmith and Zhou, Coordinated Multi-Neighborhood Learning on a Directed Acyclic Graph, arXiv:2405.15358 (constraint-based approach that exploits conditional independence relationships between variables to infer the underlying causal model)\nHiremath et al. Hybrid Global Causal Discovery with Local Search, arXiv:2405.14496 (global causal discovery by learning local causal substructures using topological sorting and pruning of spurious edges)\nBhaskar et al. Inferring dynamic regulatory interaction graphs from time series data with perturbations, LOG 2023\n\nQUESTIONS:\nFor the experiments on synthetic datasets (Figure 1), how are the graphs for graph ODEs obtained? Are all models using the ground truth graph? If so, how is the conclusion that “DAG is better than other graph structures at modeling dependencies” (line 252) drawn?\n\nDoes the performance gain over existing graph ODE models come from the learnable graph? Is there an ablation study for that?\n\nIs the model learning a constant graph independent of time? How does it perform when there is a significant change in data dependence?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a continuous-time model to discover the causal structure from irregular multivariate time series data. The idea is to introduce the DAG (directed acyclic graph) to model the interaction of different time series at the same time step in the vector field that generates the multivariate time series（that is, the $n$ trajectories). Instead of learning the vector field, the paper proposes to learn the solution of the ODE system directly with neural flows. The structural interaction is implemented with graph convolution neural networks and three particular parameterization methods are proposed for the solution function $F$ to guarantee its invertibleness. The experiments on both synthetic and real-world irregular time series datasets are conducted to assess the efficacy of the proposed method for both prediction and classification tasks.\n\nSTRENGTHS:\n* The proposed method is well motivated and clearly illustrated, and the presentation is easy to follow. The reviewer appreciates the way to establish the solution framework presented in Section 4.1, which is very well explained.\n* It is interesting to introduce the DAG structure learning to irregular time series modeling and the proposed method seems reasonable to the reviewer. \n* The proposed method achieves better results than the existing method.\n\nWEAKNESSES:\n* In Section 3.2, it states \"(i) the vector fields $\\mathbf{B}\\mathbf{x}^1, \\ldots, \\mathbf{B}\\mathbf{x}^n$ follow the same conditional dependence structure governed by $\\mathbf{A}$\",  but according to Eq. (2), the vector fields seems to be $\\mathbf{B}\\mathbf{x}^j - \\sum_{i=1}^{j-1}a_{ij} \\mathbf{Bx}^{i}$ for $j=1,\\ldots,n$, so I would suggest changing the description to \"the vector fields that generate the $n$ trajectories follow the same conditional dependence structure governed by $\\mathbf{A}$\" to avoid confusion.\n* To learn the DAG matrix, the proposed method employs NOTEARS and DAG-GNN. How do these DAG learning algorithms impact the model performance?\n\n* The proposed method (graph neural flows) is dubbed GNeuralFlow in the main text but is referred to as GNF in the tables of experiments. It is better to unify the abbreviation. Besides, no code is released.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nNo potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-19"}