Title: CARE: Modeling Interacting Dynamics Under Temporal Environmental Variation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Dynamical System

Distribution Shift

Neural ODE

Graph Neural Network
Abstract: Modeling interacting dynamical systems, such as fluid dynamics and intermolecular interactions, is a fundamental research problem for understanding and simulating complex real-world systems. Many of these systems can be naturally represented by dynamic graphs, and graph neural network-based approaches have been proposed and shown promising performance. However, most of these approaches assume the underlying dynamics does not change over time, which is unfortunately untrue. For example, a molecular dynamics can be affected by the environment temperature over the time. In this paper, we take an attempt to provide a probabilistic view for time-varying dynamics and propose a model Context-attended Graph ODE (CARE) for modeling time-varying interacting dynamical systems. In our CARE, we explicitly use a context variable to model time-varying environment and construct an encoder to initialize the context variable from historical trajectories. Furthermore, we employ a neural ODE model to depict the dynamic evolution of the context variable inferred from system states. This context variable is incorporated into a coupled ODE to simultaneously drive the evolution of systems. Comprehensive experiments on four datasets demonstrate the effectiveness of our proposed CARE compared with several state-of-the-art approaches.

CARE: Modeling Interacting Dynamics Under Temporal Environmental Variation

Abstract

Modeling interacting dynamical systems, such as fluid dynamics and intermolecular interactions, is a fundamental research problem for understanding and simulating complex real-world systems. Many of these systems can be naturally represented by dynamic graphs, and graph neural network-based approaches have been proposed and shown promising performance. However, most of these approaches assume the underlying dynamics does not change over time, which is unfortunately untrue. For example, a molecular dynamics can be affected by the environment temperature over the time. In this paper, we take an attempt to provide a probabilistic view for time-varying dynamics and propose a model Context-attended Graph ODE (CARE) for modeling time-varying interacting dynamical systems. In our CARE, we explicitly use a context variable to model time-varying environment and construct an encoder to initialize the context variable from historical trajectories. Furthermore, we employ a neural ODE model to depict the dynamic evolution of the context variable inferred from system states. This context variable is incorporated into a coupled ODE to simultaneously drive the evolution of systems. Comprehensive experiments on four datasets demonstrate the effectiveness of our proposed CARE compared with several state-of-the-art approaches.

1 Introduction

Modeling interacting dynamical systems is a fundamental machine learning problem [66, 65, 23, 59, 37] with a wide range of applications, including social network analysis [12, 17, 36] and particlebased physical simulations [46, 43, 35]. Geometric graphs [27] are utilized to formalize these interactions between objects. For example, in particle-based physical systems, edges are constructed based on the geographical distance between particles, which represents the transfer of energy.

In the literature, numerous data-driven approaches have been proposed for understanding interacting dynamical systems [2, 25, 49]. Among them, graph neural networks [26, 62, 72, 33, 13] (GNNs) are widely utilized to predict trajectories at the next timestamp due to their strong capacity to capture interactions in graph-structured data. In particular, each object is considered as a graph node, and edges represent interactions between neighboring objects. Given the observations and their corresponding graph structure, these methods forecast states in the next timestamp using the message passing mechanism. This process involves aggregating information from the neighbors of each node to update its representation in an iterative fashion, effectively capturing the dynamics of the system.

Although impressive progress has been achieved on GNN-based approaches, capturing long-term dependency in interacting dynamical systems is still a practical but underexplored problem. Existing next-step predictors [42, 46, 45] can send the predictions back to generate rollout trajectories, which could suffer from serious error accumulation for long-term predictions. More importantly, system

environments and relational structures could be changeable [69] (e.g., unsteady flow [11, 16]), which implies the potential temporal distribution variation during the evolution. In particular, in physical systems, there are various potential factors which can influence the trajectories extensively. For example, high temperatures [69] or pressure [68] could speed up molecular movement. Their continuous variation would make understanding interacting dynamic systems more challenging. First, temporal environmental variation would indicate different data distributions over the time [52], which requires the model equipped with superior generalization capability. In contrast, existing methods typically focus on in-distribution trajectories [7, 1, 58, 3, 21, 66], which would perform worse when it comes to out-of-distribution data. Second, recent out-of-distribution generalization methods [50, 39, 57, 64, 44] usually focus on vision and text with discrete shift across different domains. However, our scenarios would face the continuous distribution variation, which is difficult to capture in interacting dynamical systems.

In this paper, we propose a novel method named Context-attended Graph ODE (CARE) to capture interacting system dynamics. The core of our CARE approach is to characterize the temporal environmental variation by introducing the context variable. In particular, we propose a probability model to depict the interaction between the context variable and trajectories. Based on the probabilistic decomposition, we divide each training sequence into two parts for initializing embeddings and predictions. Here, we first construct a temporal graph and then leverage an attention-based encoder to generate node representations and the context representation from spatial and temporal signals simultaneously. More importantly, we introduce coupled ODEs to model the dynamic evolution of node representations and the context variable. On the one hand, we adopt a graph-based ODE system enhanced with context information to drive the evolution. On the other hand, the context information can also be updated using summarized system states and the current context. We also provide a theoretical analysis that indicates, at least locally, the future system trajectory and context information are predictable based on their historical values. Finally, we introduce efficient dynamical graph updating and robust learning strategies to enhance the generalization capability and efficiency of the framework, respectively. Extensive experiments on various dynamical systems demonstrate the superiority of our proposed CARE compared with state-of-the-art approaches.

To summarize, in this paper we make the following contributions: (1) Problem Formalization . We formalize the problem of temporal environmental variation in interacting dynamics modeling. (2) Novel Methodology . We analyze the problem under a probabilistic framework, and propose a novel approach CARE, which incorporates the continuous context variations and system states into a coupled ODE system. (3) Extensive Experiments. Extensive experiments conducted on four datasets validate the superiority of our CARE. The performance gain of our proposed CARE over the best baseline is up to 36.35%.

2 Related Work

Interacting Dynamics Modeling. Deep learning approaches have been extensively used in recent years to model interacting systems across various fields [9, 53, 29, 20], including molecular dynamics and computational fluid dynamics. Early efforts focus on incorporating convolutional neural networks to learn from interacting regular grids [41]. To address more generalized scenarios, graph neural network (GNN) methods have been developed [42, 46, 45], leveraging message passing mechanisms to extract complex spatial signals. However, these methods often fail to account for environmental fluctuations, which hinders their ability to make reliable long-term predictions. In contrast, our CARE adopts a context-attended ODE architecture to explicitly represent both the observations and the underlying environment, enabling the generation of accurate future trajectories.

Neural Ordinary Differential Equations (ODEs). Drawing inspiration from the approximation of ResNet [5], neural ODEs equip neural networks with a continuous-depth architecture by parameterizing the derivatives of hidden states. Several attempts have been made to increase the expressiveness of neural ODEs [65], including adding regularization [8] and designing high-order ODEs [61]. Neural ODEs have also been incorporated into GNNs, producing continuous message passing layers to avoid oversmoothing [60] and increase the interpretability of predictions [70]. In this study, we employ a graph ODE architecture to capture the continuous nature of interacting system dynamics, relieving potential error accumulation caused by discrete prediction models.

...

...

...

...

Out-of-distribution (OOD) Generalization. OOD generation aims to make models more effective when the training and testing distributions diverge [50, 39, 57, 63]. This problem has drawn considerable attention in several areas, including text and vision [48]. One effective solution is to learn domain-invariant representations in the hidden space [31, 55, 30], which has been achieved under the guidance of invariant learning theory [6, 32]. Additionally, uncertainty modeling [34], causal learning [54, 14], and model selection [38, 56] are employed to improve the performance. Interacting systems inherently exhibit dynamic distribution variation caused by environmental changes, an aspect that remains underexplored in the literature. To address this, our paper proposes a novel approach named CARE to model context information from the perspective of a probabilistic model.

3 Background

3.1 Problem Definition

In a multi-agent dynamical system, the state at time t is represented by G t = ( V, E t , X t ) , where each node in V corresponds to an object, E t denotes the current edge set, and X t signifies the node attribute matrix. Specifically, the state vector for each i ∈ V is given by x t i = [ p t i , q t i , a i ] , with p t i ∈ R 3 and q t i ∈ R 3 representing the position and velocity, respectively, and a i representing static attributes. We are given the sequence { G 0 , G 1 , · · · , G t } and aim to learn a model that produces the target dynamic states Y s ( s > t ) (e.g., velocity), which are part of X s at the corresponding time. The temporal environmental variation would result in data distribution changes during the evolution of interacting systems. If we utilize C 0: t to indicate the dynamical environment factor till timestamp t , we have data from variable distribution, i.e., ( G 0: t , Y s ) ∼ P ( G 0: t , Y s | C 0: t ) . Thus, we must take these changes into account for accurate trajectory predictions.

3.2 GNNs for Modeling Dynamical Systems

Graph neural networks (GNNs) are extensively employed in dynamical system modeling to investigate the interactive relationships between objects [42, 46, 45]. These methods typically use the current states to predict the states of nodes at the next timestamp. Specifically, omitting the time notation, GNNs first initialize node representations and edge representations using encoders:

where f v ( · ) and f e ( · ) are two encoders for node and edge representations, respectively. Then, they utilize two propagation modules to update these representations at the l -th layer, i.e., v ( l ) i and e ( l ) ij following the message passing mechanism:

where N i collects the neighbours of node i . ϕ e ( · ) and ϕ v ( · ) are two functions for representation updating. Finally, they generate the target velocity vectors at the next timestamp using a decoder.

4 Methodology

In this paper, we propose a novel method called CARE for modeling interacting dynamics under temporal environmental variation. We start by formalizing a probabilistic model to understand the relationships between trajectories and contexts. Based on this foundation, we construct a spatiotemporal encoder to initialize the representations of nodes and contexts. Then, to simultaneously model their evolution, a coupled graph ODE is introduced where node representations are evolved with the guidance of contexts and the states of the context variable are inferred from current trajectories. Additionally, we introduce a regularization term and dynamic graph updating strategies to enhance our framework. An illustration of our CARE can be found in Figure 1.

4.1 Probabilistic Model for System Dynamics under Temporal Distribution Drift

In this work, to tackle the challenge brought by temporal distribution drift, we inject a context variable c t in our dynamic system modeling, which indicates the environment state at timestamp t . For example, the context variable could indicate flow speed, density and viscosity in fluid dynamics.

Here, we make two basic assumptions in our probabilistic model.

Assumption 4.1. (Independence-I) The context variable is independent of the sequences before the last observed timestamp, i.e., P ( c t | c t -k , G 0: t ) = P ( c t | c t -k , G t -k : t ) , where t -k is the last observed timestamp.

Assumption 4.2. (Independence-II) Given the current states and contexts, the future trajectories are independent of the previous trajectories and contexts, i.e., P ( Y t -k : t + l | G 0: t -k , c 0: t -k ) = P ( Y t -k : t -k + l | G t -k , c t -k ) where l is the length of the prediction.

Then, we can have the following lemma:

Lemma 4.1. With Assumptions 4.1 and 4.2, we have:

The proof of Lemma 4.2 can be found in Appendix. From Lemma 4.1, we decompose the probability P ( Y t | G 0: t -1 ) into three terms. Specifically, the last term necessitates encoding context information based on the historical trajectory G 0: t -k . The second term aims to update the context vector according to the recent trajectory G t -k : t -1 . The first term suggests using the context variable in conjunction with the current states to make predictions for the next timestamp. Besides making a single next-time prediction, our model can also predict trajectories ( Y t -k , Y t -k +1 , · · · , Y t ) by modifying Eqn. 14.

Consequently, we divide each training sequence into two parts, namely [0 , t -k ] and ( t -k, t ] as in [19, 18]. The first part is used to encode contexts and nodes in the system, while the second part serves for updating and prediction purposes.

4.2 Context Acquirement from Spatio-temporal Signals

In this part, our goal is to acquire knowledge from the historical trajectory, i.e., { G 0 , · · · , G t -k } to encode contexts and nodes for initialization. To be specific, we first construct a temporal graph to capture spatial and temporal signals simultaneously. Subsequently, we employ the attention mechanism to update temporal node representations, which will be used to initialize the context representation and node representations for { G t -k , · · · , G T } .

To begin, we construct a temporal graph containing two types of edges, i.e., spatial and temporal edges. Spatial edges are built when the distance between two nodes at the same timestamp is less than a threshold while temporal edges are between every two consecutive observations for each node. Specifically, in the constructed temporal graph G H , there are N ( t -k +1) nodes { i s } s ∈ [0: t -k ] ,i ∈ V

in total. The adjacent matrix A contains both spatial and temporal edges as follows:

where d s ( i, j ) denotes the distance between particles i and j at timestamp s and τ is the predefined threshold. Then, we utilize an attention-based GNN to extract spatio-temporal relationships into node representations. Here, we first compute the interaction scores between each node in G H with its neighboring nodes, and then aggregate their embeddings at the previous layer. Let d represent the hidden dimension, the interaction score between i s and j s ′ at layer l is:

where W query ∈ R d × d and W key ∈ R d × d are two matrices to map temporal node representations into different spaces. ⋆ computes the cosine similarity between two vectors. With the interaction scores, we can compute temporal node representations at the next layer:

where W value is a weight matrix, N ( i s ) collects all the neighbours of i s and σ ( · ) is an activation function. After stacking L layers, we add temporal encoding and then summarize all these temporal node representations to initialize node representations for the upcoming ODE module:

where TE( s )[2 i ] = sin ( s 10000 2 i/d ) , TE( s )[2 i +1] = cos ( s 10000 2 i/d ) and W sum denotes a projection matrix. The initial context variable c t -k is driven by summarizing all node representations:

where W context is a learnable matrix and β t i calculates the attention score for each node.

4.3 Context-attended Graph ODE

In this module, to model continuous evolution, we incorporate an ODE system into our approach. The precondition requires assuming that both the context variable and node representations are continuous to fit neural ODE models, which inherently holds for common dynamical systems in practice. We then introduce coupled ODEs to model the dynamic evolution of node representations and the context variable. Specifically, the context variable can be inferred during the evolution of node representations, which in turn drives the evolution of the system. We first introduce the assumption:

Assumption 4.3. (Continuous) We assume that both context variable c s and node representations v s i are continuously differentiable with respect to s .

Then, to utilize the context variable and the current state for making future predictions, we introduce a graph ODE model. Let ˆ A s denote the adjacency matrix at timestamp s with self-loop, we have:

where N s ( i ) denotes the neighbours of node i at timestamp s and ˆ D s i represents the degree of node i according to ˆ A s . The first term in Eqn. 9 aggregates information from its instant neighbors and the second term captures information from the current context information.

The next question is how to model the evolution of c s . Notice that we have:

where ∆ t denotes a small time interval. With Assumption 4.3, we can simplify P ( c t -k +∆ t | c t -k , G t -k : t -k +∆ t ) into P ( c t -k +∆ t | c t -k , V t -k , d V t -k ) where V t -k denotes the node embedding matrix at timestamp t -k and d V t -k is its differentiation. On this basis, we introduce another ODE to update the context variable as:

where Φ c is an MLP with the concatenated input and AGG( · ) is an operator to summarize node representations such as averaging and sum. Compared to previous methods, the key to our CARE is to take into account the mutual impact between the environment and the trajectories, and model their evolution simultaneously by coupling Eqn. 9 and Eqn. 11. We also provide a theoretical analysis of the uniqueness of the solution to our system. To simplify the analysis, we set AGG( · ) to summation and rewrite Eqn. 11 with learnable matrices W 3 , W 4 and W 5 as:

Then, we introduce the following assumption:

Assumption 4.4. All time-dependent coefficients in Eqn. 9, i.e A t ij , ˆ D t i are continuous with respect to t and bounded by a constant C > 0 . All parameters in the weight matrix are also bounded by a constant W > 0 .

With Assumption 4.4, we can deduce the following lemma:

Lemma 4.2. Given the initial state ( t 0 , v t 0 1 , · · · , v t 0 N , c t 0 ) , we claim that there exists ε > 0, s.t. the ODE system 9 and 12 has a unique solution in the interval [ t 0 -ε, t 0 + ε ] .

The proof of Lemma 4.2 can be found in Appendix. Our theoretical analysis indicates that at least locally, the future system trajectory and context information are predictable based on their historical values [51], which is also an important property for dynamical system modeling [5, 28].

4.4 Decoder and Optimization

Decoder. We introduce an MLP Φ d ( · ) to predict both the position and velocity vectors using corresponding node representations, i.e., [ ˆ p s i , ˆ q s i ] = Φ d ( v s i ) .

Dynamic Graph Updating. We can estimate the instant distance between nodes using the encoder and then construct the graphs, which could suffer from a large computational burden. To improve the efficiency of graph construction during ODE propagation, we not only update the graph structure every ∆ s , and but also introduce a graph updating strategy that calculates the distance between first-order and second-order neighbors in the last graph. By doing so, we can delete edges between first-order neighbors and add edges between second-order neighbors, reducing quadratic complexity to linear complexity in sparse graphs. We will also validate this empirically.

Learning Objective. Given the ground truth, we first minimize the mean squared error (MSE) of the predicted trajectory. Moreover, we require both node and context representations to be robust to noise attacks to improve the robustness of the ODE system. The overall objective is written as:

where ˆ Y s denotes the predictions from the encoder and η is a parameter set to 0 . 1 to balance two losses. ˜ V s and ˜ c s denote the perturbed representations under noise attack to the input.

5 Experiments

We evaluate our proposed CARE on both particle-based and mesh-based physical systems. To ensure the accuracy of our results, we use a rigorous data split strategy, where first 80 % of the samples are reserved for training purposes and the remaining 10 % are set aside for testing and validating,

Prediction Length

+1

+5

+10

+20

Variable

v

x

v

y

v

z

v

x

v

y

v

z

v

x

v

y

v

z

v

x

v

y

v

z

Lennard-Jones Potential

LSTM

3.95

3.92

3.68

9.12

9.21

9.15

10.84

10.87

10.76

14.82

14.94

14.67

GNS

3.28

3.75

3.39

7.97

8.05

7.68

10.09

10.15

10.13

13.65

13.62

13.59

STGCN

2.91

3.08

2.95

5.06

5.17

5.11

6.89

6.90

6.93

9.31

9.32

9.44

MeshGraphNet

2.89

3.13

2.94

5.29

5.53

5.28

7.03

7.09

7.11

9.12

9.21

9.24

CG-ODE

1.79

2.05

1.71

3.47

3.92

3.38

5.46

5.99

5.36

9.03

9.26

8.92

TIE

1.62

1.98

1.47

3.25

3.90

3.15

5.24

5.82

5.17

8.24

8.34

8.47

Ours

0.76

0.89

1.01

2.94

3.16

2.85

5.01

4.69

4.71

5.75

5.91

5.82

3-body Stillinger-Weber Potential

LSTM

17.11

17.14

17.18

23.64

23.69

23.60

25.46

25.42

25.48

28.44

28.45

28.44

GNS

15.39

15.27

15.33

22.14

22.19

22.17

25.29

25.36

25.31

27.18

27.15

27.14

STGCN

12.33

12.31

12.35

17.94

17.96

17.91

20.08

20.14

20.13

23.49

23.51

23.52

MeshGraphNet

12.16

12.10

12.13

18.33

18.38

18.34

20.65

20.62

20.71

23.62

23.54

23.61

CG-ODE

9.78

9.74

9.75

12.11

12.05

12.14

15.55

15.58

15.50

16.17

16.24

16.22

TIE

10.18

10.26

10.19

14.75

14.70

14.73

18.42

18.45

18.41

20.92

21.04

21.36

Ours

4.21

4.29

4.18

9.74

9.79

9.71

13.65

13.71

13.57

15.30

15.39

15.35

CARE

Ground

Truth

separately. During training, we split each trajectory sample into two parts, i.e., a conditional part and a prediction part. We initialize node representations and the context representation based on the first part and utilize the second part to supervise the model. The size of the two parts is represented as conditional length and prediction length, respectively. Our approach is compared with various baselines for interacting systems modeling, i.e., LSTM [15], STGCN [67], GNS [45], MeshGraphNet [42], TIE [46] and CG-ODE [19].

5.1 Performance on Particle-based Physical Simulations

Datasets. We evaluate our proposed CARE on two particle-based simulation datasets with temporal environmental variations, i.e., Lennard-Jones Potential [47] and 3-body Stillinger-Weber Potential [4]. Lennard-Jones Potential is popular in modeling electronically neutral atoms or molecules. 3-body Stillinger-Weber Potential provides more complex relationships in atom systems The temperature in two particle-based systems is continuously changed along with the time to model the environmental variations. The objective is to predict the future velocity values in all directions, i.e., v x , v y and v z . More details can be found in Appendix.

Performance Comparison. We evaluate the performance in terms of RMSE with different prediction lengths. Table 1 show the compared results on these two datasets. We can observe that our proposed CARE outperforms all the baselines on two datasets. In particular, compared with TIE, CARE accomplishes an error reduction of 24.03% and 36.35% on two datasets, respectively. The remarkable performance can be attributed to two factors: (1) Introduction of the context variable. Our CARE infers the context states during the evolution of the system, which can help the model understand environmental variations. (2) Introduction of robust learning. We add noise attack to both system and context states, which improves the model generalization to potential distribution changes.

Prediction Length

+1

+10

+20

+50

Variable

v

x

v

y

p

v

x

v

y

p

v

x

v

y

p

v

x

v

y

p

CylinderFlow

LSTM

3.35

29.4

12.5

7.06

44.8

17.8

9.47

49.5

19.9

14.3

73.6

42.3

GNS

3.12

28.8

11.9

7.18

44.3

17.3

9.01

49.6

19.2

13.5

73.2

41.6

STGCN

2.68

26.7

11.0

5.47

42.1

16.9

6.72

45.6

18.4

9.15

68.7

40.0

MeshGraphNet

1.75

22.4

10.6

4.09

39.7

15.7

5.38

44.5

17.2

7.92

64.3

37.7

CG-ODE

1.05

20.4

8.51

3.44

36.8

13.6

4.15

38.5

17.1

5.14

61.2

32.3

TIE

1.22

20.8

8.94

3.75

35.2

13.0

4.62

40.6

16.0

5.87

59.5

32.1

Ours

0.87

19.1

7.21

3.02

32.9

11.8

3.95

37.8

13.9

4.97

55.8

29.4

Airfoil

LSTM

7.49

7.73

1.92

8.86

9.02

3.78

10.8

11.0

4.71

14.9

15.7

4.96

GNS

6.95

7.14

1.69

8.20

8.34

3.34

10.2

10.5

3.98

14.2

14.1

4.11

STGCN

6.24

5.35

1.07

6.57

6.51

2.33

7.88

8.01

3.16

11.6

11.8

3.17

MeshGraphNet

4.72

4.68

0.50

5.89

5.74

1.23

6.32

6.48

1.85

9.03

9.12

2.08

CG-ODE

4.26

4.32

0.35

4.78

4.70

0.46

5.81

5.66

1.04

7.39

7.85

1.69

TIE

4.17

4.39

0.33

4.99

4.86

0.51

5.75

5.62

0.95

7.25

7.63

1.44

Ours

3.51

4.11

0.19

3.86

3.75

0.34

4.16

4.12

0.45

6.74

6.82

0.81

CARE

TIE

Ground

Truth

Figure 3: Visualization of the CylinderFlow Dataset with multiple timestamps. We render the velocity in the x -axis in the fluid field of our CARE and the ground truth.

Visualization. Figure 2 visualizes the prediction of positions in comparison to the ground truth on Lennard-Jones Potential . Here, we sample six timestamps in every trajectory to validate the performance of both short-term and long-term predictions. From the qualitative results, we can observe that in the first three frames, the particle motion is not strenuous due to low temperature in the system. Surprisingly, our proposed CARE can always make faithful physical simulations close to the ground truth even though the system environment is highly variable.

5.2 Performance on Mesh-based Physical Simulations

Datasets. We employ two popular mesh-based simulation datasets, i.e., CylinderFlow , and Airfoil . CylinderFlow consists of simulation data from modeling an incompressible flow governed by the Navier-Stokes equations. Notably, the initial flow velocity of the incoming water flow to the cylinder varies cyclically over time, meaning the Reynolds number of the flow field also changes periodically. Airfoil is generated in a similar manner through simulations of a compressible flow, wherein the inlet velocity over the wing varies cyclically over time. We aim to forecast the velocity values v x and v y , as well as the pressure p . More details can be found in Appendix.

Performance Comparison. The performance with respect to different variables is recorded in Table 2. From the results, we can observe that the average performance of the proposed CARE is over the best baseline by 12.99% and 22.78% on two datasets, respectively. Note that unsteady flow [11, 16] is a crucial problem in recent fluid dynamics, our proposed CARE can benefit abundant complex mesh-based simulations under environmental variations.

Visualization. Moreover, we show the qualitative results of the best baseline and our CARE in comparison to the ground truth. From the results, we can observe that our CARE can capture more accurate signals in unsteady fluid dynamics. In particular, in the last two frames with complicated

(a)

(b)

(c)

(d)

Error

Root Mean Square Error

Error

16

2.5

CARE(+50)

CARE(+20)

CG-ODE(+50)

8

CARE(+20)

CARE(+50)

CG-ODE(+50)

LJP(+20)

SWP(+20)

LJP(+10)

2.0

CARE

CARE E

6

CG-ODE(+20)

7

CG-ODE(+20)

6

SWP(+10)

15

(s)

1.5

5

6

5

14

Time

1.0

5

4

0.5

10

15

20

25

30

4

10

15

20

25

30

4

1

2

3

4

5

13

0.0

LJP

SWP

ConditionLength

ConditionLength

Interval

Figure 4: (a), (b) The performance with respect to different condition and prediction lengths on CylinderFlow and Airfoil. (c) The sensitivity of interval on Lennard-Jones Potential (LJP) and 3-body Stillinger-Weber Potential (SWP) datasets. (d) The comparison of running time for our dynamic graph updating and full pairwise calculation on two particle-based datasets.

structures, our CARE still can generate superior simulations in both scenarios under potential environmental variation while the baseline fails, which shows the superiority of our proposed CARE.

5.3 Further Analysis

Ablation Study. To analyze the effectiveness of different components in our CARE, we introduce two different variants: (1) CARE V1, which removes the context variable in Eqn. 9; (2) CARE V2, which removes the robust learning term in Eqn. 13. The compared performance is recorded in Figure 3 and we have two observations. First, our full model outperforms CARE V1, which indicates the incorporation of the context variable would benefit interacting system modeling under temporal environmental variation. Second, without the robust learning term, the performance would get worse, implying that improving the robustness can also benefit tackling the distribution changes.

Parameter Sensitivity. We begin by analyzing the performance with respect to different condition lengths and prediction lengths. Here the condition length and prediction length vary from {10,15,20,25,30}, {20,50}, respectively. From the results in Figure 4 (a) and (b), we can observe that our proposed CARE can always achieve superior performance compared with CG-ODE. Moreover, we can observe that a longer condition length would benefit the performance in most cases due to more provided information. It can also be seen that a smaller interval for graph updating would improve the performance before saturation from Figure 4 (c).

Efficiency. To show the efficiency of our proposed dynamic graph updating, we propose a model variant named CARE E, which calculates all pairwise distances to update graph structure during the evolution. The computational cost is recorded in Figure 4 (d) and we can observe that our strategy can reduce a large number of computational costs, which validates the complexity analysis before.

6 Conclusion

This paper studies the problem of modeling interacting dynamics under temporal environmental variation and we propose a probabilistic framework to depict the dynamical system. Then, a novel approach named CARE is proposed. CARE first constructs an encoder to initialize the context variable indicating the environment and then utilizes a coupled ODE system, which combines both the context variable and node representation to drive the evolution of the system. Finally, we introduce both efficient dynamical graph updating and robust learning strategies to enhance our framework. Extensive experiments on four datasets validate the superiority of our CARE.

Broader Impacts and Limitations. This work presents an effective learning-based model for interacting dynamical systems under temporal environmental variation, which can benefit complex physical simulations such as unsteady flow. Moreover, our study provides a new perspective on

modeling environmental variations for fluid dynamics and intermolecular interactions. One potential limitation is that our CARE cannot directly fit more physical scenarios requiring abundant external knowledge. In future work, we would extend our CARE to more complicated applications such as rigid dynamics.

References

Kelsey R Allen, Yulia Rubanova, Tatiana Lopez-Guevara, William Whitney, Alvaro SanchezGonzalez, Peter Battaglia, and Tobias Pfaff. Learning rigid dynamics with face interaction graph networks. arXiv preprint arXiv:2212.03574 , 2022.

Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In NeurIPS , 2016.

Suresh Bishnoi, Ravinder Bhattoo, Jayadeva Jayadeva, Sayan Ranu, and NM Anoop Krishnan. Enhancing the inductive biases of graph neural ode for modeling physical systems. In ICLR , 2023.

W Michael Brown and Masako Yamada. Implementing molecular dynamics on hybrid high performance computers-three-body potentials. Computer Physics Communications , 184(12):2785-2793, 2013.

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In NeurIPS , 2018.

Yongqiang Chen, Yonggang Zhang, Yatao Bian, Han Yang, MA Kaili, Binghui Xie, Tongliang Liu, Bo Han, and James Cheng. Learning causally invariant representations for out-ofdistribution generalization on graphs. In NeurIPS , pages 22131-22148, 2022.

Yitong Deng, Hong-Xing Yu, Jiajun Wu, and Bo Zhu. Learning vortex dynamics for fluid inference and prediction. arXiv preprint arXiv:2301.11494 , 2023.

Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. In NeurIPS , 2019.

Zheng Fang, Qingqing Long, Guojie Song, and Kunqing Xie. Spatial-temporal graph ode networks for traffic flow forecasting. In KDD , pages 364-373, 2021.

Jayesh K Gupta, Sai Vemprala, and Ashish Kapoor. Learning modular simulations for homogeneous systems. In NeurIPS , 2022.

Rohit Gupta and Phillip J Ansell. Unsteady flow physics of airfoil dynamic stall. AIAA journal , 57(1):165-175, 2019.

Nesrine Hafiene, Wafa Karoui, and Lotfi Ben Romdhane. Influential nodes detection in dynamic social networks: A survey. Expert Systems with Applications , 159:113642, 2020.

Mingguo He, Zhewei Wei, and Ji-Rong Wen. Convolutional neural networks on graphs with chebyshev approximation, revisited. In NeurIPS , 2022.

Yue He, Zimu Wang, Peng Cui, Hao Zou, Yafeng Zhang, Qiang Cui, and Yong Jiang. Causpref: Causal preference learning for out-of-distribution recommendation. In WWW , pages 410-421, 2022.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation , 9(8):1735-1780, 1997.

C Huang, L Zhao, JP Niu, JJ Di, JJ Yuan, QL Zhao, FQ Zhang, ZH Zhang, JM Lei, and GP He. Coupled particle and mesh method in an euler frame for unsteady flows around the pitching airfoil. Engineering Analysis with Boundary Elements , 138:159-176, 2022.

Chao Huang, Xian Wu, Xuchao Zhang, Chuxu Zhang, Jiashu Zhao, Dawei Yin, and Nitesh V Chawla. Online purchase prediction via multi-scale modeling of behavior dynamics. In KDD , pages 2613-2622, 2019.

Zijie Huang, Yizhou Sun, and Wei Wang. Learning continuous system dynamics from irregularly-sampled partial observations. In NeurIPS , pages 16177-16187, 2020.

Zijie Huang, Yizhou Sun, and Wei Wang. Coupled graph ode for learning interacting system dynamics. In KDD , 2021.

Zijie Huang, Yizhou Sun, and Wei Wang. Generalizing graph ode for learning complex system dynamics across environments. In KDD , pages 798-809, 2023.

Steeven Janny, Aurélien Beneteau, Nicolas Thome, Madiha Nadri, Julie Digne, and Christian Wolf. Eagle: Large-scale learning of turbulent fluid dynamics with mesh transformers. arXiv preprint arXiv:2302.10803 , 2023.

Hrvoje Jasak. Openfoam: open source cfd in research and industry. International Journal of Naval Architecture and Ocean Engineering , 1(2):89-94, 2009.

Ruoxi Jiang and Rebecca Willett. Embed and emulate: Learning to estimate parameters of dynamical systems with uncertainty quantification. In NeurIPS , 2022.

Patrick Kidger, Ricky T. Q. Chen, and Terry J. Lyons. "hey, that's not an ode": Faster ode adjoints via seminorms. ICML , 2021.

Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In ICML , pages 2688-2697, 2018.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR , 2017.

Miltiadis Kofinas, Naveen Nagaraja, and Efstratios Gavves. Roto-translated local coordinate frames for interacting dynamical systems. In NeurIPS , 2021.

Lingkai Kong, Jimeng Sun, and Chao Zhang. Sde-net: Equipping deep neural networks with uncertainty estimates. In ICML , 2020.

Shiyong Lan, Yitong Ma, Weikang Huang, Wenwu Wang, Hongyu Yang, and Pyang Li. Dstagnn: Dynamic spatial-temporal aware graph neural network for traffic flow forecasting. In ICML , pages 11906-11917, 2022.

Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In CVPR , pages 5400-5409, 2018.

Haoliang Li, YuFei Wang, Renjie Wan, Shiqi Wang, Tie-Qiang Li, and Alex Kot. Domain generalization for medical imaging classification with linear-dependency regularization. In NeurIPS , pages 3118-3129, 2020.

Haoyang Li, Ziwei Zhang, Xin Wang, and Wenwu Zhu. Learning invariant graph representations for out-of-distribution generalization. In NeurIPS , 2022.

Xiang Li, Renyu Zhu, Yao Cheng, Caihua Shan, Siqiang Luo, Dongsheng Li, and Weining Qian. Finding global homophily in graph neural networks when meeting heterophily. In ICML , pages 13242-13256, 2022.

Xiaotong Li, Yongxing Dai, Yixiao Ge, Jun Liu, Ying Shan, and Ling-Yu Duan. Uncertainty modeling for out-of-distribution generalization. arXiv preprint arXiv:2202.03958 , 2022.

Xuan Li, Yadi Cao, Minchen Li, Yin Yang, Craig Schroeder, and Chenfanfu Jiang. Plasticitynet: Learning to simulate metal, sand, and snow for optimization time integration. In NeurIPS , pages 27783-27796, 2022.

Siyuan Liao, Shangsong Liang, Zaiqiao Meng, and Qiang Zhang. Learning dynamic embeddings for temporal knowledge graphs. In WSDM , pages 535-543, 2021.

Andreas Look, Melih Kandemir, Barbara Rakitsch, and Jan Peters. Cheap and deterministic inference for deep state-space models of interacting dynamical systems. arXiv preprint arXiv:2305.01773 , 2023.

Wang Lu, Jindong Wang, Yidong Wang, Kan Ren, Yiqiang Chen, and Xing Xie. Towards optimization and model selection for domain generalization: A mixup-guided solution. arXiv preprint arXiv:2209.00652 , 2022.

Lucas Mansilla, Rodrigo Echeveste, Diego H Milone, and Enzo Ferrante. Domain generalization via gradient surgery. In ICCV , pages 6630-6638, 2021.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.

Jiang-Zhou Peng, Siheng Chen, Nadine Aubry, Zhihua Chen, and Wei-Tao Wu. Unsteady reduced-order model of flow over cylinders based on convolutional and deconvolutional neural network structure. Physics of Fluids , 32(12):123609, 2020.

Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W Battaglia. Learning mesh-based simulation with graph networks. In ICLR , 2021.

Lukas Prantl, Benjamin Ummenhofer, Vladlen Koltun, and Nils Thuerey. Guaranteed conservation of momentum for learning particle-based fluid dynamics. arXiv preprint arXiv:2210.06036 , 2022.

Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of-distribution generalization. In ICML , pages 18347-18377, 2022.

Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In ICML , pages 8459-8468, 2020.

Yidi Shao, Chen Change Loy, and Bo Dai. Transformer with implicit edges for particle-based physics simulation. In ECCV , pages 549-564, 2022.

BJTJ Smit. Phase diagrams of lennard-jones fluids. The Journal of Chemical Physics , 96(11):8639-8640, 1992.

Yaguang Song, Xiaoshan Yang, Yaowei Wang, and Changsheng Xu. Recovering generalization via pre-training-like knowledge distillation for out-of-distribution visual question answering. IEEE Transactions on Multimedia , 2023.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In ICLR , 2018.

Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In NeurIPS , 2018.

Charles Vorbach, Ramin Hasani, Alexander Amini, Mathias Lechner, and Daniela Rus. Causal navigation by continuous-time neural networks. In NeurIPS , pages 12425-12440, 2021.

Rui Wang, Yihe Dong, Sercan Ö Arik, and Rose Yu. Koopman neural forecaster for time series with temporal distribution shifts. arXiv preprint arXiv:2210.03675 , 2022.

Xiaoyang Wang, Yao Ma, Yiqi Wang, Wei Jin, Xin Wang, Jiliang Tang, Caiyan Jia, and Jian Yu. Traffic flow prediction via spatial temporal graph neural network. In WWW , pages 1082-1092, 2020.

Yuqing Wang, Xiangxian Li, Zhuang Qi, Jingyu Li, Xuelong Li, Xiangxu Meng, and Lei Meng. Meta-causal feature learning for out-of-distribution generalization. In ECCV , pages 530-545, 2022.

Ziqi Wang, Marco Loog, and Jan van Gemert. Respecting domain relations: Hypothesis invariance for domain generalization. In ICPR , pages 9756-9763, 2021.

Florian Wenzel, Andrea Dittadi, Peter Gehler, Carl-Johann Simon-Gabriel, Max Horn, Dominik Zietlow, David Kernert, Chris Russell, Thomas Brox, Bernt Schiele, et al. Assaying out-ofdistribution generalization in transfer learning. In NeurIPS , pages 7181-7198, 2022.

Qitian Wu, Hengrui Zhang, Junchi Yan, and David Wipf. Handling distribution shifts on graphs: An invariance perspective. arXiv preprint arXiv:2202.02466 , 2022.

Tailin Wu, Takashi Maruyama, Qingqing Zhao, Gordon Wetzstein, and Jure Leskovec. Learning controllable adaptive simulation for multi-resolution physics. In ICLR , 2023.

Tailin Wu, Qinchen Wang, Yinan Zhang, Rex Ying, Kaidi Cao, Rok Sosic, Ridwan Jalali, Hassan Hamam, Marko Maucec, and Jure Leskovec. Learning large-scale subsurface simulations with a hybrid graph network simulator. In KDD , pages 4184-4194, 2022.

Louis-Pascal Xhonneux, Meng Qu, and Jian Tang. Continuous graph neural networks. In ICML , pages 10432-10441, 2020.

Hedi Xia, Vai Suliafu, Hangjie Ji, Tan Nguyen, Andrea Bertozzi, Stanley Osher, and Bao Wang. Heavy ball neural ordinary differential equations. In NeurIPS , pages 18646-18659, 2021.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR , 2019.

Chenxiao Yang, Qitian Wu, Qingsong Wen, Zhiqiang Zhou, Liang Sun, and Junchi Yan. Towards out-of-distribution sequential event prediction: A causal treatment. arXiv preprint arXiv:2210.13005 , 2022.

Nianzu Yang, Kaipeng Zeng, Qitian Wu, Xiaosong Jia, and Junchi Yan. Learning substructure invariance for out-of-distribution molecular representations. In NeurIPS , 2022.

Ça˘ gatay Yıldız, Melih Kandemir, and Barbara Rakitsch. Learning interacting dynamical systems with latent gaussian process odes. In NeurIPS , pages 9188-9200, 2022.

Yuan Yin, Matthieu Kirchmeyer, Jean-Yves Franceschi, Alain Rakotomamonjy, and Patrick Gallinari. Continuous pde dynamics forecasting with implicit neural representations. In NeurIPS , 2022.

Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: a deep learning framework for traffic forecasting. In IJCAI , pages 3634-3640, 2018.

Xiaofeng Yuan, Lin Li, Yuri AW Shardt, Yalin Wang, and Chunhua Yang. Deep learning with spatiotemporal attention-based lstm for industrial soft sensor model development. IEEE Transactions on Industrial Electronics , 68(5):4404-4414, 2020.

Wei Zhang, Li Zhou, Bin Yang, and Tinggui Yan. Molecular dynamics simulations of licl ion pairs in high temperature aqueous solutions by deep learning potential. Journal of Molecular Liquids , 367:120500, 2022.

Yanfu Zhang, Shangqian Gao, Jian Pei, and Heng Huang. Improving social network embedding via new second-order continuous graph neural networks. In KDD , pages 2515-2523, 2022.

Tong Zhao, Gang Liu, Daheng Wang, Wenhao Yu, and Meng Jiang. Learning from counterfactual links for link prediction. In ICML , pages 26911-26926, 2022.

Xin Zheng, Yixin Liu, Shirui Pan, Miao Zhang, Di Jin, and Philip S Yu. Graph neural networks for graphs with heterophily: A survey. arXiv preprint arXiv:2202.07082 , 2022.

A Proof of Lemma 4.1

Lemma 4.1. With Assumptions 4.1 and 4.2, we have:

Proof. We have:

B Proof of Lemma 4.2

For convenience, Eqn. 9 in the main paper is repeated as:

Eqn. 12 is repeated as:

Lemma 4.2. Given the initial state ( t 0 , v t 0 1 , · · · , v t 0 N , c t 0 ) , we claim that there exists ε > 0, s.t. the ODE system Eqn. 16 and Eqn. 17 has a unique solution in the interval [ t 0 -ε, t 0 + ε ] .

To begin, we introduce the Picard-Lindel ¨ o f Theorem as follows:

Theorem B.1. (Picard-Lindel ¨ o f Theorem) Let D ⊆ R × R n be a closed rectangle with ( t 0 , y 0 ) ∈ D . Let f : D → R n be a function which is continuous with respect to t and Lipschitz continuous with respect to y . Then, there exists some ε > 0 such that the initial value problem:

has a unique solution y ( t ) in the interval [ t 0 -ε, t 0 + ε ] .

Proof. Let A s ij = 0 if j / ∈ N s ( i ) and denote M s ij = ˆ A s ij √ ˆ D s i · ˆ D s j . Then, we transpose them with Eqn. 16 and 17 becoming:

From the Eqn. 19 we get the ODE system d Y s ds = f ( Y s ; θ ) with fixed parameters θ for the ODE solver. Here the function f ( Y t ; θ ) is continuous w.r.t t since all components in the vector Y t are continuous w.r.t t and θ does not depend on t.

Now consider activation functions σ , such as ReLU, that satisfy the following inequality for all x and y :

Then, for any two solutions Y s 1 , Y s 2 , we have:

To simplify the representation, denote:

Then by triangular inequality, we have ∀ i ∈ { 1 , · · · , N } ,

Without loss of generality, we assume constants M,W > 1 . Thus, we have the following result:

Therefore the function f is L-Lipschitz with L = √ NW 2 M 2 +( W + MNW 2 ) 2 . By Picard-Lindel ¨ o f theorem, we prove the uniqueness of the solution.

C Dataset Details

We evaluate our proposed CARE on four physical simulation datasets with temporal environmental variation. All these four datasets involve at least one thousand nodes. Then we introduce the details of these four datasets.

Lennard-Jones Potential (a.k.a. 6-12 potential) is popular in modeling electronically neutral atoms or molecules, which can be formulated as:

where r is the distance between particle pairs, σ denotes the size of the particle, ϵ denotes the depth of the potential well. The first term denotes the attractive force, which decreases as the distance between particles increases. The second term denotes the repulsive force, which increases when two particles are too close. The temperature in the system is changed along with the time to model the environmental variations and a high temperature would bring a more intense molecular motion.

3-body Stillinger-Weber Potential provides more complex relationships besides pairwise relationships in Lennard-Jones Potential . It contains both two-body and three-body terms with the following formulation:

̸

where ϕ 2 ( r ij ) = A ij ϵ ij [ B ij ( σ ij r ij ) p ij -( σ ij r ij ) q ij ] exp ( σ ij r ij -a ij σ ij ) is the two-body term and ϕ 3 ( r ij , r ik , θ ijk ) = λ ijk ϵ ijk [cos θ ijk -cos θ 0 ijk ] 2 exp ( γ ij σ ij r ij -a ij σ ij ) exp ( γ ik σ ik r ik -a ik σ ik ) is the three-body term. The two body term is similar to Lennard-Jones Potential to model the pairwise relationships and the three body term can consider the angles among atom triplets. Similarly, the temperature is changed along with the time to model the environmental variations and a high temperature would also bring a more intense molecular motion.

CylinderFlow is a popular computational fluid dynamics (CFD) simulation dataset, which models the fluid flow around a given cylinder by OpenFoam [22]. It consists of simulation data from modeling an incompressible flow governed by the Navier-Stokes equations. The Reynolds number, denoted by Re, is a dimensionless quantity that characterizes the flow regime of the fluid. It is defined as: Re = ρV D µ , where ρ is the density of the fluid, V is the velocity of the fluid relative to the cylinder, D is the diameter of the cylinder, and µ is the dynamic viscosity of the fluid. The transition from the laminar to turbulent flow usually happens at a critical Reynolds number, which

depends on the geometry of the cylinder and the properties of the fluid. In general, the flow is more likely to be laminar for small cylinders and viscous fluids, and more likely to be turbulent for large cylinders and low-viscosity fluids. Notably, the initial flow velocity V of the incoming water flow to the cylinder varies cyclically over time, meaning the Reynolds number of the flow field also changes periodically.

Airfoil is generated in a similar manner through simulations of a compressible flow by OpenFoam [22]. The lift coefficient of an airfoil relies on a number of factors, e.g., the angle of attack, the shape of the airfoil, and the Reynolds number of the flow. The angle of attack is the one between the chord line of the airfoil (the straight line connecting the leading and trailing edges) and the direction of the incoming flow. The Reynolds number, as mentioned in the previous question, is a dimensionless quantity that characterizes the flow regime of the fluid. It also plays a crucial role in the lift produced by an airfoil, as it determines whether the flow around the airfoil is laminar or turbulent. For laminar flow, the air moves smoothly over the surface of the airfoil, while for turbulent flow, it moves in a chaotic, swirling pattern. The Reynolds number is given by: Re = ρV c µ , where c is the chord length of the airfoil, and µ denotes the viscosity of the fluid. The lift coefficient is typically higher for laminar flow than for turbulent flow, up to a certain point where the flow separates from the airfoil. Notably, in our simulation datasets, the inlet velocity V over the wing also varies cyclically over time.

D Details of Baselines

Our proposed method is compared with a range of competing baselines as follows:

LSTM [15] is a widely recognized approach for sequence prediction problems. It involves three gates, i.e., forget gate, input gate and output gate, enabling the model to acquire knowledge of long-term relationships.

STGCN [67] is a deep learning approach to handle spatial dependencies and temporal dynamics in complicated spatio-temporal data. It involves a recurrent component and a message passing component for effective analysis of spatio-temporal signals.

GNS [45] utilizes a graph to represent a physical dynamical system and then utilizes a message passing neural network to explore complicated dynamics and interactions among multiple objects.

MeshGraphNet [42] characterize each physical system as meshes, followed by graph neural networks to learn interacting dynamics. Moreover, remeshing techniques are adopted to fit the multi-resolution nature in irregular meshes.

CG-ODE [19] models both nodes and edges jointly through two groups of ODEs, which can capture the evolution of both objective and interaction in the system.

TIE [46] attempts to improve the particle-based simulations by decomposing edges into both ends, and introducing abstract nodes to capture global information in the system.

E Algorithm

The whole learning algorithm of CARE is summarized in Algorithm 1.

F Implementation Details

To ensure the accuracy of our results, we use a rigorous data split strategy, where first 80 % of the samples are reserved for training purposes and the remaining 10 % are set aside for testing and validating, separately. Following [19], we also ensure that no sequence overlap exists on training, validation and testing sets. In particular, each of these particle-based datasets consists of 14400 training trajectories, 1800 validation trajectories, and 1800 test trajectories while 7200 training samples, 900 validation samples and 900 test trajectories are for mesh-based datasets. During training, we split each trajectory sample into two parts, i.e., a conditional part and a prediction part. We initialize node representations and the context representation based on the first part and utilize the second part to supervise the model. The size of the two parts is represented as conditional length and prediction length, respectively. We would vary two lengths to show the performance comprehensively.

Algorithm 1 Learning Algorithm of the proposed CARE

Input:

The training sequences { G 0 , G 1 , · · · , G t } .

Output

: The parameters in our CARE.

1: Initialize the parameters in our model;

2: while not convegence do

3: for each training sequence do

4: Divide the sequence into two segments;

5: Build the temporal graph using Eqn. 4;

6: Initialize both node representations and context representations for ODEs using Eqn. 8;

7: Solve the coupled ODEs, i.e., Eqns. 9 and 11;

8: Add noise into the input for perturbed hidden states;

9: Feed these hidden states into a decoder Φ d ( · ) to get the predictions;

10: Calculate the loss in Eqn. 13;

11: Update the parameters in CARE using back propagation;

12: end for

13: end while

In our implementation, we utilize the sum operator as AGG in Eqn. 11. To solve the ODE systems on a time grid which is five times denser than the observed time steps, we employ the fourth-order Runge-Kutta method as in the torchdiffeq Python package [24], using PyTorch [40]. We also use the adjoint method [5] to reduce memory usage. All experiments are conducted on a single NVIDIA A100 GPU. We set the latent dimension to 256 and the dropout rate to 0 . 2 . For optimization, we use the Adam optimizer with weight decay by mini-batch stochastic gradient descent, setting the learning rate to 0 . 01 . Overall, our proposed CARE offers a novel approach for modeling and predicting complex systems with multiple interacting objects.

G More Experiment Results

G.1 Model Comparison

Wefirst compare our CARE with MP-NODE [10], which is an ODE-based approach for homogeneous dynamical systems. The compared result on particle-based simulation datasets and mesh-based simulation datasets are recorded in Table 4 and Table 5, respectively. From the results, we can validate the superiority of our CARE in tackling the temporal environmental variation and making accurate long-term predictions.

G.2 Results with Different Prediction Lengths

To evaluate the performance of the proposed CARE in different settings, we vary the prediction length and compare the performance of different approaches. In particular, we show the results with the prediction length { 8 , 15 } on two particle-based simulation datasets in Table 6. The results with the prediction length { 15 , 30 } on two mesh-based simulation datasets are shown in Table 7. From the compared results, we can validate the superiority of our CARE in various settings.

G.3 Visualization

Moreover, we show more visualization of our proposed CARE and the best baseline (i.e., TIE) compared with the ground truth. In particular, given that we have shown the time steps in { 1 , 100 , 200 , 300 , 400 , 500 } in Figure 3 and Figure 4, now we select them in { 5 , 150 , 250 , 350 , 450 } for Lennard-Jones Potential and CylinderFlow, respectively. The compared results are shown in Figure 5 and Figure 6. We can observe that our CARE can generate more accurate trajectories compared with the baselines in most cases, which validates the superiority of our CARE again. For example, in the last row in Figure 5, our CARE can accurately recover the velocity distribution around the cylinder while the baseline fails.

H More Related Work

Graph Neural Networks. Graph Neural Networks (GNNs) have garnered significant success for their remarkable capabilities in graph representation learning [26, 62, 72, 33, 13], which is integral to a variety of downstream applications, including node classification [26], link prediction [71], and graph classification [62]. These approaches typically employ the message-passing mechanism, enabling the iterative updating of node representations with the aid of neighboring information. Recently, GNNs have successfully branched out into modeling interactive dynamics [42, 46, 45]. For

CARE

5 step

Ground Truth

TIE

150 step

250 step

350 step

450 step

CARE

5 step

Ground Truth

TIE

150 step

250 step

350 step

450 step

instance, MeshGraphNet [42] employs a message passing neural network to facilitate the modeling of interactions between objectives, thereby outputting the next-time predictions. However, an inherent drawback lies in the inability of discrete GNNs to encapsulate the continuous nature of system

CARE

TIE

Ground Truth

150 step

5 step

250 step

350 step

450 step

dynamics. To relieve this limitation, we present a novel graph-based ODE system named CARE for the modeling of interacting dynamics, which enriches the capabilities of making long-term predictions under potential environmental variation.

Graph-based ODE. Neural ODEs have been integrated into GNNs, resulting in the development of Graph ODEs that are applicable to both static and dynamic graphs. Graph ODEs on static graphs [60, 70] primarily aim to mitigate overfitting by formalizing derivatives using both initial and immediate node representations. Meanwhile, Graph ODEs on dynamic graphs are utilized for tasks such as traffic flow forecasting [9] and social analysis [19], demonstrating effective performance on irregularly sampled partial observation data. For example, STGODE [9] employs tensor computation to conduct continuous message passing, which facilitates accurate long-term predictions by overcoming the network depth limitations. Despite these advancements, existing works fall short of addressing the temporal environmental variation in interacting dynamics. To fill this gap, we propose a novel approach CARE to handle this problem by injecting a context variable in the Graph ODE system.

I Potential Negative Impacts

To the best of our knowledge, we have not found any negative impact of our work.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to model out-of-distribution dynamics of environments with interacting entities using a seq-to-seq style graph neural net architecture. The proposed model attempts to capture the invariant aspects of the dynamical environment via a context embedding that follows a neural ordinary differential equation. The paper reports results on a number of physical systems modeling tasks, as well as some theoretical aspects of the proposed approach.\n\nSTRENGTHS:\nThe paper is very well written. Figure 2 describes the methodology simply and clearly. The paper reports a large set of experiments and shows consistent improvement over a large set of benchmarks, though I think it misses the most essential ones (see my comments below). The paper exercises a systematic scientific writing approach where the key assumptions are pointed out and their most important analytical conclusions, such as model consistency, are analyzed, although with rather straightforward proof techniques.\n\nWEAKNESSES:\nThe proposed method is novel per se. It is also intuitive and well-justified, but it appears to put together a number of existing tools in the most straightforward way one can think of. While I am convinced by the quality of the proposed solution, I am a bit skeptical about its scientific value, i.e. how exactly it enhances our knowledge base.\n\nThe conceptual novelty of the proposed method over some existing graph-based probabilistic ODE approaches such as IGP-ODE [60] and [Ref1] is not clarified. It is also not obvious to the reader why these methods that have suitable specs for handling out-of-distribution data, due to rigorous uncertainty modeling, should not be among the list of models in comparison. For instance IGP-ODE also model interaction dynamics, it reports disentanglement results with the motivation of out-of-distribution detection, and it also takes the dynamics of context variables to its center.\n\n[Ref1] Look et al., Cheap and Deterministic Inference for Deep State-Space Models of Interacting Dynamical Systems.\n\nQUESTIONS:\nAlthough the paper motivates the proposed methodology with the primary application of generalization over out-of-distribution cases, I am not able to point out how the proposed experiments test this aspect and according to which metric. The results tables appear to quantify model performance only in in-distribution use cases, i.e. both training and test data are assumed to follow the same dynamics. Is my understanding correct? If not, how exactly is the out-of-distribution data acquired and what is the actual difference between the train and test splits that makes the test observations out-of-distribution?\n\nSince the answers to the above questions play a critical role in my evaluation, for now I set my grade to borderline. It is likely to swing based on further interactions with the authors and other reviewers.\n\nLIMITATIONS:\nSection 6 of the paper discusses the limitations of the proposed approach in sufficient detail.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new model architecture for modeling multi-agent dynamics. The model consists of three parts (encoder, dynamics, decoder) where the encoder initializes the latent state of each agent and the context, the dynamics is modeled via NeuralODEs and the decoder is a standard MLP. New to the paper is that the model explicitly consider a context variable that evolves over time.\n\nSTRENGTHS:\n### New Architecture\nThe explicit modeling of the context variable and its temporal evolution is new to me. I like the idea and the experiments support its benefits. \n\n### Experiments\nThe experiments in the paper are exhaustive and show the advantage of the new method compared to competitors. I also like the ablation study that shows the merits of the individual contributions.\n\nWEAKNESSES:\nI think the paper slightly oversells what it is doing:\n\n### Temporal distribution shift\nThe paper is called \"modeling interacting dynamics under temporal distribution shift\" but there is very little about temporal distribution shift in the data. I would have expected that for instance the temperature in Sec. 5.1 is different between training and test data. If this is the case, the authors should state it more explicitly in the experiments. If not, I would suggest to tone it down a bit.\n\n### Probabilistic Modeling\n I like the new model architecture and it seems principled to me. However, the paper states in the abstract \"provide a probabilistic view for out-of-distribution dynamics\", whlie the learnt model is completely deterministic. I would also suggest here to not overstate the contribution.\n\nQUESTIONS:\n### Interpretability\nCan you please comment on the interpretability of the context variable? I think it is not and this should also be clearly stated in the manuscript or demonstrated otherwise.\n\n### Naive solutions\nI could also come up with a naive solution in which the context variable is modeled as N+1-th object in the graph that is connected to all other variables but does not have any observations. How would that compare to the aforementioned solution?\nHow does a static context variable compare to the dynamic solution? Static context variables have for instance also been considered in [1] which should be cited.\n\n### References\n[1] Yıldız, Çağatay, Melih Kandemir, and Barbara Rakitsch. \"Learning interacting dynamical systems with latent Gaussian process ODEs.\" Advances in Neural Information Processing Systems 35 (2022): 9188-9200.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main idea of this work is to propose a novel approach called Context-attended Graph ODE (CARE) for modeling interacting dynamical systems under temporal distribution shift. The paper formalizes the problem of temporal distribution shift in interacting dynamics modeling and proposes a probabilistic framework to understand the relationships between trajectories and contexts. Experimental results demonstrate the exceptional performance of the CARE model, outperforming state-of-the-art methods in accurately predicting long-term trajectories, even in the presence of environmental variations.\n\nSTRENGTHS:\n- Novel approach: The CARE model introduced in this work presents a novel methodology by incorporating continuous context variations and system states into a coupled ODE system for modeling interacting dynamical systems under temporal distribution shift.\n- Superior performance: Extensive experimental results demonstrate that the CARE model outperforms state-of-the-art approaches in accurately predicting long-term trajectories, even in the presence of environmental variations.\n- Clarity: The paper effectively summarizes its contributions, including problem formalization, novel methodology, and comprehensive experiments. This clarity allows readers to readily grasp the significance of the research.\n- Extensive experiments: The paper conducts a thorough evaluation of the CARE model by performing experiments on diverse dynamical systems. The results highlight the superiority of the proposed approach compared to state-of-the-art methods. Additionally, the paper includes an ablation study and a parameter sensitivity analysis to further validate the effectiveness of the CARE model.\n\nWEAKNESSES:\nI don't see any weakness in the paper.\n\nQUESTIONS:\n- Where is Lemma 4.2?\n- How to set the threshold mentioned in Line 159?\n- Will the dynamic graph updating with larger $\\delta s$ leads to many indirect connections and how will it affect the results?\n- How is the performance of proposed CARE on dynamical systems with more nodes, let's say, 10, 100 and more?\n- As shown in table 2, CARE still suffers from accumulated error on longer time-series. Do you have any idea on how to decrease the accumulated error?\n\nLIMITATIONS:\nThe limitation is discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}