CARE: Modeling Interacting Dynamics Under Temporal Environmental Variation
Authors: Xiao Luo
Haixin Wang
Zijie Huang
Huiyu Jiang
Abhijeet Sadashiv Gangan
Song Jiang
Yizhou Sun
Venue: NeurIPS 2023 poster (2023)
Keywords: Dynamical System
Distribution Shift
Neural ODE
Graph Neural Network

Abstract: Modeling interacting dynamical systems, such as fluid dynamics and intermolecular interactions, is a fundamental research problem for understanding and simulating complex real-world systems. Many of these systems can be naturally represented by dynamic graphs, and graph neural network-based approaches have been proposed and shown promising performance. However, most of these approaches assume the underlying dynamics does not change over time, which is unfortunately untrue. For example, a molecular dynamics can be affected by the environment temperature over the time. In this paper, we take an attempt to provide a probabilistic view for time-varying dynamics and propose a model Context-attended Graph ODE (CARE) for modeling time-varying interacting dynamical systems. In our CARE, we explicitly use a context variable to model time-varying environment and construct an encoder to initialize the context variable from historical trajectories. Furthermore, we employ a neural ODE model to depict the dynamic evolution of the context variable inferred from system states. This context variable is incorporated into a coupled ODE to simultaneously drive the evolution of systems. Comprehensive experiments on four datasets demonstrate the effectiveness of our proposed CARE compared with several state-of-the-art approaches.

CARE: Modeling Interacting Dynamies Under Temporal Environmental Variation
Xiao Luo', Haixin Wang", Zijie Huang', Huiyu Jiang?, Abhiject Sadashiv Gangan', Song Jiang', Yizhou Sun! University of California, Los Angeles, PPeking University, University of California, Santa Barbara yzsun)oss ucla. ucla.edu, wang hw@stu.tt...... aku edu
Abstract
Modeling interacting dynamical systems, such as fluid dynamics and intermolecular interactions, is fundamental research problem for understanding and simulating complex real-world systems. Many of these systems can naturally represented by dynamic graphs, and graph neural network-based approaches have been proposed and shown promising performance. However, most of these approaches assume the underlying dynamics does not change over time, which is unfortunately untrue. For example, molecular dynamics can be affected by the environment temperature over the time. In this paper, we take attempt provide probabilistic view for time-varying dynamics and propose model Context-attended Graph ODE (CARE) for modeling time-varying interacting dynamical systems. In our CARE, ex- plicitly use context variable to model time-vary mareiig environment and construct an encoder to initialize the context variable from historical trajectories. Furthermore, we employ a neural ODE model to depict the dynamic evolution of the context variable inferred from system states. This context variable is incorporated into a coupled ODE to simultaneously drive the evolution of systems. Comprehensive experiments on four datasets demonstrate the effectiveness of our proposed CARE compared with several state-of-the-an approaches.
Introduction
Modeling interacting dynamical systems is fundamental machine learning problem 66 65 23 59
37 with a wide range of applications, including social network analysis [12] 36] and particle- based physical simulations [46] 43 35). Geometric graphs (27 are utilized to formalize these interactions between objects. For example, in particle-based physical systems, edges are constructed based on the geographical distance between particles, which represents the transfer of energy.
In the literature, numerous data-driven approaches have been proposed for understanding interacting
dynamical systems [2] 25 49). Among them, graph neural networks [26] 62 72
33 [3] (GNNs)
widely utilized to predict trajectories at the timestamp due their strong capacity to capture interactions in graph-structured data. In particular, each object is considered as a graph node, and edges represent interactions between neighboring objects. Given the observations and their corresponding graph structure, these methods forecast states in the next timestamp using the message passing mechanism. This process involves aggregating information from the neighbors of each node to update its representation in an iterative fashion, effectively capturing the dynamics of the system.
Although impressive progress has been achieved dependency in interacting dynamical systems is still a practical but underexplored problem. Existing next-step predictors [42 46 45) send the predictions back to generate rollout trajectories, which could suffer from serious error accumulation for long-term predictions. More importantly, system
GNN-based approaches, capturing long-term
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

environments and relational structures could be changeable [69 (e.g.. unsteady flow [T [6]). which implies the potential temporal distribution variation during the evolution. In particular. in physical systems, there are various potential factors which can influence the trajectories extensively. For example, high temperatures [69] or pressure [68] could speed up molecular movement. Their continuous variation would make understanding interacting dynamic systems more challenging. First, temporal environmental variation would indicate different data distributions over the time [52), which requires the model equipped with superior generalization capability. In contrast, existing methods typically focus on in-distribution trajectories 77 I 58 3 21 66). which would perform worse when it comes to out-of-distribution data. Second, recent out-of-distriee distribbbttoon generalization methods [50] 39 57 64 usually focus on vision and text with discrete shift across different domains. However, scenarios would face the continuous distribution variation, which is difficult to capture in interacting dynamical systems.
In this paper, we propose novel method named Context-attended Graph ODE (CARE) to capture interacting system dynamies. The core of our CARE approach to characterize the temporal environmental variation by introducing the context variable. In particular, we propose a probability model to depict the interaction between the context variable and trajectories. Based on the probabilistic decomposition, we divide each training sequence into two parts for initializing embeddings and predictions. Here, we first construct a temporal graph and then leverage an attention attention-based encoder to generate node representations and the context representation from spatial and temporal signals simultancously. More importantly, we introduce coupled ODEs to model the dynamic evolution of node representations and the context variable. On the one hand, adopt a graph-based ODE system enhanced with context information to drive the evolution. On the other hand, the context information can also be updated using summarized system states and the current context. We also provide a theoretical analysis that indicates, at least locally, the future system trajectory and context information are predictable based on their historical values. Finally, we introduce efficient dynamical graph updating and robust learning strategies to enhance the generalization capability and efficiency of the framework, respectively. Extensive experiments on various dynamical systems demonstrate the superiority of our proposed CARE compared with state-of of-the-art approaches.
To summarize, in this paper make the following contributions: (1) Problem Formalization. We formalize the problem of temporal environmental variation in interacting dynamics modeling. (2) Novel Methodology. We analyze the problem under probabilistic framework, and propose novel approach CARE, which incorporates the continuous context variations and system states into a coupled ODE system. (3) Extensive Experiments. Extensive experiments conducted on four datasets validate the superiority of our CARE. The performance gain of our proposed CARE over the best baseline is up to 36.35%.
2 Related Work
Interacting Dynamics Modeling, Deep learning approaches have been extensively used in recent including molecular dynamics and computational fluid dynamics. Early efforts focus on incorporating convolutional neural networks to learn from interacting regular grids 41). To address more generalized scenarios, graph neural network (GNN) methods have been developed [42 46 45), leveraging message passing mechanisms to extract complex spatial signals. However, these methods often fail to account for environmental fluctuations, which hinders their ability to make reliable long-term predictions. In contrast, our CARE adopts a context-attender ODE architccture to explicitly represent both the observations and the underlying environment, enabling the generation of accurate future trajectories.
years to model interacting systems across various fields 53 29 20
Neural Ordinary Differential Equations (ODEs). Drawing inspiration from the approximation of ResNet [5] neural ODEs equip neural networks with a continuous-depth architecture by parameteriz- ing the derivatives of hidden states. Several attempts have been made to increase the expressiveness of neural ODEs [65), including adding regularization [8] and designing high-order ODEs [61. Neural ODEs have also been incorporated into GNNs, producing continuous message passing layers to avoid oversmoothing (60) and increase the interpretability of predictions (70). In this study, we employ graph ODE architecture to capture the continuous nature of interacting system dynamics, relieving potential error accumulation caused by discrete prediction models.

---

ODE solver
time
Context represcutation
Node pedeeepeppeeenntee
Go
G1
G2
G
Decoder
Decoder
Decoclor
Decoder
Historical trajactories
Figure 1: An overview of the proposed CARE. To begin, we construct a temporal graph and utilize an encoder to initialize both the context variable and node representations from historical trajectories. Then, a coupled ODE model simulates the evolution of both nodes and context. Finally, CARE feeds node representations into decoders, which output the predicted trajectories at any timestamp.
-iitti--iittitttionn (OOD) Generalization. OOD generation aims to make models more effective This problem has drawn consid- learn erable attention in several areas, including text and vision 488 One effective solution
when the training and testing distributions diverge (50 39
57
633
which has been achieved under the guidance invariant learning theory [6 32). Additionally, uncertainty modeling [34], causal learning [54] [4, and model selection (38 56 are employed to improve the performance. Interacting systems inherently exhibit dynamic distribution variation caused by environmental changes, an aspect that remains underexplored in the literature. To address this, paper proposes novel approach named CARE to model context information from the perspective of a probabilistic model.
domain-invarte invariant representations in the hidden space B1 55
30),
3 Background
3.1 Problem Definition
In multi-agent dynamical system, the state at time t is represented by G = (V, E*), where each node in V corresponds an object, E' denotes the current edge set, and X' signifies the node attribute matrix. Specifically, the state vector for each i E V is given by x = [p', q a,l, with pf R&3 and qf R33 representing the position and velocity, respectively, and a; representing static attributes. We are given the sequence (G", G ..... G') and aim to learn a model that produces the target dynamic states Y* > t) (e-g., velocity), which are part of X the corresponding time. The temporal environmental variation would result in data distribution changes during the evolution of interacting systems. we utilize 0hdtt to indicate the dynamical environment factor timestamp t, we have data from variable distribution, i.e., (Gtet Y*) P(G C*(""::). Thus, must take these changes into account for accurate trajectory predictions.
3.2 GNNs for Modeling Dynamical Systems
Graph neural networks (GNNs) extensively employed in dynamical system modeling to investigate the interactive relationships between objects [42 46 45). These methods typically use the current states to predict the states of nodes at the next timestamp. Specifically, omitting the time notation, GNNs first initialize node representations and edge representations using encoders:
f
(x xz))
(I)
where f*(-) and (.) are two encoders for node and edge representations, respectively. Then, they (!) and e
utilize two propagation modules to update these representations at the I-th layer, i.e., n, following the message passing mechanism:
(L+1) = eij
60 (v.0)
00
e
11)
v
(++)
= =0 1. Σ (++1)
Ejj
(2)
JEN,
3

---

where Mr collects the neighbours of node i. (-(-) and (-(-) are two functions for representation updating, Finally, they generate the target velocity vectors the next timestamp using a decoder.
Methodology
In this paper, we propose novel method called CARE for modeling interacting dynamics under temporal environmental variation. We start by formalizing probabilistic model to understand the relationships between trajectories and contexts. Based on this foundation, we construct spatio- temporal encoder to initialize the representations of nodes and contexts. Then, to simultaneously model their evolution, a coupled graph ODE is introduced where node representations are evolved with the guidance of contexts and the states of the context variable are inferred from current trajectories. Additionally, we introduce a regularization term and dynamic graph updating strategies to enhance our framework. An illustration of our CARE can be found in Figure
4.1 Probabilistic Model for System Dynamies under Temporal Distribution Drift
In this work, to tackle the challenge brought by temporal distribution drift, we inject a context variable et in our dynamic system modeling, which indicates the environment state timestamp I. For example, the context variable could indicate flow speed, density and viscosity in fluid dynamics.
Here, make two basic assumptions in our probabilistic model.
Assumption 4.1. (Indepentence-1) The context vuriable is independent of the sequences before the last observed timestamp, ie., P(cllct Gbt))= P(ctlcc " G'-kit),, where 1- k is the last observed timestamp.
Assumption 4.2. (Independence-I]) Given the current states and contexts, the future trajecto- ries are independent of the previous trajectories and contexts, iee, P(yt-kittl clt-k)-
- c.-kk where is the length of the prediction.
Then, we can have the following lemma: Lemma 4.1. With Assumptions
4.1 and
we have;
PYY' Gtd-1)= = / p d'-1,G'-)).
(3)
p ci-k G--ke--)))) P ( dc
The proof of Lemma 4.2 can be found in Appendix. From Lemma we decompose the probability P PYY (0:t-1) into three terms. Specifically, the last term necessitates encoding context information based on the historical trajectory G The second term aims to update the context vector according to the recent trajectory G k The first term suggests using the context variable in conjunction with the current states make predictions for the timestamp. Besides making single next-time prediction, our model can also predict trajectories (Y'-* y
Y') by modifying Eqn. [4 Consequently, we divide each training sequence into two parts, namely [0, - A]] and (1 k, () as in [8). The first part is used to encode contexts and nodes in the system, while the second part serves for updating and prediction purposes.
4.2 Context Acquirement from Spatio-temporal Signals
In this part, our goal is to acquire knowledge from the historical trajectory, i.e., (G",... ..'k encode contexts and nodes for initialization. To be specific, first construct temporal graph to capture spatial and temporal signals simultaneously. Subsequently, we employ the attention mechanism update temporal node representations, which will be used to initialize the context representation and node representations for
To begin, construct a temporal graph containing two types of edges, i.e., spatial and temporal edges. Spatial edges are built when the distance between two nodes at the same timestamp less than threshold while temporal edges are between every two consecutive observations for each node. Specifically, in the constructed temporal graph HHH, there are N k + 1) nodes {ig { [0

---

in total. The adjacent matrix A contains both spatial and temporal edges as follows:
8 == s', e(p((_d((,,)) <
A
j=
otherwise.
where d' (i. denotes the distance between particles and j at timestamp and T is the predefined threshold. Then, we utilize an attention- based GNN to extract spatio-temporal relationships into node representations. Here, we first compute the interaction scores between each node in G with its neighboring nodes, and then aggregate their embeddings at the previous layer. Let d represent the hidden dimension, the interaction score between and jd at layer /
w (i*,j*) (.'''')) = A(i (W query (.)" (Wkeyh) (())
where W queryy Raxd and W Wey Raxd two matrices map temporal node representations into different spaces. * computes the cosine similarity between two vectors. With the interaction scores, we can compute temporal node representations at the next layer:
h (.(++)
h a,0)) to
Σ ('EEN(P)
W value hz (0000)
(6)
where W Wellue is a weight matrix, ((t))) collects all the neighbours of 1s and a--) an activation function. After stacking L layers, we add temporal encoding and then summarize all these temporal node representations to initialize node representations for the upcoming ODE module:
(L) qf = h
TE(s),
u
1-k
W s.
0
where [2i)) sin 10000
[E(s) cos
10000277 and W sum denotes projection
matrix. The initial context variable e² is driven by summarizing all node representations:
B tonh((
W Weontent) utt
3
(8)
PEV
iEV
where W.on context is a learnable matrix and 8 calculates the attention score for each node.
4.3 Context-attended Graph ODE
In this module, to model continuous evolution, we incorporate an ODE system into our approach. The precondition requires assuming that both the context variable and node representations are continuous to fit neural ODE models, which inherently holds for common dynamical systems in practice. We then introduce coupled ODEs model the dynamic evolution of node representations and the context variable. Specifically, the context variable can be inferred during the evolution of node representations, which turn drives the evolution of the system. We first introduce the assumption: Assumption 4.3. (Continuous) We assume that both context variable c" and node representations vū are continuously differentiable with respect to
Then, to utilize the context variable and the current state for making future predictions, we introduce a graph ODE model. Let A denote the adjacency matrix at timestamp s with self-loop, we have:
dv ds
((v ... c²)) = = Σ A
D W - W2).
(ENT() V D DJ
where (1) denotes the neighbours of node i at timestamp and Dj represents the degree of node i according to A The first term in Eqn. aggregates information from its instant neighbors and the second term captures information from the current context information.
The next question is how to model the evolution of c². Notice that we have:
P (c - Gr-kit) = / P(clle'-sr, Airi-ssGGAAAA Gt-
(10)
- Gt -i- ktdr. .c.
5

---

where At denotes small time interval. With Assumption 4.3 we can simplify P(ct-k+bMl c' (o--kt-k+Ay into P(c' P(c'-R+Arle" - V dv*=k) where V denotes the
node embedding matrix timestamp k and dVt-k is its differentiation. On this basis, introduce another ODE to update the context variable as:
de ds (AGG((( AGG(|
du, ds )ev), ,,")),
(11)
where d5 is MI.P with the concatenated input and AGG(.) is operator summarize node representations such as averaging and sum. Compared to previous methods, the key to our CARE is take into account the mutual impact between the environment and the trajectories, and model their evolution simultaneously by coupling Eqn. 9 and Eqn. [I] We also provide theoretical analysis of the uniqueness the solution our system. To simplity the analysis, set AGG(-) to summation and rewrite Eqn. 11 with learnable matrices W3. W4 and Ws as:
de ds
N
+ dup Wa) + teWs) ds
Σ
(o
(12)
Then, introduce the following assumption:
Assumption 4.4. All time-dependent coefficients Eqn. ie Alg. D4 are continuous with respect to and bounded by constant C O. All parameters in the weight matrix are also bounded by constant W > O.
With Assumption 4.4 we can deduce the following lemma:
Lemma 4.2. Given the initial state (lo, , :..... , c.a), ) we claim that there exists & > 0, s.f. the ODE system 9 and 12 has unique solution in the interval [to a, +
The proof of Lemma 4.2 can be found in Appendix. Our theoretical analysis indicates that at least locally, the future system trajectory and context information are predictable based their historical values (5 511, which is also an important property for dynamical system modeling [5] 28
4.4 Decoder and Optimization
Decoder. We introduce MLP (-) to predict both the position and velocity vectors using corresponding node representations, i.e., [p q (vf))
Dynamic Graph Updating. We can estimate the instant distance between nodes using the encoder and then construct the graphs, which could suffer from a large computational burden. To improve the efficiency of graph construction during ODE propagation, every As, and but also introduce graph updating strategy that calculates the distance between first-order and second-order neighbors in the last graph. By doing so, we can delete edges between first-order neighbors and add edges between second-ou -oderr neighbors, reducing quadratic complexity to linear complexity in sparse graphs. We will also validate this empirically.
only update the graph structure
Learning Objective. Given the ground truth, we first minimize the mean squared error (MSE) of the predicted trajectory. Moreover, we require both node and context representations be robust to noise attacks to improve the robustness of the ODE system. The overall objective is written as:
Σ A k
||
Y + + V*I + || c²)l),
(13)
where X denotes the predictions from the encoder and 7j is a parameter set to 0.1 balance two losses. 4 and ō denote the perturbed representations under noise attack the input.
5 Experiments
We evaluate our proposed CARE on both particle-based and mesh-based physical systems. To ensure the accuracy of our results, we use rigorous data split strategy, where first 80% of the samples reserved for training purposes and the remaining 10% are aside for testing and validating,

---

Table The RMSE (x10-2) results the compared methods with the prediction lengths 1, 5, 10 and 20. Uus Byy and represent the velocity in the direction of each coordinate axis.
Prediction Length
+1
Variahle
Ux
U/2
1/2
Myy
1/2
02
1/2
Fy
V/
Lennord-Jones Potential
LSTM
3.95
3.92
3.68
9.12
9.21
9.15
10.84
10.87
10.76
14.82
14.94
14.67
GNS
3.28
3.75
3.39
7.97
8.05
7.68
10.09
10.15
10.13
13.65
13.62
13.59
STGCN
2.91
3.08
2.95
5.06
5.17
5.11
6.89
6.90
6.93
9.31
9.32
9.44
MeshGraphNet
2.89
3.13
2.94
5.29
5.53
5.28
7.03
7.09
7.11
9.12
9.21
9.24
CG-ODE
1.79
2.05
1.71
3.47
3.92
3.38
5.46
5,99
5,36
9.03
9.26
TIE
1.62
1.98
1.47
3.25
3.15
Ours
0.76
0.89
1.01
2.94
3.16
2.85
5.01
4.69
4.71
5.75
5.91
5.82
3-body Stillinger- Weber Potential
LSTM
17.11
17.14
17.18
23.64
23.69
23.60
25.46
25.42
25.48
28.44
28.45
28.44
GNS
15.39
15.27
15.33
22.14
22.19
22.17
25.29
25.36
25.31
27.18
27.15
27.14
STGCN
12.33
12.31
12.35
17.94
17.96
17.91
20.08
20.14
20.13
23.51
23.52
MeshGraphNeet
12.16
12.10
12.13
18.33
18.38
18.34
20.65
20.62
20.71
23.62
23.54
23.61
CG-ODE
9.78
9.74
9.75
12.11
12.05
12.14
15.55
15.58
15.50
16.17
16.24
16.22
TIE
10.18
10.26
10.19
14.75
14.70
14.73
18.42
18.45
18.41
20.92
21.04
21.36
Ours
4.21
4.29
4.18
9.74
9.79
9.71
13.65
13.71
13.57
15.30
15.39
15.35
CARE
Ground Truth
Figure 2: Visualization of Lennard-Jone Potential with multiple timestamps. We render the 3D positions of each particle according to the historical positions and predicted velocities.
separately. During training, we split each trajectory sample into two parts, i.e., a conditional part and prediction part. We initialize node representations and the context representation based on the first part and utilize the second part to supervise the model. The size of the two parts is represented as conditional length and prediction length, respectively. Our approach is compared with various baselines for interacting systems modeling, i.e., LSTM [1], STGCN 67], GNS 45
MeshGraphNet 42 TIE 46 and CG-ODE 19
5.1 Performance on Particle-based Physical Simulations
Datasets. We evaluate our proposed CARE on two particle-base simulation datasets with temporal environmental variations, i.e., Lennard-Jones Potential [47 and 3-body Stillinger- Weber Potential 4 Lennard-Jones Potential is popular in modeling electronically neutral atoms or molecules. 3-body Stillinger-
Wellinger-Weber Weber Potential provides more complex relationships in atom systems The temperature in two particle-based systems is continuously changed along with the time to model the environmental variations. The objective to predict the future velocity values all directions, i.c., Ugs By and Ug- More details can be found in Appendix.
Performance Comparison. We evaluate the performance in terms of RMSE with different prediction lengths. Table show the compared results on these two datasets. We can observe that our proposed CARE outperforms all the baselines on two datasets. In particular, compared with TIE, CARE accomplishes an eтror reduction of 24.03% and 36.35% on two datasets, respectively. The remarkable performance can be attributed to two factors: (1) Introduction of the context variable. Our CARE infers the context states during the evolution of the system, which can help the model understand environmental variations. (2) Introduction of robust learning. We add noise attack to both system and context states, which improves the model generalization to potential distribution changes.

---

Table 2: The RMSE results of the compared methods different prediction lengths 1, 10, 20 and 50. Uz. My and represent the velocity in different directions and the pressure field, respectively.
Prediction Length
+10
+20
+50
Variable
1/2
By
V2
Uy
P
V/
V/y
0/
Vy
CylinderFlow
LSTM
3.35
29.4
12.5
7.06
44.8
17.8
9.47
49.5
19.9
14.3
73.6
42.3
GNS
3.12
28.8
11.9
7.18
44.3
17.3
9.01
49.6
19.2
13.5
73.2
41.6
STGCN
2.68
26.7
11.0
5.47
42.1
16.9
6.72
45.6
18.4
9.15
68.7
40.0
MeshGraphNet
1.75
10.6
4.09
39.7
15.7
5.38
44.5
17.2
7.92
64.3
37.7
CG-ODE
1.05
20.4
8.51
3.44
36.8
13.6
4.15
38.5
17.1
5.14
61.2
32.3
TIE
1.22
20.8
8.94
3.75
35.2
13.0
4.62
40.6
16.0
5.87
59.5
32.1
Ours
0.87
19.1
7.21
3.02
32.9
11.8
3.95
37.8
13.9
4.97
55.8
29.4
Airfoil
LSTM
7.49
7.73
1.92
8.86
9.02
3.78
10.8
11.0
4.71
14.9
15.7
4.96
GNS
6.95
7.14
1.69
8.20
8.34
3.34
10.2
10.5
3.98
14.2
14.1
4.11
STGCN
6.24
5.35
6.57
6.51
2.33
7.88
8.01
3.16
11.8
3.17
MeshGraphNet
4.72
4.68
0.50
5.89
5.74
1.23
6.32
6.48
1.85
9.03
9.12
2.08
CG-ODE
4.26
4.32
0.35
4.78
4.70
0.46
5.81
5.66
1.04
7.39
7.85
1.69
TIE
4.17
4.39
0.33
4.99
4.86
0.51
5.75
5.62
0.95
7.25
7.63
1.44
Ours
3.86
3.75
0.34
4.16
0.45
6.74
6.82
0.81
TIE
CARE
Ground Truth
Figure Visualization the CylinderFlow Dataset with multiple timestamps. We render the velocity in the r-axis in the fluid field of our CARE and the ground truth.
Visualization. Figure 2 visualizes the prediction of positions comparison to the ground truth on Lennard-Jones Potential. Here, we sample six timestamps in every trajectory validate the performance of both short-term and long-term predictions. From the qualitative results, we can observe that in the first three frames, the particle motion is not strenuous due to low temperature in the system. Surprisingly, our proposed CARE can always make faithful physical simulations close to the ground truth even though the system environment is highly variable.
Performance on Mesh-based Physical Simulations
Datasets. We employ two popular mesh- based simulation datasets, i.e., CylinderFlow: and Airfoil. CylinderFlow consists of simulation data from modeling an incompressible flow governed by the Navier-Stokes equations. Notably, the initial flow velocity of the incoming water flow to the cylinder varies cyclically over time, meaning the Reynolds number of the flow field also changes periodically Airfoil is generated in similar manner through simulations of compressible flow, wherein the inlet velocity over the wing varies cyclically over time. We aim to forecast the velocity values v2, and by. as well the pressure p. More details can be found in Appendix.
Performance Comparison. The performance with respect to different variables is recorded in Table From the results, we can observe that the average performance of the proposed CARE is over the best baseline by 12.99% and 22.78% on two datasets, respectively. Note that unsteady flow [1T [6] is crucial problem in recent fluid dynamics, proposed CARE benefit abundant complex mesh-based simulations under environmental variations.
Visualization. Moreover, we show the qualitative results of the best baseline and our CARE in comparison the ground truth. From the results, we can observe that our CARE can capture more accurate signals in unsteady fluid dynamics. In particular, in the last two frames with complicated

---

Table 3: Ablation study on four datasets.
Datasets
Lennard-Jone
3-body Stillinger- Weber
CylinderFlow
Airfoil
Variable
Vir
Fvy
2/
U/:
f/y
1/2
Fy
0/
Fy
CARE VI
6.98
7.12
7.06
18.2
18.3
18.3
6.13
60.4
32.2
7.13
7.21
1.43
CARE V2
6.03
6.35
6.30
16.8
16.5
16.6
5.21
57.2
29.8
6.94
6.99
1.15
Ours
5.75
5.91
5.82
15.3
15.4
15.4
4.97
55.8
29.4
6.74
6.82
0.81
(a)
(b)
(c)
(d)
Fooo Mean Squarre Brror
Root Square Ecrorr
16
25
1JP+22m
SW01+200
CAREZ
Square
000
00-00611220
Time Coot
10
25
13
Contition Length
Condition Length
Interval
Figure 4: (a). (b) The performance with respect different condition and prediction lengths on
CylinderFlow and Airfoil. (c) The sensitivity of interval on Lennard- Potential (LJP) and 3-body Stillinger- Weber Potential (SWP) datasets. (d) The comparison of running time for our dynamic graph updating and full pairwise calculation on two particle-based datasets.
structures, our CARE still can generate superior simulations in both scenarios under potential environmental variation while the baseline fails, which shows the superiority of our proposed CARE.
5.3 Further Analysis
Ablation Study. To analyze the effectiveness of different components in our CARE, we introduce two different variants: (1) CARE VI, which removes the context variable in Eqn. 9 (2) CARE V2, which removes the robust learning term in Eqn. [3] The compared performance is recorded in 3 and we have two observations. First, our full model outperforms CARE V1, which indicates the incorporation of the context variable would benefit interacting system modeling under temporal environmental variation. Second, without the robust learning term, the performance would get worse, implying that improving the robustness can also benefit tackling the distribution changes.
Parameter Sensitivity. We begin by analyzing the performance with respect to different condi- tion lengths and prediction lengths. Here the condition length and prediction length vary from (10,
(20,50), respectively. From the results in Figure 4 (a) and (b), we can observe that our proposed CARE can always achieve superior performance compared with CG-ODE. Moreover, we can observe that a longer condition length would benefit the performance in most cases due to more provided information. It can also be seen that a smaller interval for graph updating would improve the performance before saturation from Figure 4 (c).
Efficiency. To show the efficiency of our proposed dynamic graph updating, we propose model variant named CARE E, which calculates all pairwise distances to update graph structure during the evolution. The computational cost is recorded in Figure and we can observe that our strategy can reduce a large number of computational costs, which validates the complexity analysis before.
6 Conclusion
This paper studics the problem of modeling interacting dynamics under temporal environmental variation and we propose probabilistic framework to depict the dynamical system. Then, novel approach named CARE is proposed. CARE first constructs an encoder to initialize the context variable indicating the environment and then utilizes a coupled ODE system, which combines both the context variable and node representation to drive the evolution of the system. Finally, we introduce both efficient dynamical graph updating and robust learning strategies to enhance our framework. Extensive experiments on four datasets validate the superiority of our CARE.
Broader Impacts and Limitations. This work presents an effective learning-based model for interacting dynamical systems under temporal environmental variation, which can benefit complex physical simulations such as unsteady flow. Moreover, our study provides perspective

---

modeling environmental variations for fluid dynamics and intermolecular interactions. One potential limitation is that knowledge. In future work, we would extend our CARE to more complicated applications such as rigid dynamics.
our CARE cannot directly fit more physical scenarios requiring abundant external
Acknowledgement
This work was partially supported by NSF 2211557, NSF 1937599, NSF 2119643, NSF 2303037, NSF 2312501, NASA, SRC Jump 2.0, Okawa Foundation Grant, Amazon Research Awards, Cisco research grant, Picsart Gifts, and Snapchat Gifts.
References
[1] Kelsey R Allen, Yulia Rubanova, Tatiana Lopez-Guevara, William Whitney, Alvaro Sanchez- Gonzalez, Peter Battaglia, and Tobias Pfaff. Learning rigid dynamics with face interaction graph networks. arXiv preprint arXiv:2212.0355 2022.
[2] Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In NeurlPS, 2016.
[3] Suresh Bishnoi, Ravinder Bhattoo, Jayadeva Jayadeva, Sayan Ranu, and NM Anoop Krishnan. Enhancing the inductive biases of graph neural ode for modeling physical systems. In ICLR, 2023.
[4] W Michael Brown and Masako Yamada. Implementing molecular dynamics hybrid high performance computers- 184(12):785-2793, 2013.
ceattters-trreebbooe three-body potentials. Computer Physics Communications,
[5] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In NeurIPS, 2018
[6] Yongqiang Chen, Yonggang Zhang, Yatao Bian, Han Yang, MA Kaili, Binghui Xie, Tongliang Liu, Bo Han, and James Cheng. Learning causally invariant representations for out-of- distribution generalization graphs. In NeurlPS, pages 22131-22148, 2022.
[7] Yitong Deng, Hong-Xing Xing Yu, Jiajun Wu, and Bo Zhu. Learning vortex dynamics for fluid inference and prediction. arXiv preprint arXiv:2301. 110222001.1149 2023.
[8] Emilien Dupont, Arnaud Doucet, and Yee Whye Tch. Augmented neural odes. In NeurIPS, 2019.
[9] Zheng Fang, Qingqing Long, Guojie Song, and Kunqing Xie. Spatial-temporal graph ode networks for traffic flow forecasting. In KDD, pages 364-373, 2021.
[10] Jayesh K Gupta, Sai Vemprala, and Ashish Kapoor. Learning modular simulations for homoge- neous systems. In NeurlPS, 2022.
[11] Rohit Gupta and Phillip J Ansell. Unsteady flow physics of airfoil dynamic stall. AIAA journal, 17(1):165-175, 2019.
[12] Nesrine Hafiene, Wafa Karoui, and Lotfi Ben Romdhane. Influential nodes detection in dynamic social networks: survey. Expert Systems with Applications, 159:1 13642, 2020.
[13] Mingguo He, Zhewei Wei, and Ji-Rong Wen. Convolutional neural networks on graphs with chebyshev approximation, revisited. In NewrIPS, 2022.
[14] Yue Hc, Zimu Wang, Peng Cui, Hao Zou, Yafeng Zhang, Qiang Cui, and Yong Jiang. Causpref: Causal preference learning for out-of-distribution recommendation. In WWW, pages 410-421, 2022.
[15] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 173551735 1997.
10

---

[16] C Huang, L Zhao, JP Niu, JJ Di, JJ Yuan, QL Zhao, FQ Zhang, ZH Zhang, JM Lei, and GP He. Coupled particle and mesh method in an euler frame for unsteady flows around the pitching airfoil. Engineering Analysis with Boundary Elements, 138: 159-176, 2022.
[17] Chao Huang, Xian Wu, Xuchao Zhang, Chuxu Zhang, Jiashu Zhao, Dawei Yin, and Nitesh Chawla. Online purchase prediction via multi-scale modeling of behavior dynamics. In KDD, pages 2613-2622, 2019.
[18] Zijie Huang, Yizhou Sun, and Wei Wang. Learning continuous system dynamics from irregularly-saman sampled partial observations. In NewrIPS, pages 16177-16187, 2020.
[19] Zijie Huang, Yizhou Sun, and Wei Wang. Coupled graph ode for learning interacting system dynamics. In KDD, 2021.
[20] Zijie Huang, Yizhou Sun, and Wei Wang. Generalizing graph ode for learning complex system dynamics across environments. In KDD, pages 798-809, 2023.
[21] Steeven Janny, Aurélien Beneteau, Nicolas Thome, Madiha Nadri, Julie Digne, and Christian Wolf. Eagle: Large-scale learning of turbulent fluid dynamics with mesh transformers. arXiv preprint 2.0.::33022.0003333 2023.
[22] Hrvoje Jasak. Openfoam: open source cfd in research and industry. International Journal of Naval Architecture and Ocean Engineering. 1(2):89-94) 94, 2009.
[23] Ruoxi Jiang and Rebecca Willett. Embed and emulate: Learning to estimate parameters of dynamical systems with uncertainty quantification. NeurIPS, 2022.
[24] Patrick Kidger, Ricky T. Q. Chen, and Terry J. Lyons. "hey, that's not an ode": Faster ode adjoints via seminorms. ICML, 2021.
[25] Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In ICML, pages
2018.
[26] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
[27] Miltiadis Kofinas, Naveen Nagaraja, and Efstratios Gavves. Roto-translated local coordinate frames for interacting dynamical systems. In NeurIPS, 2021.
[28] Lingkai Kong, Jimeng Sun, and Chao Zhang. Sde-net: Equipping deep neural networks with uncertainty estimates. In /CML, 2020.
[29] Shiyong Lan, Yitong Ma, Weikang Huang, Wenwu Wang, Hongyu Yang, and Pyang Li. Dstagnn: Dynamic spatial-temporal aware graph neural network for traffic flow forecasting. In ICML, pages 11906 119906-11911 11917, 2022.
[30] Haoliang Li. Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In CVPR, pages 5400-5409, 2018.
[31] Haoliang Li, YuFei Wang, Renjie Wan, Shiqi Wang. Tie-Qiang Li, and Alex Kot. Domain generalization for medical imaging classification with linear-dependency regularization. In NeurIPS, pages 3118 18-3129, 2020.
[32] Haoyang Li. Ziwei Zhang, Xin Wang, and Wenwu Zhu. Learning invariant graph representations for -iitt---iittiittionn generalization. In NewrIPS, 2022.
[33] Xiang Li, Renyu Zhu, Yao Cheng, Caihua Shan, Siqiang Luo, Dongsheng Li, and Weining Qian. Finding global homophily graph neural networks when meeting heterophily. In ICMI., pages 13242-13256, 2022.
[34] Xiaotong Li, Yongxing Dai, Yixiao Ge, Jun Liu, Ying Shan, and Ling- Yu Duan. Uncertainty modeling for out-of-distribution generalization. arXiv preprint 2XXx:2202.03955: 2022.
11

---

[35] Xuan Li, Yadi Cao, Minchen Li, Yin Yang, Craig Schroeder, and Chenfanfu Jiang. Plasticitynet: Learning to simulate metal, sand, and snow for optimization time integration. In NewrlPS, pages 27783-27796, 2022.
[36] Siyuan Liao, Shangsong Liang, Zaiqiao Meng, and Qiang Zhang. Learning dynamic embeddings for temporal knowledge graphs. In WSDM, pages 535-543, 2021.
[37] Andreas Look, Melih Kandemir, Barbara Rakitsch, and Jan Peters. Cheap and determinis- tic inference for deep state-space models of interacting dynamical systems. arXiv preprint arXiv:2305.01773 2023.
[38] Wang Lu, Jindong Wang, Yidong Wang, Kan Ren, Yiqiang Chen, and Xing Xie. Towards optimization and model selection for domain generalization: A mixup-guided solution. arXiv preprint arXiv:2209.000 00652, 2022.
[39] Lucas Mansilla, Rodrigo Echeveste, Diego H Milone, and Enzo Ferrante. Domain generalization via gradient surgery. In ICCV, pages 6630-6638, 2021.
[40] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito,, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[41] Jiang-Zhou Peng, Siheng Chen, Nadine Aubry, Zhihua Chen, and Wei-Tao Wu. Unsteady reduced-order model of flow over cylinders based on convolutional and deconvolutional neural network structure. Physics of Fluids, 32(12): 2020.
[42] Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W Battaglia. Learning mesh-based simulation with graph networks. ICLR, 2021.
[43] Lukas Prantl, Benjamin Ummenhofer, Vladlen Koltun, and Nils Thuerey. Guaranteed conserva- tion of momentum for learning particle-based fluid dynamics. arXiv preprint arXiv:2210.06036. 2022.
[44] Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of-distribution generalization. In ICML, pages 18347-18377, 2022.
[45] Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In ICML, pages 8459-8468, 2020.
[46] Yidi Shao, Chen Change Loy, and Bo Dai. Transformer with implicit edges for particle-based physics simulation. In ECCV, pages 549-564, 2022.
[47] BJTJ Smit. Phase diagrams of lennard-jones fluids. The Journal of Chemical Physics, 96(1 1):6699-8644 1992.
[48] Yaguang Song, Xiaoshan Yang. Yaowei Wang, and Changsheng Xu. Recovering generalization via pre-training-like knowledge distillation for out-of-distribution visual question answering. IEEE Transactions on Multimedia, 2023.
[49] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In ICLR, 2018
[50] Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In NeurlPS, 2018.
[51] Charles Vorbach, Ramin Hasani, Alexander Amini, Mathias Lechner, and Daniela Rus. Causal navigation by continuous-time neural networks. In NeurIPS, pages 12425-12440, 2021.
[52] Rui Wang, Yihe Dong, Sercan Ô Arik, and Rose Yu. Koopman neural forecaster for time series with temporal distribution shifts. arXiv preprint arXiv:2210.03675, 2022.
[53] Xiaoyang Wang, Yao Ma, Yiqi Wang, Wei Jin, Xin Wang, Jiliang Tang, Caiyan Jia, and Jian Yu. Traffic flow prediction via spatial temporal graph neural network. In WWW, pages 1082-1092, 2020.
12

---

[54] Yuqing Wang, Xiangxian Li, Zhuang Qi, Jingyu Li, Xuelong Li, Xiangxu Meng, and Lei Meng. Meta-causal feature learning for 2022.
generalization. In ECCV, pages 530-545,
[55] Ziqi Wang, Marco Loog, and Jan van Gemert. Respecting domain relations: Hypothesis invariance for domain generalization. In ICPR, pages 7766-9763, 2021.
[56] Florian Wenzel, Andrea Dittadi, Peter Gehler, Carl-Johann Simon Gabone,, Max Horn, Dominik Zietlow, David Kernert, Chris Russell, Thomas Brox, Bernt Schiele, et al. Assaying out-of- distribution generalization in transfer learning. In NeurlPS, pages 7181-7198, 2022.
[57] Qitian Wu, Hengrui Zhang, Junchi Yan, and David Wipf. Handling distribution shifts on graphs: An invariance perspective. arXiv preprint arXiv:2202.02-660 2022.
[58] Tailin Wu, Takashi Maruyama, Qingqing Zhao, Gordon Wetzstein, and Jure Leskovec. Learning controllable adaptive simulation for multi-resolution physics. In ICI.R, 2023.
[59] Tailin Wu, Qinchen Wang, Yinan Zhang, Rex Ying, Kaidi Cao, Rok Sosic, Ridwan Jalali, Hassan Hamam, Marko Maucec, and Jure Leskovec. Learning large-scale subsurface simulations with a hybrid graph network simulator. In KDD, pages 4184-4194, 2022.
[60] Louis-Pascal Xhonneux, Meng Qu, and Jian Tang. Continuous graph neural networks. In ICML, pages 10432-10441, 2020.
[61] Hedi Xia, vai Suliafu, Hangjie Ji, Tan Nguyen, Andrea Bertozzi, Stanley Osher, and Bao Wang. Heavy ball neural ordinary differential equations. In NeurIPS, pages 18646-18659, 2021.
[62] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR, 2019.
[63] Chenxiao Yang, Qitian Wu, Qingsong Wen, Zhiqiang Zhou, Liang Sun, and Junchi Yan. Towards distrriiit sequential event prediction: A causal treatment. arXiv preprint arXiv:2210.13000 13005, 2022.
[64] Nianzu Yang, Kaipeng Zeng, Qitian Wu, Xiaosong Jia, and Junchi Yan. Learning substructure invariance for out-of- cittof-distribution molecular representations. In NewrIPS, 2022.
[65] Çağatay Yıldiz, Melih Kandemir, and Barbara Rakitsch. Learning interacting dynamical systems with latent gaussian process odes. In NeurIPS, pages 9188-9200 2022.
[66] Yuan Yin, Matthieu Kirchmeyer, Yves Franceschi, Alain Rakotomamonjy, and Patrick Gallinari. Continuous pde dynamics forecasting with implicit neural representations, In NeurIPS, 2022.
[67] Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: a deep learning framework for traffic forecasting. In ISCAI, pages 3634-3640) 2018.
[68] Xiaofeng Yuan, Lin Li, Yuri AW Shardt, Yalin Wang, and Chunhua Yang. Deep learning with spatiotemporal attention-based Istm for industrial soft sensor model development. IEEE Transactions on Industrial Electronics, 6805):440444444 2020.
[69] Wei Zhang, Li Zhou, Bin Yang, and Tinggui Yan. Molecular dynamics simulations of licl ion pairs in high temperature aqueous solutions by deep learning potential. Journal of Molecular Liquids, 367:120500 2022.
[70] Yanfu Zhang, Shangqian Gao, Jian Pei, and Heng Huang. Improving social network embedding via new second-order continuous graph neural networks. In KDD, pages 2515-2523 2022.
[71] Tong Zhao, Gang Liu, Daheng Wang, Wenhao Yu, and Meng Jiang. Learning from counterfac- tual links for link prediction. In ICML, pages 26911 1-966, 2022.
[72] Xin Zheng, Yixin Liu, Shirui Pan, Miao Zhang, Di Jin, and Philip S Yu. Graph neural networks for graphs with heterophily: A survey, arXiv preprint arX1v:2202.07082. 2022.
13

---

A Proof of Lemma 4.1
Lemma 4.1. With Assumptions 4.1 and 4.2, we have:
P(Y') -hi-1)== = P(r d-
(14)
P ,c",,,"' P P(ct-h (001-A) de
Proof. We have:
P Y Gu:--1)
/ P P c² G":-1) P P(c--1 (h:-1)de'
/ p p( | G P(c--1 (d
(15)
/ p P( 1 - p (c c 1 (c 6 (uett de
/ p
P c² -(- P (ce G kd dc.
de!
B Proof of Lemma 4.2
For convenience, Eqn. 9 in the main paper is repeated
dvp'
A oj W1 c2W2),
([v",... ,N, c²1) of Σ JEN-(0
(16)
ds
VD: D3
Eqn. 12
repeated as:
de ds
Cos + dof ds
Wa) + c Ws
(17)
Lemma 4.2. Given the initial state (to, ........ N ) we claim that there exists s.t. the ODE system Eqn. 16 and Eqn. has has a unique solution in the interval [to s, to + e].
To begin, introduce the Picard Lindelöf Theorem follows:
Theorem B.1. Siccard-Lindelj Theorem) Let D S IR x R be closed rectangle with (to, yo) D. Let f D -> Ra be a function which is continuous with respect to and Lipschitz continuous with respect to y. Then, there exists some > 0 such that the initial value problem:
(t) = f(t, v(t)),
y (to) = yo-
(18)
has unique solution y(t) in the interval (to E, + +
A TDg.0z
Proof.
Let A
0 & (i) and denote Mi
Then, we transpose them with Eqn.
16 and
becoming:
d(
Σ
M
+WI(0)7
ds
1=1
(19)
d(ct)T
N
w d(u;)) + W W3 (c))r)
Σ (o;)"
ds
ds
14

---

(ot)²
Let Y
R(Nxxx+ where 2, R ,c R
(o&)) (c*)T)
From the Eqn. 19 we get the ODE system di 0( with fixed parameters 0 for the ODE solver. Here the function ,(Y'; 0) is continuous w.r.t / since all components in the vector Y are continuous w.r.t t and 0 does not depend on t.
Now consider activation functions o, such as ReLU, that satisfy the following inequality for all a and y/
|((xx) - (v)l & 1 vl
Then, for any two solutions Yi' Y2 we have:
61
-
NN
ÖN+1
N
6,
Σ
Mr WT - -155,) + W. (et)² - (c))²] 1= 1,,...,,N N
1=1
N
+ WT ( d(vu.))
or ( 11 + W (c))
dx+1=
Σ
W (m1 - (mg)) |"|
ds
ds
==1
To simplify the representation, denote:
Ac = ci Cj. Auf vi 2zj.
Then by triangular inequality, we have Vi ,I,",,,,,, ,N,
N
1
Σ Aug (M W2ll2
j=l
Wi (Mr + +|Acc
j=1
(WM M
Auulla + W11
du
duy
1 =
Doy Wa+
[o(
o
) Wa Ac'WWsl
ds
ds
dvi
du
A Walla + Σ
|
o ds
Walla + I| 2
ds
N
N
duj
du
< (W
Au + W
1a2 + W Ac)))22
ds
ds
N
= (W)
| + W
|107 1|2 + 1c)
<
W
A la + WN(WM Σ | + W Al)) W1|Ac*112) +/]
N A 2|2 + (W + \N33))
=( + MNW2
15

---

Without loss of generality, we assume constants M, W > 1. Thus, we have the following result:
= 11 ... N. 6 >12
= =106f ....
+ |1 1|2 | N
||2
NNWM Σ | + A 112
(W + MNN3) MNW? Σ | + (W + ||
(WW + (W 2NW3)
Au) |
NW + (W + MNWW3
| - 12'12
Therefore the function f is L-Lipschitz with L /NW3M2 + (W + MNW3y. (icard-Linden theorem, we prove the uniqueness of the solution.
By
C Dataset Details
We evaluate our proposed CARE on four physical simulation datasets with temporal environmental variation. All these four datasets involve least one thousand nodes. Then introduce the details of these four datasets.
Lennard-Jones Potential (a.k.a. 6-12 potential) is popular in modeling electronically neutral atoms or molecules, which can be formulated as:
VL 4s (3)) -1)=-(())
12
(20)
where is the distance between particle pairs, denotes the size of the particle, denotes the depth the potential well. The first term denotes the attractive force, which decreases as the distance between particles increases. The second term denotes the repulsive force, which increases when two particles are too close. The temperature in the system is changed along with the time to model the environmental variations and high temperature would bring a more intense molecular motion.
3-body Stillinger-Wede Weber Potential provides more complex relationships besides pairwise rela- tionships in Lennard-Jone Potential. It contains both two-body and three-body terms with the following formulation:
Vsw = ΣΣ
re (ri))+ + ΣΣΣ
Φ (raj, rak, Oigh),
(21)
where 02 (ris) = AisEis Bij
Pio
721
exp
is the two-body term
Fig
Fig Pryoa
and ds (ryy, Tek, O1jk)) AesnEEsHl [cos 0igk - cos Boisk]
7a is the three-body term. The two body term similar to Lennard-Jones Potential to model the pairwise relationships and the three body term can consider the angles among atom triplets. Similarly, the temperature is changed along with the time to model the environmental variations and a high temperature would also bring more intense molecular motion.
exp
exp
CylinderFlow is popular computational fluid dynamics (CFD) simulation dataset, which models the fluid flow around a given cylinder by OpenFoam (22]. consists of simulation data from modeling an incompressible flow governed by the Navier-Stokes equations. The Reynolds number, denoted by Re, is a dimensionless quantity that characterizes the flow regime of the fluid. It defined as: Re pVD where p the density of the fluid, V is the velocity of the fluid relative the cylinder, D is the diameter of the cylinder, and μ is the dynamic viscosity of the fluid. The transition from the laminar turbulent flow usually happens at critical Reynolds number, which
16

---

depends on the geometry of the cylinder and the properties of the fluid. In general, the flow is more likely be laminar for small cylinders and viscous fluids, and more likely be turbulent for large cylinders and low-viscosity fluids. Notably, the initial flow velocity V of the incoming water flow the cylinder varies cyclically over time, meaning the Reynolds number of the flow field also changes periodically.
Airfoil is generated in a similar manner through simulations of a compressible flow by Open- Foam [22]. The lift coefficient of an airfoil relies on a number factors, e.g., the angle of attack, the shape of the airfoil, and the Reynolds number of the flow. The angle of attack is the one between the chord line of the airfoil (the straight line connecting the leading and trailing edges) and the direction of the incoming flow. The Reynolds number, as mentioned in the previous question, is dimensionless quantity that characterizes the flow regime of the fluid. It also plays a crucial role in the lift produced by an airfoil, as it determines whether the flow around the airfoil is laminar or turbulent. For laminar flow, the air moves smoothly over the surface of the airfoil, while for turbulent flow, it moves a chaotic, swirling pattern. The Reynolds number is given by: Re aVe where is the chord length of the airfoil, and μ denotes the viscosity of the fluid. The lift coefficient is typically higher for laminar flow than for turbulent flow, up to a certain point where the flow separates from the airfoil. Notably, in our simulation datasets, the inlet velocity over the wing also varies cyclically over time.
D Details of Baselines
Our proposed method is compared with a range of competing baselines as follows:
LSTM [15]
is widely recognized approach for sequence prediction problems. involves three gates, i.e., forget gate, input gate and output gate, enabling the model to acquire knowledge of long-term relationships.
STGCN (77 is deep learning approach to handle spatial dependencies and temporal dynamics in complicated spatio-temporal data. It involves a recurrent component and a message passing component for effective analysis of spatio-temporal signals.
GNS 45) utilizes a graph to represent a physical dynamical system and then utilizes message passing neural network to explore complicated dynamics and interactions among multiple objects.
MeshGraphNet [42] characterize each physical system meshes, followed by graph neural networks to learn interacting dynamics. Moreover, remeshing techniques are adopted to fit the multi-resolution nature in irregular meshes.
CG-ODE 19 models both nodes and edges jointly through two groups of ODEs, which can capture the evolution of both objective and interaction in the system.
TIE [46] attempts to improve the particle-based simulations by decomposing edges into both ends, and introducing abstract nodes to capture global information in the system.
E
Algorithm
The whole learning algorithm of CARE is summarized in Algorithm
F
Implementation Details
To ensure the accuracy of our results, we use a rigorous data split strategy, where first 80% of the samples are reserved for training purposes and the remaining 10% are set aside for testing and validating, separately, Following [19] also ensure that no sequence overlap exists on training, validation and testing sets. In particular, each of these particle-based datasets consists of 14400 training trajectories, 1800 validation trajectories, and 1800 test trajectories while 7200 training samples, 900 validation samples and 900 test trajectories are for mesh-based based datasets. During training, we split each trajectory sample into two parts, i.e., a conditional part and a prediction part. We initialize node representations and the context representation based on the first part and utilize the second part to supervise the model. The size of the two parts represented as conditional length and prediction length, respectively. We would vary two lengths to show the performance comprehensively.
17

---

Algorithm Learning Algorithm of the proposed CARE
Input: The training sequences { G , ,
Output: The parameters in our CARE.
1: Initialize the parameters in model;
2: while not convegence do
3: for each training sequence do
4: Divide the sequence into two segments;
5: Build the temporal graph using Eqn. 4
6:
Initialize both node representations and context representations for ODEs using Eqn. 8
7:
Solve the coupled ODEs, i.e., Eqns. I1
8:
Add noise into the input for perturbed hidden states;
9:
Feed these hidden states into a decoder (-).) to get the predictions;
10:
Calculate the loss in Eqn. 13]
Update the parameters
CARE using back propagation;
12:
end for end while
13:
In our implementation, we utilize the sum operator AGG in Eqn. To solve the ODE systems on time grid which five times denser than the observed time steps, we employ the fourth-order Runge- Kutta method as in the torchdiffeq Python package [24), using Py PyToochh [40]. We also use the adjoint method [5] to reduce memory usage. All experiments are conducted on single NVIDIA A100 GPU. We set the latent dimension to 256 and the dropout rate to 0.2. For optimization, the Adam optimizer with weight decay by mini-batch stochastic gradient descent, setting the learning rate to 0.01. Overall, our proposed CARE offers novel approach for modeling and predicting complex systems with multiple interacting objects.
G More Experiment Results
G.1 Model Comparison
We first compare our CARE with MP-NODE [10], which ODE-based approach for homogeneous dynamical systems. The compared result on particle-based simulation datasets and mesh-based simulation datasets recorded in Table 4 and Table [ respectively. From the results, can validate the superiority our CARE in tackling the temporal environmental variation and making accurate long-term predictions.
Table Results on particle-based physical simulations with the prediction lengths 1, 5, 10 and 20. Up. by and v. represent the velocity in the direction of each coordinate axis.
Prediction Length
+1
+10
Variahle
U2
1/s
0yy
Fyy
0yy
Sennard-Joner Potential
MP-NODE
1.45
1.79
1.41
3.08
3.74
3.02
5.36
5.91
5.26
8.46
8.36
8.95
Ours
0.76
0.89
1.01
2.94
3.16
2.85
5.01
4.69
4.71
5.75
5.91
5.82
3-body Stillinger Weber Potential
MP-NODE
9.95
9.82
9.87
12.47
12.41
12.49
16.05
16.14
16.10
17.06
17.15
17.09
Ours
4.21
4.29
4.18
9.74
9.79
9.71
13.65
13.71
13.57
15.30
15.39
15.35
G.2 Results with Different Prediction Lengths
To evaluate the performance of the proposed CARE in different settings, we vary the prediction length and compare the performance of different approaches. In particular, we show the results with the prediction length (8, 15) on two particle-based simulation datasets in 6 The results with the prediction length (15, (0) on two mesh-based simulation datasets are shown in Table From the
compared results, we can validate the superiority of our CARE in various settings.
18

---

Table 5: Results mesh- based physical simulations different prediction lengths 1, 10, 20 and 50. Uz My and represent the velocity in different directions and the pressure field, respectively.
Prediction Length
+10
+20
+50
Variable
1/2
8y
V2
Ey
P
Vp
V/
0/
Vy
p
FylinderFlow
MP-NODE
1.11
20.6
8.62
3.68
37.2
13.8
4.36
38.8
17.7
5.59
61.8
32.7
Ours
0.87
19.1
7.21
3.02
32.9
11.8
3.95
37.8
13.9
4.97
55.8
29.4
Airfoil
MP-NODE
4.41
4.44
0.38
4.85
4.76
0.49
5.89
5.72
1.23
7.45
7.97
1.78
Ours
3.51
4.11
0.19
3.86
3.75
0.34
4.16
4.12
0.45
6.74
6.82
0.81
Table 6: Results particle-based physical simulations with the prediction lengths and 15. Ugy Uyy and t;2 represent the velocity in the direction of each coordinate axis.
Prediction Length
+8
+15
Variable
Ug
Vy
U=
Ua
By
U=
Lennard-Jones Potential
LSTM
9.44
9.40
9.57
12.68
12.75
12.61
GNS
8.86
8.92
8.85
11.99
11.84
12.08
STGCN
6.28
6.24
6.33
7.85
8.01
8.09
MeshGraphNet
6.47
6.42
6.55
7.99
8.08
8.14
CG-ODE
5.33
5,16
5.17
7.94
7.59
7.71
TIE
4.97
4.68
4.71
7.65
7.28
7.44
Ours
4.85
4.17
4.55
5,52
5.19
5.06
3-body Stillinger- Weber Potential
LSTM
22.89
22.93
22.90
26.74
26.79
26.78
GNS
22.36
22.38
22.31
26.04
26.05
26.02
STGCN
17.79
17.88
17.83
21.48
21.42
21.46
MeshGraphNet
17.92
17.84
17.95
21.86
21.82
21.84
CG-ODE
13.61
13.68
13.63
17.11
17.15
17.08
TIE
16.79
16.76
16.81
20.04
20.08
20.02
Ours
11.94
11.97
11.88
14.89
14.95
14.81
G.3 Visualization
Moreover, we show more visualization of our proposed CARE and the best baseline (i.e., TIE) compared with the ground truth. In particular, given that have shown the time steps in {1, 100, 200, 300, 400, 500} in Figure and Figure 4, now we select them in {5, 150, 250, 350, 450} for Lennard-Jonee Potential and CylinderFlow. respectively. The compared results
shown in Figure 5 and Figure 6 We can observe that our CARE can generate more accurate trajectories compared with the baselines in most cases, which validates the superiority of our CARE again. For example, in the last row in Figure(3] our CARE can accurately recover the velocity distribution around the cylinder while the baseline fails.
H More Related Work
Graph Neural Networks. Graph Neural Networks (GNNs) have garnered significant success for
their remarkable capabilities in graph representation learning [26 62 72 33 13), which is integral
to a variety of downstream applications, including node classification 26) link prediction [7]],
and graph classification [22] These approaches typically employ the message-passing mechanism, enabling the iterative updating of node representations with the aid of neighboring information. Recently, GNNs have successfully branched out into modeling interactive dynamics [2 46 45 For
19

---

Table 7: Results mesh-based physical simulations different prediction lengths 15 and 30. Uz. By and represent the velocity in different directions and the pressure field, respectively.
Prediction Length
+15
+30
Variable
yy
p
Va
Vy
CylinderFlow
LSTM
8.25
47.43
17.62
11.09
59.86
33,24
GNS
8.16
47.95
17.54
11.36
60.49
33.72
STGCN
6.27
44.37
16.97
7.03
56.18
31.45
MeshGraphNet
5.13
42.05
14.65
6,54
52.96
28.63
CG-ODE
3.92
37.45
13.79
5.28
47.12
23.69
TIE
4.07
37,91
13.72
5.31
47.17
23.65
Ours
3.48
35.6
12.9
4.26
44.9
20.7
Airfoil
LSTM
8.75
8.84
3.96
12.86
18.74
4.78
GNS
8.14
8.02
3.51
11.52
11.44
4.04
STGCN
7.64
7.38
2.79
10.14
10.16
3.14
MeshGraphNet
6.05
6.19
1.27
7.11
7.09
1.43
CG-ODE
5.68
5.53
0.81
6.82
6.88
1,21
TIE
5.41
5.33
0.79
6.64
6.71
1.19
Ours
4.08
4.02
0.41
4.95
5.11
0.62
TIE
CARE
Ground Truth
step
150 step
250 step
350 step
450 step
Figure 5: More visualization velocity in CylinderFlow dataset with varying time steps in [5, 150, 250, 350, 450).
employs message passing neural network to facilitate the modeling of interactions between objectives, thereby outputting the next-time predictions. However, an inherent drawback lies in the inability of discrete GNNs to encapsulate the continuous nature of system
instance, MeshGraphNet 42
20

---

5 step
150 step
250 step
350 step
450 step
TIE
CARE
Ground Truth
Figure More visualization of Lennard-Jones Potential with varying time steps in (5, 150, 250, 350, 450).
dynamics. To relieve this limitation, we present a novel graph- based ODE system named CARE for the modeling of interacting dynamics, which enriches the capabilities of making long-term predictions under potential environmental variation.
Graph-based ODE. Neural ODEs have been integrated into GNNs, resulting in the development of Graph ODEs that are applicable to both static and dynamic graphs. Graph ODEs static graphs (60 770 primarily aim to mitigate overfitting by formalizing derivatives using both initial and immediate node representations. Meanwhile, Graph ODEs on dynamic graphs are utilized for tasks such as traffic flow forecasting [9] and social analysis [19] demonstrating effective performance on irregularly sampled partial observation data. For example, STGODE [ employs tensor computation conduct continuous message passing, which facilitates accurate long-term predictions by overcoming the network depth limitations. Despite these advancements, existing works fall short of addressing the temporal environmental variation in interacting dynamics. To fill this gap. we propose novel approach CARE handle this problem by injecting a context variable in the Graph ODE system.
I Potential Negative Impacts
To the best of our knowledge, we have not found any negative impact of our work.
21


Reviews:
[{"review_id": "T93WpB7UXq", "reviewer": "Reviewer_ZaqZ", "summary": "The main idea of this work is to propose a novel approach called Context-attended Graph ODE (CARE) for modeling interacting dynamical systems under temporal distribution shift. The paper formalizes the problem of temporal distribution shift in interacting dynamics modeling and proposes a probabilistic framework to understand the relationships between trajectories and contexts. Experimental results demonstrate the exceptional performance of the CARE model, outperforming state-of-the-art methods in accurately predicting long-term trajectories, even in the presence of environmental variations.", "questions": "- Where is Lemma 4.2?\n- How to set the threshold mentioned in Line 159?\n- Will the dynamic graph updating with larger $\\delta s$ leads to many indirect connections and how will it affect the results?\n- How is the performance of proposed CARE on dynamical systems with more nodes, let's say, 10, 100 and more?\n- As shown in table 2, CARE still suffers from accumulated error on longer time-series. Do you have any idea on how to decrease the accumulated error?", "limitations": "The limitation is discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Novel approach: The CARE model introduced in this work presents a novel methodology by incorporating continuous context variations and system states into a coupled ODE system for modeling interacting dynamical systems under temporal distribution shift.\n- Superior performance: Extensive experimental results demonstrate that the CARE model outperforms state-of-the-art approaches in accurately predicting long-term trajectories, even in the presence of environmental variations.\n- Clarity: The paper effectively summarizes its contributions, including problem formalization, novel methodology, and comprehensive experiments. This clarity allows readers to readily grasp the significance of the research.\n- Extensive experiments: The paper conducts a thorough evaluation of the CARE model by performing experiments on diverse dynamical systems. The results highlight the superiority of the proposed approach compared to state-of-the-art methods. Additionally, the paper includes an ablation study and a parameter sensitivity analysis to further validate the effectiveness of the CARE model.", "weaknesses": "I don't see any weakness in the paper. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful for the time you have taken to review our paper, your insightful comments and support. Your positive feedback is incredibly encouraging for us! In the following response, we would like to address your major concern and provide additional clarification.\n\n> Q1: Where is Lemma 4.2?\n\nA1: Thanks for the comment. Lemma 4.2 can be found on line 206, and its corresponding proof is detailed in Appendix C.\n\n> Q2: How to set the threshold mentioned in Line 159?\n\nA2: Thanks for the comment. When constructing the graph structure, we set the threshold at the 30th percentile of all distances. For further validation of the robustness of this parameter choice, we experiment with varying percentiles in {25, 30, 35} on two datasets. The results are shown below, , which demonstrates the robustness of our parameter selection.\n\n| Percentile          \t| 25 |  \t|  \t|  30 |  \t|  \t| 35|  \t|  \t|\n|-------------------------|:----:|:----:|:----:|:----:|:----:|:----:|:----:|------|------|\n| Lennard-Jones Potential |  \t|  \t|  \t|  \t|  \t|  \t|  \t|  \t|  \t|\n| Variable            \t| $v_x$  | $v_y$  | $v_z$  | $v_x$  | $v_y$  | $v_z$  | $v_x$  | $v_y$  | $v_z$  |\n| Ours                \t| 5.98 | 6.07 | 6.05 | 5.75 | 5.91 | 5.82 | 5.93 | 6.06 | 6.02\t|\n| CylinderFlow        \t|  \t|  \t|  \t|  \t|  \t|  \t| \t |  \t|  \t|\n| Variable            \t|  $v_x$ |  $v_y$ |  $p$  |  $v_x$ |  $v_y$ |   $p$  |  $v_x$ |  $v_y$ |  $ p$  |\n| Ours                \t| 4.13 | 38.2 | 13.9 | 3.95 | 37.8 | 13.9 | 4.08 | 37.9 | 13.9 |\n\n\n> Q3: Will the dynamic graph updating with larger $\\Delta_s$ leads to many indirect connections and how will it affect the results?\n\nA3: Thanks for the comment. Indeed, an excessively large $\\Delta_s$ might introduce a number of indirect connections, which in turn could potentially degrade performance. To empirically validate this, we have conducted experiments. As evidenced in Fig.5(c), it is clear that a large $\\Delta_s$ (exceeding 20) would decrease the performance.\n\n> Q4: How is the performance of proposed CARE on dynamical systems with more nodes, let's say, 10, 100 and more?\n\nA4. Thanks for the comment. Actually, the experiments we conducted already involved systems with a significant number of nodes, well over the figures you have mentioned. Specifically, the datasets and their respective node numbers are as follows:\n\n| Dataset                        | Number of Nodes |\n|--------------------------------|-----------------|\n| Lennard-Jones Potential        | 1000            |\n| 3-body Stillinger-Weber Potential | 1000        |\n| CylinderFlow                   | 9200            |\n| Airfoil                        | 10720           |\n\nWe'll ensure these details are more clearly stated in our revised manuscript.\n\n> Q5: As shown in table 2, CARE still suffers from accumulated error on longer time-series. Do you have any idea on how to decrease the accumulated error?\n\nA5. Thanks for the comment. Error accumulation is always a longstanding problem in long-term prediction. To mitigate this issue, we're considering several strategies for our future research:\n- **Enhanced Model Capacity**. We can develop more advanced graph ODE models with higher capacity, e.g., second-order ODE and augmented ODE. These advanced models could fit the spatial data better, which has the potential to decrease the accumulated error.\n- **Adaptive Settings**. We can utilize active learning and online learning to have more observations, which would provide valuable feedback to enable further fine-tuning and adjustments during long-term predictions. \n- **Ensemble Learning**. Incorporating multiple models with varying time horizons can be a promising strategy. By combining their strengths, we can potentially minimize both bias and variance of the prediction, further decreasing accumulated error.\n\nThanks again for appreciating our work and for your constructive suggestions. Please let us know if you have further questions.\n"}, {"review_id": "MqWdVCIHKR", "reviewer": "Reviewer_P3pg", "summary": "The paper proposes a new model architecture for modeling multi-agent dynamics. The model consists of three parts (encoder, dynamics, decoder) where the encoder initializes the latent state of each agent and the context, the dynamics is modeled via NeuralODEs and the decoder is a standard MLP. New to the paper is that the model explicitly consider a context variable that evolves over time.", "questions": "### Interpretability\nCan you please comment on the interpretability of the context variable? I think it is not and this should also be clearly stated in the manuscript or demonstrated otherwise.\n\n### Naive solutions\nI could also come up with a naive solution in which the context variable is modeled as N+1-th object in the graph that is connected to all other variables but does not have any observations. How would that compare to the aforementioned solution?\nHow does a static context variable compare to the dynamic solution? Static context variables have for instance also been considered in [1] which should be cited.\n\n### References\n[1] Yıldız, Çağatay, Melih Kandemir, and Barbara Rakitsch. \"Learning interacting dynamical systems with latent Gaussian process ODEs.\" Advances in Neural Information Processing Systems 35 (2022): 9188-9200.\n\n", "limitations": "None.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### New Architecture\nThe explicit modeling of the context variable and its temporal evolution is new to me. I like the idea and the experiments support its benefits. \n\n### Experiments\nThe experiments in the paper are exhaustive and show the advantage of the new method compared to competitors. I also like the ablation study that shows the merits of the individual contributions.\n", "weaknesses": "I think the paper slightly oversells what it is doing:\n\n### Temporal distribution shift\nThe paper is called \"modeling interacting dynamics under temporal distribution shift\" but there is very little about temporal distribution shift in the data. I would have expected that for instance the temperature in Sec. 5.1 is different between training and test data. If this is the case, the authors should state it more explicitly in the experiments. If not, I would suggest to tone it down a bit.\n\n### Probabilistic Modeling\n I like the new model architecture and it seems principled to me. However, the paper states in the abstract \"provide a probabilistic view for out-of-distribution dynamics\", whlie the learnt model is completely deterministic. I would also suggest here to not overstate the contribution.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful for the time you have taken to review our paper, your insightful comments and support. Your positive feedback is incredibly encouraging for us! In the following response, we would like to address your major concern and provide additional clarification.\n\n> Q1: The paper is called \"modeling interacting dynamics under temporal distribution shift\" but there is very little about temporal distribution shift in the data. I would have expected that for instance the temperature in Sec. 5.1 is different between training and test data. If this is the case, the authors should state it more explicitly in the experiments. If not, I would suggest to tone it down a bit.\n\nA1. Thanks for your comment. Indeed, in particle-based systems, individual samples often exhibit different initial temperatures and laws of change, leading to distinct dynamics. As a result, the training and test data inherently possess varying dynamic contexts, which in turn introduces different dynamics. We acknowledge the potential lack of clarity on this aspect in the paper and will also tone it down a bit to make it more clear.\n\n> Q2: I like the new model architecture and it seems principled to me. However, the paper states in the abstract \"provide a probabilistic view for out-of-distribution dynamics\", while the learnt model is completely deterministic. I would also suggest here to not overstate the contribution.\n\nA2. Thanks for your comment. We take your suggestion and will adjust the phrasing to more accurately state \"provide a view for context-varying dynamics\". We value your feedback and strive for clarity and accuracy in our work.\n\n> Q3: Can you please comment on the interpretability of the context variable? I think it is not and this should also be clearly stated in the manuscript or demonstrated otherwise.\n\nA3. Thanks for your comment. We have visualized the first 10 dimensions of the context variable in Figure A. From this visualization, we can find that the context variable experiences rapid changes over time. This variation supports our assertion that a dynamic context variable is crucial for adapting to the shifting environments found in dynamical systems. Moreover, the complex dynamics of the context variable indicates that the variance of environments (e.g., temperature) could bring in complicated impacts to the dynamical systems. \n\n> Q4. I could also come up with a naive solution in which the context variable is modeled as N+1-th object in the graph that is connected to all other variables but does not have any observations. How would that compare to the aforementioned solution? How does a static context variable compare to the dynamic solution? Static context variables have for instance also been considered in [1] which should be cited.\n\nA4. Thanks for your comment. Following your suggestion, we have added two model variants below:\n- CARE-O, which includes the N+1-th object in the graph connected to all other variables;\n- CARE-S, which utilizes a static context variable instead. \n\nThe compared performance on two datasets is recorded as below. From the results, we can obtain that the full model perform better than CARE-O, which shows that directly combines heterogeneous nodes, i.e., objects and contexts on a simple graph could overlook critical information, e.g., the gradients of node representations in Eq. 11. \nMoreover, we can find that models relying on static context variables, i.e., CARE-S performs worse since it is hard to capture the dynamic nature of varied environments. We will definitely cite [1] in our revised version. \n\n****\n| Dataset  | Lennard-Jones Potential |  \t|  \t| CylinderFlow |  \t|  \t|\n|----------|:-----------------------:|:----:|:----:|:------------:|:----:|:----:|\n| Variable |       \t$v_x$       \t|  $v_y$ |  $v_z$ |  \t$v_x$ \t|  $v_y$ |   $p$  |\n| CARE-O   |       \t6.69      \t| 6.80 | 6.73 | \t4.27 \t| 38.4 | 14.2 |\n| CARE-S  |       \t7.96      \t| 8.18 | 8.01 | \t5.32 \t| 39.6 | 15.8 |\n| CARE \t|       \t5.75      \t| 5.91 | 5.82 | \t3.95 \t| 37.8 | 13.9 |\n\nThanks again for appreciating our work and for your constructive suggestions. Please let us know if you have further questions.\n"}, {"review_id": "39AFVSJqSe", "reviewer": "Reviewer_QmzE", "summary": "The paper aims to model out-of-distribution dynamics of environments with interacting entities using a seq-to-seq style graph neural net architecture. The proposed model attempts to capture the invariant aspects of the dynamical environment via a context embedding that follows a neural ordinary differential equation. The paper reports results on a number of physical systems modeling tasks, as well as some theoretical aspects of the proposed approach.", "questions": "Although the paper motivates the proposed methodology with the primary application of generalization over out-of-distribution cases, I am not able to point out how the proposed experiments test this aspect and according to which metric. The results tables appear to quantify model performance only in in-distribution use cases, i.e. both training and test data are assumed to follow the same dynamics. Is my understanding correct? If not, how exactly is the out-of-distribution data acquired and what is the actual difference between the train and test splits that makes the test observations out-of-distribution?\n\nSince the answers to the above questions play a critical role in my evaluation, for now I set my grade to borderline. It is likely to swing based on further interactions with the authors and other reviewers.\n", "limitations": "Section 6 of the paper discusses the limitations of the proposed approach in sufficient detail.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is very well written. Figure 2 describes the methodology simply and clearly. The paper reports a large set of experiments and shows consistent improvement over a large set of benchmarks, though I think it misses the most essential ones (see my comments below). The paper exercises a systematic scientific writing approach where the key assumptions are pointed out and their most important analytical conclusions, such as model consistency, are analyzed, although with rather straightforward proof techniques.", "weaknesses": "The proposed method is novel per se. It is also intuitive and well-justified, but it appears to put together a number of existing tools in the most straightforward way one can think of. While I am convinced by the quality of the proposed solution, I am a bit skeptical about its scientific value, i.e. how exactly it enhances our knowledge base.\n\nThe conceptual novelty of the proposed method over some existing graph-based probabilistic ODE approaches such as IGP-ODE [60] and [Ref1] is not clarified. It is also not obvious to the reader why these methods that have suitable specs for handling out-of-distribution data, due to rigorous uncertainty modeling, should not be among the list of models in comparison. For instance IGP-ODE also model interaction dynamics, it reports disentanglement results with the motivation of out-of-distribution detection, and it also takes the dynamics of context variables to its center.\n\n[Ref1] Look et al., Cheap and Deterministic Inference for Deep State-Space Models of Interacting Dynamical Systems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful for the time you have taken to review our paper and your insightful review. Here we address your comments in the following.\n\n> Q1. The proposed method is novel per se. It is also intuitive and well-justified, but it appears to put together a number of existing tools in the most straightforward way one can think of. While I am convinced by the quality of the proposed solution, I am a bit skeptical about its scientific value, i.e. how exactly it enhances our knowledge base.\n\nA1. Thanks for your acknowledgement of our novelty. Our method is totally based on the proposed probabilistic model, which could make our method a little straightforward. The scientific value of our proposed method is two-fold:\n- **Real-world Applications.** Our approach is tailored to dynamical systems affected by fluctuating environments. Such scenarios are prevalent in real-world applications. Examples include particle-based systems experiencing variable temperatures or unsteady flows influenced by different Reynolds numbers. By incorporating context variances, we can tackle these complex and real-world dynamical systems.\n\n- **Disentangling Representations.** A salient feature of our method is its ability to distinguish between object-specific and context representations. This disentanglement offers deeper insights into the impact of shifting environments on dynamical systems. We believe this perspective can serve as a crucial foundation for subsequent research in the area.\n\n\n\n> Q2. The conceptual novelty of the proposed method over some existing graph-based probabilistic ODE approaches such as IGP-ODE [60] and [Ref1] is not clarified. It is also not obvious to the reader why these methods that have suitable specs for handling out-of-distribution data, due to rigorous uncertainty modeling, should not be among the list of models in comparison. For instance IGP-ODE also model interaction dynamics, it reports disentanglement results with the motivation of out-of-distribution detection, and it also takes the dynamics of context variables to its center.\n\nA2. Thanks for your comment. We first compare these two methods with the proposed methods. IGP-ODE combines latent Gaussian process with ODEs while GDSSM [ref1] utilizes the graph neural networks with a Gaussian mixture model. The differences between these two works and ours lie in three points:\n- **Different Objectives**. Our method primarily hones in on addressing the intricacies of dynamical systems in the presence of fluctuating environments. In contrast, both IGP-ODE and GDSSM, do not encompass this specific challenge.\n- **Different Methodology**: Our methods introduce a context variable to model the dynamics of environments based on a probabilistic model while these two methods focus on modeling the dynamics of objects using Gaussian processes and Gaussian Mixture Model, respectively. \n- **Different Efficiency**. Our method is tailored to efficiently cater to both particle-based and mesh-based systems, even when these encompass a vast array of nodes. Conversely, the models like IGP-ODE and GDSSM concentrate on vehicular and kinematic systems, leveraging intricate modeling techniques through Gaussian processes and the Gaussian Mixture Model. \n\n\nIn our practical experiments, both these methods faced Out-Of-Memory challenges in our setup. Specifically, these models encountered OOM issues with datasets exceeding 30 nodes, whereas our datasets consistently feature thousands of nodes.\n\n\n| Number of nodes | IGP-ODE   | GDSSM    |\n|-----------------|-----------|----------|\n| 10              | 23.57GB   | 19.65GB  |\n| 20              | 62.54GB   | 68.23GB  |\n| 30              | OOM       | OOM      |\n\n\n\n> Q3. Although the paper motivates the proposed methodology with the primary application of generalization over out-of-distribution cases, I am not able to point out how the proposed experiments test this aspect and according to which metric. The results tables appear to quantify model performance only in in-distribution use cases, i.e. both training and test data are assumed to follow the same dynamics. Is my understanding correct? If not, how exactly is the out-of-distribution data acquired and what is the actual difference between the train and test splits that makes the test observations out-of-distribution?\n\nA3. Thanks for your comment. Indeed, in particle-based systems, individual samples often exhibit different initial temperatures and laws of change, leading to distinct dynamics. As a result, the training and test data inherently possess varying dynamic contexts, which in turn introduces different dynamics. Furthermore, within individual samples, the ever-changing temperature introduces temporal distribution shifts, thereby causing the data distribution to evolve over time. It is this temporal variability and context-driven change that we refer to when discussing \"out-of-distribution.\" To ensure greater clarity, we will rephrase our approach as addressing \"context-varying dynamics\" to better convey the challenges which our method seeks to tackle.\n \nIn light of these responses, we hope we have addressed your concerns, and hope you will consider raising your score. If there are any additional notable points of concern that we have not yet addressed, please do not hesitate to share them, and we will promptly attend to those points.\n\n**Reference**\n\n[Ref1] Look et al., Cheap and Deterministic Inference for Deep State-Space Models of Interacting Dynamical Systems.\n"}]

Author rebuttal:
Dear Reviewers,

We thank you for your careful reviews and constructive suggestions. We acknowledge the positive comments such as "Novel approach" (Reviewer ZaqZ), “Superior performance” (Reviewer ZaqZ), "Clarity" (Reviewer ZaqZ), "Extensive experiments” (Reviewer ZaqZ), “I like the idea and the experiments support its benefits.” (Reviewer P3pg), “The experiments in the paper are exhaustive” (Reviewer P3pg), “The proposed method is novel per se.” (Reviewer QmzE), “The paper is very well written” (Reviewer QmzE), “reports a large set of experiments and shows consistent improvement” (Reviewer QmzE). We have also responded to your questions point by point.

The figure results are attached in the PDF. Please let us know if you have any follow-up questions. We will be happy to answer them.

Best regards,

the Authors


Meta-review: The paper extends works on interacting dynamical systems to handle temporal distribution shift caused by environmental variations. The scenario present as well as the approach to handle it is novel, and the paper will be a useful contribution to the research community. The reviewers also agreed that the paper is well-written and proved its claims with extensive experiments. As such we decided that this paper should be accepted to this conference.
Decision: Accept (poster)