SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| f \left(\mathbf {Y} _ {1} ^ {s}; \theta\right) - f \left(\mathbf {Y} _ {2} ^ {s}; \theta\right) \right\| _ {2} ^ {2} \leq \left\| \left(\delta_ {1} ^ {T}, \dots , \delta_ {N} ^ {T}, \delta_ {N + 1} ^ {T}\right) \right\| _ {2} ^ {2}, \\ = \left\| \delta_ {1} ^ {T} \right\| _ {2} ^ {2} + \dots + \left\| \delta_ {N} ^ {T} \right\| _ {2} ^ {2} + \left\| \delta_ {N + 1} ^ {T} \right\| _ {2} ^ {2} \\ \leq N \left(W M \sum_ {j = 1} ^ {N} \| \Delta \boldsymbol {v} _ {j} ^ {s} \| _ {2} + W \| \Delta \boldsymbol {c} ^ {s} \| _ {2}\right) ^ {2} + \\ \left(\left(W + M N W ^ {2}\right) \sum_ {i = 1} ^ {N} \| \Delta \boldsymbol {v} _ {i} ^ {s} \| _ {2} + \left(W + N W ^ {2}\right) \| \Delta \boldsymbol {c} ^ {s} \| _ {2}\right) ^ {2} \\ \leq \left[ N W ^ {2} M ^ {2} + (W + M N W ^ {2}) ^ {2} \right] \left(\sum_ {j = 1} ^ {N} \| \Delta \boldsymbol {v} _ {j} ^ {s} \| _ {2} + \| \Delta \boldsymbol {c} ^ {s} \| _ {2}\right) ^ {2} \\ = \left[ N W ^ {2} M ^ {2} + (W + M N W ^ {2}) ^ {2} \right] \| \mathbf {Y} _ {1} ^ {s} - \mathbf {Y} _ {2} ^ {s} \| _ {2} ^ {2}. \\ \end{array}
$$
Therefore the function  $f$  is L-Lipschitz with  $L = \sqrt{NW^2M^2 + (W + MNW^2)^2}$ . By Picard-Lindelöf theorem, we prove the uniqueness of the solution.
C Dataset Details
We evaluate our proposed CARE on four physical simulation datasets with temporal environmental variation. All these four datasets involve at least one thousand nodes. Then we introduce the details of these four datasets.
- Lennard-Jones Potential (a.k.a. 6-12 potential) is popular in modeling electronically neutral atoms or molecules, which can be formulated as:
$$
V _ {\mathrm {L J}} = 4 \varepsilon \left[ \left(\frac {\sigma}{r}\right) ^ {1 2} - \left(\frac {\sigma}{r}\right) ^ {6} \right], \tag {20}
$$
where  $r$  is the distance between particle pairs,  $\sigma$  denotes the size of the particle,  $\epsilon$  denotes the depth of the potential well. The first term denotes the attractive force, which decreases as the distance between particles increases. The second term denotes the repulsive force, which increases when two particles are too close. The temperature in the system is changed along with the time to model the environmental variations and a high temperature would bring a more intense molecular motion.
- 3-body Stillinger-Weber Potential provides more complex relationships besides pairwise relationships in Lennard-Jones Potential. It contains both two-body and three-body terms with the following formulation:
$$
V _ {\mathrm {S W}} = \sum_ {i} \sum_ {j > i} \phi_ {2} \left(r _ {i j}\right) + \sum_ {i} \sum_ {j \neq i} \sum_ {k > j} \phi_ {3} \left(r _ {i j}, r _ {i k}, \theta_ {i j k}\right), \tag {21}
$$
where  $\phi_2(r_{ij}) = A_{ij}\epsilon_{ij}\left[B_{ij}\left(\frac{\sigma_{ij}}{r_{ij}}\right)^{p_{ij}} - \left(\frac{\sigma_{ij}}{r_{ij}}\right)^{q_{ij}}\right]\exp \left(\frac{\sigma_{ij}}{r_{ij} - a_{ij}\sigma_{ij}}\right)$  is the two-body term and  $\phi_3(r_{ij},r_{ik},\theta_{ijk}) = \lambda_{ijk}\epsilon_{ijk}[\cos \theta_{ijk} - \cos \theta_{0ijk}]^2\exp \left(\frac{\gamma_{ij}\sigma_{ij}}{r_{ij} - a_{ij}\sigma_{ij}}\right)\exp \left(\frac{\gamma_{ik}\sigma_{ik}}{r_{ik} - a_{ik}\sigma_{ik}}\right)$  is the three-body term. The two body term is similar to Lennard-Jones Potential to model the pairwise relationships and the three body term can consider the angles among atom triplets. Similarly, the temperature is changed along with the time to model the environmental variations and a high temperature would also bring a more intense molecular motion.
- CylinderFlow is a popular computational fluid dynamics (CFD) simulation dataset, which models the fluid flow around a given cylinder by OpenFoam [22]. It consists of simulation data from modeling an incompressible flow governed by the Navier-Stokes equations. The Reynolds number, denoted by Re, is a dimensionless quantity that characterizes the flow regime of the fluid. It is defined as:  $Re = \frac{\rho V D}{\mu}$ , where  $\rho$  is the density of the fluid,  $V$  is the velocity of the fluid relative to the cylinder,  $D$  is the diameter of the cylinder, and  $\mu$  is the dynamic viscosity of the fluid. The transition from the laminar to turbulent flow usually happens at a critical Reynolds number, which
depends on the geometry of the cylinder and the properties of the fluid. In general, the flow is more likely to be laminar for small cylinders and viscous fluids, and more likely to be turbulent for large cylinders and low-viscosity fluids. Notably, the initial flow velocity  $V$  of the incoming water flow to the cylinder varies cyclically over time, meaning the Reynolds number of the flow field also changes periodically.
- Airfoil is generated in a similar manner through simulations of a compressible flow by OpenFoam [22]. The lift coefficient of an airfoil relies on a number of factors, e.g., the angle of attack, the shape of the airfoil, and the Reynolds number of the flow. The angle of attack is the one between the chord line of the airfoil (the straight line connecting the leading and trailing edges) and the direction of the incoming flow. The Reynolds number, as mentioned in the previous question, is a dimensionless quantity that characterizes the flow regime of the fluid. It also plays a crucial role in the lift produced by an airfoil, as it determines whether the flow around the airfoil is laminar or turbulent. For laminar flow, the air moves smoothly over the surface of the airfoil, while for turbulent flow, it moves in a chaotic, swirling pattern. The Reynolds number is given by:  $Re = \frac{\rho V c}{\mu}$ , where  $c$  is the chord length of the airfoil, and  $\mu$  denotes the viscosity of the fluid. The lift coefficient is typically higher for laminar flow than for turbulent flow, up to a certain point where the flow separates from the airfoil. Notably, in our simulation datasets, the inlet velocity  $V$  over the wing also varies cyclically over time.
D Details of Baselines
Our proposed method is compared with a range of competing baselines as follows:
- LSTM [15] is a widely recognized approach for sequence prediction problems. It involves three gates, i.e., forget gate, input gate and output gate, enabling the model to acquire knowledge of long-term relationships.
- STGCN [67] is a deep learning approach to handle spatial dependencies and temporal dynamics in complicated spatio-temporal data. It involves a recurrent component and a message passing component for effective analysis of spatio-temporal signals.
- GNS [45] utilizes a graph to represent a physical dynamical system and then utilizes a message passing neural network to explore complicated dynamics and interactions among multiple objects.
- MeshGraphNet [42] characterize each physical system as meshes, followed by graph neural networks to learn interacting dynamics. Moreover, remeshing techniques are adopted to fit the multi-resolution nature in irregular meshes.
- CG-ODE [19] models both nodes and edges jointly through two groups of ODEs, which can capture the evolution of both objective and interaction in the system.
- TIE [46] attempts to improve the particle-based simulations by decomposing edges into both ends, and introducing abstract nodes to capture global information in the system.
E Algorithm
The whole learning algorithm of CARE is summarized in Algorithm 1.
F Implementation Details
To ensure the accuracy of our results, we use a rigorous data split strategy, where first  $80\%$  of the samples are reserved for training purposes and the remaining  $10\%$  are set aside for testing and validating, separately. Following [19], we also ensure that no sequence overlap exists on training, validation and testing sets. In particular, each of these particle-based datasets consists of 14400 training trajectories, 1800 validation trajectories, and 1800 test trajectories while 7200 training samples, 900 validation samples and 900 test trajectories are for mesh-based datasets. During training, we split each trajectory sample into two parts, i.e., a conditional part and a prediction part. We initialize node representations and the context representation based on the first part and utilize the second part to supervise the model. The size of the two parts is represented as conditional length and prediction length, respectively. We would vary two lengths to show the performance comprehensively.
Algorithm 1 Learning Algorithm of the proposed CARE
Input: The training sequences  $\{G^0, G^1, \dots, G^t\}$ .
Output: The parameters in our CARE.
1: Initialize the parameters in our model;
2: while not convergence do
3: for each training sequence do
4: Divide the sequence into two segments;
5: Build the temporal graph using Eqn. 4;
6: Initialize both node representations and context representations for ODEs using Eqn. 8;
7: Solve the coupled ODEs, i.e., Eqns. 9 and 11;
8: Add noise into the input for perturbed hidden states;
9: Feed these hidden states into a decoder  $\Phi^d (\cdot)$  to get the predictions;
0: Calculate the loss in Eqn. 13;
1: Update the parameters in CARE using back propagation;
2: end for
3: end while
In our implementation, we utilize the sum operator as AGG in Eqn. 11. To solve the ODE systems on a time grid which is five times denser than the observed time steps, we employ the fourth-order Runge-Kutta method as in the torchdiffeq Python package [24], using PyTorch [40]. We also use the adjoint method [5] to reduce memory usage. All experiments are conducted on a single NVIDIA A100 GPU. We set the latent dimension to 256 and the dropout rate to 0.2. For optimization, we use the Adam optimizer with weight decay by mini-batch stochastic gradient descent, setting the learning rate to 0.01. Overall, our proposed CARE offers a novel approach for modeling and predicting complex systems with multiple interacting objects.
G More Experiment Results
G.1 Model Comparison
We first compare our CARE with MP-NODE [10], which is an ODE-based approach for homogeneous dynamical systems. The compared result on particle-based simulation datasets and mesh-based simulation datasets are recorded in Table 4 and Table 5, respectively. From the results, we can validate the superiority of our CARE in tackling the temporal environmental variation and making accurate long-term predictions.
Table 4: Results on particle-based physical simulations with the prediction lengths 1, 5, 10 and 20.  $v_{x}, v_{y}$  and  $v_{z}$  represent the velocity in the direction of each coordinate axis.
Prediction Length	+1	+5	+10	+20
Variable	vx	vy	vz	vx	vy	vz	vx	vy	vz	vx	vy	vz
Lennard-Jones Potential
MP-NODE	1.45	1.79	1.41	3.08	3.74	3.02	5.36	5.91	5.26	8.46	8.36	8.95
Ours	0.76	0.89	1.01	2.94	3.16	2.85	5.01	4.69	4.71	5.75	5.91	5.82
3-body Stillinger-Weber Potential
MP-NODE	9.95	9.82	9.87	12.47	12.41	12.49	16.05	16.14	16.10	17.06	17.15	17.09
Ours	4.21	4.29	4.18	9.74	9.79	9.71	13.65	13.71	13.57	15.30	15.39	15.35
G.2 Results with Different Prediction Lengths
To evaluate the performance of the proposed CARE in different settings, we vary the prediction length and compare the performance of different approaches. In particular, we show the results with the prediction length  $\{8,15\}$  on two particle-based simulation datasets in Table 6. The results with the prediction length  $\{15,30\}$  on two mesh-based simulation datasets are shown in Table 7. From the compared results, we can validate the superiority of our CARE in various settings.
Table 5: Results on mesh-based physical simulations over different prediction lengths 1, 10, 20 and 50.  $v_{x}$ ,  $v_{y}$  and  $p$  represent the velocity in different directions and the pressure field, respectively.
Prediction Length	+1	+10	+20	+50
Variable	vx	vy	p	vx	vy	p	vx	vy	p	vx	vy	p
CylinderFlow
MP-NODE	1.11	20.6	8.62	3.68	37.2	13.8	4.36	38.8	17.7	5.59	61.8	32.7
Ours	0.87	19.1	7.21	3.02	32.9	11.8	3.95	37.8	13.9	4.97	55.8	29.4
Airfoil
MP-NODE	4.41	4.44	0.38	4.85	4.76	0.49	5.89	5.72	1.23	7.45	7.97	1.78
Ours	3.51	4.11	0.19	3.86	3.75	0.34	4.16	4.12	0.45	6.74	6.82	0.81
Table 6: Results on particle-based physical simulations with the prediction lengths 8 and 15.  $v_{x}$ ,  $v_{y}$  and  $v_{z}$  represent the velocity in the direction of each coordinate axis.
Prediction Length	+8	+15
Variable	vx	vy	vz	vx	vy	vz
Lennard-Jones Potential
LSTM	9.44	9.40	9.57	12.68	12.75	12.61
GNS	8.86	8.92	8.85	11.99	11.84	12.08
STGCN	6.28	6.24	6.33	7.85	8.01	8.09
MeshGraphNet	6.47	6.42	6.55	7.99	8.08	8.14
CG-ODE	5.33	5.16	5.17	7.94	7.59	7.71
TIE	4.97	4.68	4.71	7.65	7.28	7.44
Ours	4.85	4.17	4.55	5.52	5.19	5.06
3-body Stillinger-Weber Potential
LSTM	22.89	22.93	22.90	26.74	26.79	26.78
GNS	22.36	22.38	22.31	26.04	26.05	26.02
STGCN	17.79	17.88	17.83	21.48	21.42	21.46
MeshGraphNet	17.92	17.84	17.95	21.86	21.82	21.84
CG-ODE	13.61	13.68	13.63	17.11	17.15	17.08
TIE	16.79	16.76	16.81	20.04	20.08	20.02
Ours	11.94	11.97	11.88	14.89	14.95	14.81
G.3 Visualization
Moreover, we show more visualization of our proposed CARE and the best baseline (i.e., TIE) compared with the ground truth. In particular, given that we have shown the time steps in  $\{1, 100, 200, 300, 400, 500\}$  in Figure 3 and Figure 4, now we select them in  $\{5, 150, 250, 350, 450\}$  for Lennard-Jones Potential and CylinderFlow, respectively. The compared results are shown in Figure 5 and Figure 6. We can observe that our CARE can generate more accurate trajectories compared with the baselines in most cases, which validates the superiority of our CARE again. For example, in the last row in Figure 5, our CARE can accurately recover the velocity distribution around the cylinder while the baseline fails.
H More Related Work
Graph Neural Networks. Graph Neural Networks (GNNs) have garnered significant success for their remarkable capabilities in graph representation learning [26, 62, 72, 33, 13], which is integral to a variety of downstream applications, including node classification [26], link prediction [71], and graph classification [62]. These approaches typically employ the message-passing mechanism, enabling the iterative updating of node representations with the aid of neighboring information. Recently, GNNs have successfully branched out into modeling interactive dynamics [42, 46, 45]. For
Table 7: Results on mesh-based physical simulations over different prediction lengths 15 and 30.  $v_{x}$ ,  $v_{y}$  and  $p$  represent the velocity in different directions and the pressure field, respectively.
Prediction Length	+15	+30
Variable	vx	vy	p	vx	vy	p
CylinderFlow
LSTM	8.25	47.43	17.62	11.09	59.86	33.24
GNS	8.16	47.95	17.54	11.36	60.49	33.72
STGCN	6.27	44.37	16.97	7.03	56.18	31.45
MeshGraphNet	5.13	42.05	14.65	6.54	52.96	28.63
CG-ODE	3.92	37.45	13.79	5.28	47.12	23.69
TIE	4.07	37.91	13.72	5.31	47.17	23.65
Ours	3.48	35.6	12.9	4.26	44.9	20.7
Airfoil
LSTM	8.75	8.84	3.96	12.86	18.74	4.78
GNS	8.14	8.02	3.51	11.52	11.44	4.04
STGCN	7.64	7.38	2.79	10.14	10.16	3.14
MeshGraphNet	6.05	6.19	1.27	7.11	7.09	1.43
CG-ODE	5.68	5.53	0.81	6.82	6.88	1.21
TIE	5.41	5.33	0.79	6.64	6.71	1.19
Ours	4.08	4.02	0.41	4.95	5.11	0.62
Figure 5: More visualization of velocity in CylinderFlow dataset with varying time steps in  $\{5,150,250,350,450\}$ .
instance, MeshGraphNet [42] employs a message passing neural network to facilitate the modeling of interactions between objectives, thereby outputting the next-time predictions. However, an inherent drawback lies in the inability of discrete GNNs to encapsulate the continuous nature of system
Figure 6: More visualization of Lennard-Jones Potential with varying time steps in  $\{5,150,250,350,$  450\}.
dynamics. To relieve this limitation, we present a novel graph-based ODE system named CARE for the modeling of interacting dynamics, which enriches the capabilities of making long-term predictions under potential environmental variation.
Graph-based ODE. Neural ODEs have been integrated into GNNs, resulting in the development of Graph ODEs that are applicable to both static and dynamic graphs. Graph ODEs on static graphs [60, 70] primarily aim to mitigate overfitting by formalizing derivatives using both initial and immediate node representations. Meanwhile, Graph ODEs on dynamic graphs are utilized for tasks such as traffic flow forecasting [9] and social analysis [19], demonstrating effective performance on irregularly sampled partial observation data. For example, STGODE [9] employs tensor computation to conduct continuous message passing, which facilitates accurate long-term predictions by overcoming the network depth limitations. Despite these advancements, existing works fall short of addressing the temporal environmental variation in interacting dynamics. To fill this gap, we propose a novel approach CARE to handle this problem by injecting a context variable in the Graph ODE system.
I Potential Negative Impacts
To the best of our knowledge, we have not found any negative impact of our work.