PURE: Prompt Evolution with Graph ODE for Out-of-distribution Fluid Dynamics Modeling
Authors: Hao Wu
Changhu Wang
Fan Xu
Jinbao Xue
Chong Chen
Xian-Sheng Hua
Xiao Luo
Venue: NeurIPS 2024 poster (2024)
Keywords: AI for Science
Primary area: machine_learning_for_physical_sciences

Abstract: This work studies the problem of out-of-distribution fluid dynamics modeling. Previous works usually design effective neural operators to learn from mesh-based data structures. However, in real-world applications, they would suffer from distribution shifts from the variance of system parameters and 
temporal evolution of the dynamical system. In this paper, we propose a novel approach named \underline{P}rompt Evol\underline{u}tion with G\underline{r}aph OD\underline{E} (\method{}) for out-of-distribution fluid dynamics modeling. The core of our \method{} is to learn time-evolving prompts using a graph ODE to adapt spatio-temporal forecasting models to different scenarios. In particular, our \method{} first learns from historical observations and system parameters in the frequency domain to explore multi-view context information, which could effectively initialize prompt embeddings. More importantly, we incorporate the interpolation of observation sequences into a graph ODE, which can capture the temporal evolution of prompt embeddings for model adaptation. These time-evolving prompt embeddings are then incorporated into basic forecasting models to overcome temporal distribution shifts. We also minimize the mutual information between prompt embeddings and observation embeddings to enhance the robustness of our model to different distributions. Extensive experiments on various benchmark datasets validate the superiority of the proposed \method{} in comparison to various baselines.

PURE: Prompt Evolution with Graph ODE for Out-of-distribution Fluid Dynamics Modeling
Hao Wu Changhu Wang", Fan Xu', Jinbao Xue",
Chong Chen', Xian-Sheng Hua", Xiao Luo
Unniversity of Science and Technology of China, University of California, Los Angeles, Tencent, *Terminus Group
vuhao2022@mail. ustc. edw..eduucoo wangch1564g ucla. edu, markxu@mail www.ustt.eee. markxu@mail. uste. edu. en, jinbaoxueeteenentt com, chenchong. mmenchong.cz@gmail. wwtgmail.e..oooooom.m com, huaxiansheng@gaail com, xiaoluoθcs www.uclllla... edu
Abstract
This work studies the problem of onttof-ddistibutiinn fluid dynamics modeling. Pre- vious works usually design effective neural operators to learn from mesh-bases data structures. However, in real-world applications, they would suffer from distribution shifts from the variance of system parameters and temporal evolution of the dynam- ical system. In this paper, we propose novel approach named Prompt Evolution with Graph ODE (PURE) for iit-of--ditrbbution fluid dynamics modeling. The core of our PURE learn time-evolving prompts using a graph ODE adapt spatio-temporal forecasting models to different scenarios. In particular, our PURE first learns from historical observations and system parameters in the frequency do- main to explore multi-view context information, which could effectively initialize prompt embeddings. More importantly, we incorporate the interpolation of obser- vation sequences into graph ODE, which can capture the temporal evolution of prompt embeddings for model adaptation. These time-evolving prompt embeddings are then incorporated into basic forecasting models to overcome temporal distribu- tion shifts. We also minimize the mutual information between prompt embeddings and observation embeddings to enhance the robustness of our model to different distributions. Extensive experiments on various benchmark datasets validate the superiority of the proposed PURE in comparison to various baselines. Our codes are available at https::/ thub. wtw.b..ooo/eeeeeeeeeeeeeeeeeeeee wtt.://eaaeaaaaannereeeeeeeeeeee
Introduction
Fluid dynamics [44, 89] is a critical area in the field of mechanics and computational fluid dynamics has emerged a powerful tool to understanding fluid flow [32, 58, 67, 45]. Recently, various machine learning approaches have been widely adopted to solve the problem in data-driven manner [59, 46, 66, 65, 18, 7, 81], which can achieve high efficiency in comparison previous traditional numerical solvers. Moreover, they enjoy strong applicability when the underlying rules not explicit, such real-world weather forecasting [4] and disease transmission [68].
In literature, existing data-driven fluid dynamics modeling approaches can be roughly divided into grid-based approaches [12, 17] and geometry-basse approaches [59, 66, 65, 20]. Griddbased approaches construct regular meshes and then utilize neural operators to explore spatio-tempore relationships. In contrast, geometry-basee based approaches focus irregular point clouds and then utilize graph neural networks (GNNs) [30, 70] to learn from the interaction between mesh points.
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Despite their great success, existing approaches [88, 25] generally assume that training and test data share the same data distribution [59, 66, 65, 47], which could be not the case in real-world applications. In particular, there are two typical types of distribution shifts in dynamical systems, i.e. parameter-based shifts, and temporal distribution shifts. Firstly, different dynamical systems could involve different parameters in underlying rules, such as coefficients in PDEs and pressures in fluid systems [3, 65]. Secondly, during long-term auto-regressive forecasting, the input data distribution could vary hugely during the temporal evolution [91]. As in previous works [22, 33], machine learning approaches usually suffer from huge performance degradation when it comes to distribution shifts. Therefore, in this paper, we focus on the problem of -uttof-diitribution fluid dynamics modeling to enhance the performance under potential distribution shifts.
In this paper, we propose a new approach named Prompt Evolution with Graph ODE (PURE) for out-of-distribution fluid dynamics modeling. The high-level idea of our proposed PURE is to adapt well-trained forecasting approaches to different out-of-distributie evolving prompts [51, 84, 9]. To begin, we extract multi-view context signals from both historical observations and system parameters in the frequency domain using the attention mechanism, which can effectively initialize prompt embeddings under parameter-based shifts. More importantly, to capture temporal distribution shifts, we combine the interpolation of observation sequences into graph ODE framework, which can utilize the interaction between prompt embeddings and observation embeddings for high-quality time-evolving prompt embeddings. Then, we concatenate our prompt embeddings and observation embeddings for model adaptation and enhance the robustness of our PURE to distribution variance by minimizing their mutual information using adversarial learning. Extensive experiments on a range of fluid dynamics datasets validate the superiority of the proposed PURE in comparison to various of-e-of-thhearr approaches.
scenarios by learning time-
In summary, the contribution of our paper can be summarized as follows: (1) Problem Connection. We are the first to connect prompt learning with dynamical system modeling to solve the issue of out-of-distribution shifts. (2) Novel Methodology. Our PURE first learns from historical observations and system parameters to initialize prompt embeddings and then adopts a graph ODE with the interpolation of observation sequences to capture their continuous evolution for model adaptation under -iit--ibiittttttioonn shifts. (3) Superior Performance. Comprchensive experiments validate the effectiveness of our PURE in different challenging settings.
2 Problem Setup
Given fluid dynamical system, we have N sensors within the domain Ω, with their locations denoted as æı,.... ,,,, where Φ; E Rd: The observations at time step are represented as "!. ,8xx where of E Rdd and d, indicates the number of observation channels. Dynamical systems are gov- erned by underlying system rules, such as PDEs with coefficient& &. Variations in system parameters may lead to different environments, potentially resulting in distribution shifts [54, 80, 6, 27J. In our study, we are provided with historical observation sequences {s! 1 } and physical parame- ters & (e.g., coefficients in the PDEs). Our goal is to predict the future observations of each sensor s To+1-To+T In dynamical systems, the
problem examines model performance when predicting under unseen parameter distributions or environments. Let r = [[$].- ... syy. these systems evolve according to du
= F(u. E), where represents the observations and denotes the sys- tem parameters. When & ~ P(E). the state trajectory u 1:To follows the distribution P(u 1:To (g). As- sume we learn a learned mapping function ff from u The to u To i.c.,, Too1I = f(u and there could be different distributions across training and test datasets, i.c., P'rmii (E) o (E))(E). which results in Purin
de
(To) * Pust (u TTo). Moreover, when conducting rollout prediction, we required to feed the output back to the model, i.e., uTotace with P(w TTo (E) # P(u Tatort To-T.tam -1 [ T Tstort).
=f f(u (atarr -T:: To:Tatort
3 The Proposed PURE
3.1 Motivation and Framework Overview
This paper addresses the challenge of -utto--distributtinn fluid system modeling, which is complicated by parameter-ber based and temporal distribution shifts. Specifically, our function f(-) can suffer from

---

Physical patamerers
MLP
Muhi-head mell-head Self- Self-attenienn
Physics Space
Sensoe valtues
Artention block
blockk
Mini hieee Sett Coneenttooon
Valuex
Valtae
Keys
Initaahiing the
proeppe emboldding
Query
Query locations
Queey
Linest
INN
KNN
RNN
Tinne-vvolving Prompt Embeddings
Observation Embeddings LMi
Values
MHSA
MHSA
MHSA
Versatile
Keys
Keys
Keys
Backbone
Basic Forecasting Model
St
LAIS
Interpolation
Physics space
Figure 1: Overview of the PURE framework.
scrious distribution shift result from different and Tatart, i.c., P(uinput E, Tstart). To reduce the impact of distribution shift, we aim to learn invariant observation embeddings μ to different environments, i.e., & and T Tatars for better generalization and utilize prompt embeddings 2' indicate the current environment for final prediction. In formulation, we have:
z' I μ² Noutput = "(In" 2²1).
(1)
The first term ensures the invariance of observation embeddings by decoupling observation embed- dings and prompt embeddings. The second term aims to combine both two embeddings to generate the future predictions. Therefore, we propose novel approach named PURE
fr' BasicMMode((unpud)
ContextMining (uipat)) z' GraphODE(2" .t). (2)
where a basic model is adopted to generate observation, and we adopt context mining and graph ODE to learn time-varying prompt embeddings. Given a basic forecasting model (Eqn. 2), PURE contains three key modules: (1) Multi-view Context Exploration, which explores spatio-temporal data using both the attention mechanism and the frequency domain initialize prompt embeddings (Eqn. 2). (2) Time-evolving Prompt Learning, which incorporates the interpolation of observation sequences into a graph ODE learn the evolution of prompt embeddings (Eqn. 2). (3) Model Adaptation with Prompt Embeddings, which leverages the time-evolving prompts to mitigate the temporal distribution shifts fluid dynamics models (Eqn. 1). More details are in Figure
3.2 Multi-view Context Exploration from Spatio-temporal Data
The main idea of our PURE is to utilize prompt learning to solve the issue of duitrf-ddistibution shifts [54, 80, 6, 27] in fluid dynamical systems. Prompt learning [51. 84, 9] is effective man- ner to adapt language models to various downstream tasks. In our scenarios, we aim to learn from both historical spatio-temporal information and system parameters initialize our prompt embed- dings, which can effectively solve the parameter-base distribution shifts. Here, we first follow the attention mechanism [69, 86, 10, 50) reconstruct the field value and then adopt the Fourier neural operator [45] to integrate multi-view context information.
In particular, given each location, map each location æ; and each initial observation sf into position embedding and an observation embedding qi using two feed-fon fordoorowddingg networks (FFNs) o P.)()) and (.). and then aggregate p, and qi using the Hadamard product followed by stacking self-attention blocks for representation learning. formulation,
e, p, qi E 8 8A,00 (E")

---

where @ denotes the Hadamard product. E is constructed by stacking {e, and & (.((() is the self-attention block the layer Afterward, we adopt the attention mechanism [69] to retrieve the representations for each query position By as:
[WQ (2 [W E4.
u² = softmax
W WVEE,
Vd
where W is learnable weight matrix for feature transformation and d is the hidden dimension. By retrieving the representations each regular grid, we generate the 3D representation tensor U. Each tensor would be concatenated with the parameter embedding u² = (E) for multi-view information integration, which results in the final tensor U. Then, we utilize the frequency domain for representation enhancement to generate prompt tensor H. Here, we first transfer the tensor into the frequency domain using a Fast Fourier Transformer (FFT) operator [45] and then adopt an FFN for feature transformation. Lastly, an inverse Fast Fourier Transformer (iFFT) operator is adopted to convert the features back to the spatial domain. Formally,
= (FF((F)EN()))()))
(5)
where F.T(.) and (.FT(.) denote the FFT and iFFT operators, respectively. Since the input of spatio-temporal models would be irregular, we flatten the tensor H, and retrieve prompts for each sensor from the prompt tensor using:
[ PF (I,)
IW flatten(H) (H)) Vd
2 softmax
W flatten(H)
(6)
where flattem(-) ) is a flattening operator to transform 3D tensors to 2D matrices. Through the frame reconstruction, can extract important spatio-temporal signals from the frequency domain, which is effective in initializing the prompt embedding for each sensor.
3.3 Time-evolving Prompt Learning with Graph ODE
To capture temporal distribution shifts within one system, static prompt embeddings [51] from context exploration are far from satisfactory. Our solution to obtain continuous time-evolving prompts at any timestamp. To achieve this, we view the output of Eqn. 6 as the initial prompt embeddings and then incorporate the attention mechanism into continuous graph ODE, which combines the interpolations of observations with the graph structure to learn the evolution of prompt embeddings.
In particular, given the initial prompt embeddings, we introduce two functions v.(() and r.)(()) for relation mining and feature aggregation, (-) calculates the interaction between the centroid node and each of its neighboring nodes and v-a(-) aggregates all the neighborhood interactions to determine the evolution. Therefore, a graph ODE can be formulated by the following formulation:
dzf dt
vi( <
(+r 21D),
JES(())
where (i) collects the sensors from the neighbours of at timestamp f. However, Eqn. 7 neglects observations themselves during evolution, which are directly related to temporal distribution shifts in dynamical systems. Thus, could generate suboptimal prompt embeddings. To tackle the issue, we conduct the interpolations of observation sequence 8, 1:To which results in of at any timestamp. Then, incorporate them into our graph ODE using the attention mechanism by rewriting Eqn. into:
dz dt
[W92]]
W s5)
vial Σ
softmax
dh (5-[2], =f)))
(8)
Vd
(ES
where Wº and W are two matrices to generate the query and key, respectively. Here, we utilize the prompt embeddings and interpolated observations serve the query and the key. In this way, we effectively model their interaction adjust the derivative in the graph ODE, which can help generate proper prompt embeddings for model adaptation under temporal distribution shifts.

---

3.4 Model Adaptation with Prompt Embeddings
Finally, we incorporate our prompt embedding into our basic spatio-temporal forecasting model, and then introduce the optimization objective for the end-to-end training.
Basic Forecasting Model. Our time-evolving prompt embeddings be easily incorporated into any spatio-temporal forecasting model. To make the best of our efficacy, we utilize a simple yet powerful basic model our default model and also explore the performance of our PURE more existing forecasting models. The input of our model is the observations of sensors from between the interval [1, To] and output the predictions in [To + 1, To Ti, i.e. 51:T0 + S where S is stacked by 8; In particular, our basic model first generates the embeddings of different observations, and then reconstructs the irregular observations into frames on grids using the reconstruction modules in Sec. 3.2. More importantly, we introduce two parallel modules, i.e., Fourier neural operator [45] and a ViT-based convolution [19] and extract complementary feature maps [81], which would be fused to generate the predicted frames in the future. More details of our basis forecasting model can be found in Appendix B. Additionally, we also use other basic models.
Adaptation and Optimization. Note that we generate observation embeddings in our basic module, i.e. 1 = (f)) To adapt our model under distribution shifts, we concatenate the observation embeddings and prompt embeddings into updated embeddings j as follows:
j [M 2
which will be fed into the subsequent modules in the basic model. To optimize the whole framework. we first minimize the mean squared error (MSE) between the predicted observation and the ground truth follows:
To+T Σ (=
LMSE =
11S - -
(10)
where St denotes our predicted observation for every node and S denotes the ground truth observa- tions. Moreover, to enhance the invariance of our model to different scenarios, we turn to invariant learning [72, 42, 77] to decouple various prompt embeddings and observation emheddings, which promotes the observation embeddings to be less sensible to different distributions. To achieve this, we minimize the mutual information between observation embeddings and observation embeddings, i.e. [u zt). In our work, we adopt a Jensen- Shannon mutual information estimator [40, 53] (--))) where y denotes the parameters to estimate their mutual information. Then, we collect all the corre- sponding pairs of (u', 2 using P and all the possible pairs of (Hr) 25) using N. The adversarial learning objective can be written as:
LMI = EmI=maryys
1 P
Σ
Σ
sp( 21)),, (11)
(U(P)))E
in which sp(x)= = log(1 + e²)) represents the softplus function. In summary, the overall objective
can be written as:
L = LMSE + XLMI,
(12)
where A is coefficient to balance two loss objectives. The algorithm is summarized in Appendix D.
3.5 Theoretical Analysis
In this part, we provide a theoretical analysis to demonstrate how PURE works. Our focus is primarily on theoretically showing the necessity of incorporating the observations themselves during evolution. For simplicity of analysis, we assume that Eqn. 7 can be rewritten as:
dzf di (1)
Σ JEST(()
(Miz) + M2z}) = M12 + #(S(11)
M2z JES*()
Σ
(13)
where #(- calucates the size of the set. For the sake of simplicity in the proof. we assume that 2f is one-dimensional and consider only the ODE above for (not the entire system of ODEs). Then, Equ. 13 can be rewritten
d2f
Σ (M + M223) +
(14)
dt
#(S( (ES
5

---

where b(t) is function. To characterize temporal distribution shifts, we assume that portion of the corresponding true zj has a constant shift. With the potential environmental change, Eqn. 13 can be rewritten as:
dæf
E (M12) + M22j) M1 + (t), JES'()
(15)
#()(1))
where lb(t) - (t) 2 co. suggesting the constant shift co-
Theorem 3.1. Given the following ODEs in R.
2i + b(t), ì = Myy + (t), v(0) =
z(0) zo:
(16)
where 16(t) - (t)) 2 cu, there exists positive constant such that
[x((() v()
ce - 1), for all >> O.
(17)
The proof of Theorem 3.1 can be found in Appendix A. Theorem 3.1 suggests that even with the simplest one-dimensional linear ODE, significant differences in the solutions will arise if temporal distribution shifts are neglected. Next, we will focus on how Eqn. addresses this issue. In this case, assume that
t>, (1==), = M1 + M22
(18)
Then, Eqn. 7 can be written as:
dzf = vaa dt
Σ
(Miz) + Mo23))
da M12t + #(s( jES((1)
Σ
M22l
t (M + b(t)),
(ES
(19)
where
b(t)
Σ M2zj.
(20)
#(S jEST())
Similarly, Eqn. can be written as:
dzf do (M1x + +(()),
(21)
dl
where
[W [W sy]
8'(t) = Σ
softmax
M22j.
(22)
Vd
(ES
For simplicity of notation, omit the superscript Write F(z,1) dh (Mi b(() and G(z,1)
va (M2²² + b/ (t)). Then, we have the following theorem with the proof Appendix
Theorem 3.2. Assume that the attention mechanism satisfies that [B(()) b(() S e, for all 1 0. and the function &a is L-Lisschitz. Given the following ODEs in RR
t dh (M + b(t)) F(x,t), z(0) 20.
(23)
ÿ = a M (1)) = Gly,), y(0) = To,
there exists two constants 02 and c3 such that
[x()) vt) coy(e - 1), for all >> 0.
(24)
Thoerem 3.2 shows that as long as the attention mechanism is sufficiently good, we can approximate the true ODE with arbitrary precision using Eqn. 8, even in the presence of environmental change.

---

Table 1: We compare our study's performance with 10 baselines. We magnify the MSE of 3D- Reaction-Diffusion by 100 times. Green Yellow Red mean best, second, worst MSE.
BENCHMARKS
MODEL
PROMETHEUS
NAVIER STOKES
SPHERICAL-S SWE
3D REACTION DIFF
ERAS
OOD
OOD
weo
w/ OOD
wo OOD
OOD
vofo OOD
OOD
U-NET [64]
0.0931
0.1067
0.1982
0.2243
0.0083
0.0087
0.0148
0.0183
0.0843
0.0932
RESNET [21]
0.0674
0.0696
0.1823
0.2301
0.0081
0.0192
0.0151
0.0186
0.0921
0.0977
VIT [10]
0.0632
0.0691
0.2342
0.2621
0.0065
0.0072
0.0157
0.0192
0.0762
0.0786
SWINT [49]
0.0652
0.0729
0.2248
0.2554
0.0062
0.0068
0.0155
0.0190
0.0782
0.0832
FNO (45]
0.0447
0.0506
0556
0.1712
0.0038
0.0045
0.0132
0.0179
0.7233
0.9821
UNO [1]
0.0532
0.0643
0.6764
0.1984
0.0034
0.0041
0.0121
0.0164
0.6652
0.7621
CNO [63]
0.0542
0.0655
0.1473
0.1522
0.0037
0.0038
0.0145
0.0182
0.5243
0.7821
NMO [82]
0.0397
0.1021
0.1032
0.0026
0.0031
0.0129
0.0168
0.0432
0.0563
CGODE [26]
0.0761
0. 0.0843
0.2035
0.2243
0.0873
0.0987
0.8371
0.9261
0.8721
0.9872
DGODE [80]
0.0344
0.0359
0.0805
0.0925
0.0022
0.0028
0.0122
0.0156
0.0543
0.0635
OURS PURB
0.0323
0.0328
0.0752
0.0763 26.07%
0.0022 0.00%
0.0024
0.0119
0.0127
0.0398 7.87%
0.0401
PROMOTION
6.10%
8.63%
6.58%
16.67%
1.65%
22.56%
28.77%
Table 2: This table shows the performance of the PURE framework across different benchmarks.
BENCHMARKS
MODEL
PROMETHEUS
MAVIER-STOKES
SPHERICAL-SWE
3D REACTION-DIFF
ERA5
ORI
+PURE
ORI
+PURE
ORI
+PURE
ORI
+PURE
ORI
+PURE
RESNET [21]
0,0674
0.0542
0,1823
0.1492
0.0081
0.0067
0.0151
0.0141
0.0921
0.0896
NMO [10]
0.0397
0.0281
0.1021
0.0876
0.0026
0.0012
0,0129
0.0123
0.0432
0.0389
DGODE [49]
0.0344
0.0201
0.0805
0.0792
0.0022
0.0020
0.0122
0.0110
0.0543
0.0462
4
Experiment
4.1 Experimental Settings
Benchmarks. We study Benchmarks from three domains, as shown in Table Computational Fluid Dynamics. We use Prometheus [80] and follow the original setup for environment segmentation. Real-world Data. We employ the ERAS [23], using different combinations of variables as the environment, In detail, we use ERA5 data with variables such as surface pressure (Sp), sea surface temperature (SST), sea surface height (SSH), and two-meter temperature (T2m) to predict temperature. b Partial Differential Equations. The 2D Navier-Stokes equations [45] describe fluid motion, with the primary variable being the viscosity coefficient v, which quantifies internal friction in the fluid, simulating vorticity values under ten different viscosity coefficients. The spherical shallow water equations [14] simulate large-scale atmospheric and oceanic fluid motion on Earth's 's surface, also with viscosity coefficient as the main variable, involving tangential vorticity (w) and fluid thickness (h) on a spherical surface. The 3D -dfffuionn equations describe the diffusion and reaction of chemicals in space [62], with the primary variable being the diffusion coefficient D, representing the rate of chemical diffusion in space, including u, v velocity components. More details see Appendix
Baselines. We select representative models from three domains as baselines. >> Visual Backbone Networks. We include ResNet [21], U-Net [64], Vision Transformer( ViT) [10], and Swin Trans- (oINe)(SWINT) [49]. > Neural Operator Architectures. We cover FNO [45], UNO [1], CNO [63], and NMO [82]. & Graph-ODE Architectures. We feature CG-ODE [26], and DGODE [80].
Tasks. We evaluate model performance for various prediction tasks through the following scenarios and use MSE as metrics. The specific tasks are as follows:
>> Generalization Experiments:
Generalization: We train the model In- Domain environment and test it in Adaptation environment to verify its generalization ability. Spatial Generalization & Temporal Generalization: In the Prometheus, we train the model at 75% sparsity and test it at {5%, 25%, 50%, 75%) sparsity. The experiment evaluates performance with equal input and output lengths ( (In,) and with output 10 times the input length (Out,).
Zero-shot Experiments. Specifically, we follow the setup from [45] and conduct two experiments In the Prometheus, we train the model In-Domain environments by, bz,..., b2n and evaluate its generalization ability in new environments 611, b12, using MSE the evaluation metric. In the
7

---

Table 3: Comparison Spatial Temporal Generalization in the Prometheus benchmark.
SPARSITY
TEST
$ts
5%
Arp
25%
Ary
50%
Sts
75%
TRAIN 4
IN-T
OUT-T
IN-T
OUT-T
IN-T
OUT-T
IN-T
OUT-T
U-NET PURE
0.1847
0.2103
0.2345
0.2877
0,2654
0.3018
0.2273
0.3391
0.1622
0.1854
0,2079
0.2581
0.2365
0.2710
0.1998
0,3024
6% 75%
FNO PURE
0.0659
0.0872
0.0921
0.1232
0.1109
0.1821
0.2109
0.2455
0.0504
0.0654
0686
0.0946
0.0805
0.1417
0.1582
0.1883
Temperature
75%
Smoke Field
Spurse Iuput
Ground Truth
OUrS-PURE Error
DGODE Error
FNO Error
U-Net Error
Figure 2: The top row shows the sparse input data used for predictions. The second row displays the true data for both fields. Red boxes highlight areas of significant error.
Navier-Stokes equations, we train the model on a 64 x 64 x 20 dataset and evaluate it on higher resolution 512 x 512 x 20 dataset, focusing on the fluid dynamics details in the last five time steps and the handling of complex flow patterns and boundary layers.
4.2 Generalization Experiment Results
In this section, we focus on the issue of generalization. Based on our experimental findings, we make the following observations. Out-of-Distribution Generalization. The results as shown in Table 1, On the Prometheus dataset, PURE outperforms all benchmark models with an MSE of 0.0323 in-distribution and 0.0328 OOD. improves over the second-best model, DGODE (MSE 0.0344 in-distribution, 0.0359 OOD), by 6.10% and 8. 8.63%, respectively. On the Navier-Stokes dataset, PURE achieves the best performance with an MSE of 0.0752 in-distribution and 0.0763 OOD, improving by 6.58% and 26.07% over the second-best model. On the Spherical-SWF dataset, PURE has an MSE of 0.0022 both in-distribution and OOD, which 41.46% better than the second- best model. Additionally. in Table 2, the performance of various benchmark models significantly improves when using PURE, in summary, the PURE framework performs excellently in handling OOD fluid dynamics modeling.
Spatial & Temporal Generalization. Table 3 shows that PURE excels in the Prometheus benchmark, notably reducing errors with sparse data. For instance, in the 75% sparsity test, U-Net's MSE drops from 0.2273 to 0. 1999,, and FNO from 0.2109 to 0.1582. Figure highlights PURE's lower errors in temperature and smoke fields compared to DGODE, FNO, and U-Net, especially in red- red-boxed areas, showcasing its advantage capturing complex dynamics. Additionally, PURE performs consistently across different prediction lengths; for example, U-Net's MSE decreases from 0.1847 to 0.1622 for in-time prediction (In-t) and from 0.2103 to 0854 for out-of-time prediction (Out-t). Overall, PURE excels with sparse and out-of-distribun demonstrating strong spatial and temporal generalization.
data and enhances performance across prediction lengths,
Visualization and Analysis. Figure compares the performance of different methods in fluid dy- namics modeling, including the Prometheus dataset, Savier-Stokes equations, and the 3D Reaction- Diffusion Equation. In the Prometheus dataset, using PURE significantly reduces DGODE's pre- diction error, especially in complex dynamic regions. For the Navier-Stoker Stokes and Spherical Shallow Water equations, FNO and NMO models combined with PURE excel in capturing complex flow features. In the 3D Reaction-Diffusion Equation, DGODE with PURE significantly reduces predic- tion errors. Overall, PURE greatly enhances the prediction accuracy of models in fluid dynamics, allowing for better capture of complex dynamic evolution.
4.3 Zero-shot Super-resolution and Environment Generalization
As shown in Figure 4, the PURE framework performs excellently in zero-shot super-resolution and environmental generalization experiments. In the Prometheus benchmark, the FNO model using

---

Ground Truch
Prediction
DGODE PURE Ereur
Ground Touund-Truthh FNO FURE Namier-tokerr
FNO
Ground-Truth NMO PURE
DGODE Error
Spheriaal thallow equations
Ground Truch
Prediction
DGODE Error
DGDDR
Ground-Truth
Promothenu
313 Reation ReadiionDiifiaiii Diffiasion equations
Figure 3: The Figure compares the performance of various methods in fluid dynamics modeling, including Prometheus, Navier-Stoke Stokes equations, and 3D -ifcion-diffusion equations. Models with PURE significantly reduce prediction errors in fluid dynamics, capturing complex dynamic evolutions.
64x122
64k1128
128v256
256x512
Last 5 time stops
Tranhs
Ground-truth MSB
Prediction 0.4439955
Prediction
Prediction
0.083678]
0,0471683
Outa
FNO
"Top diplays
ENO+PURE
Ground Troth
PURE Prediction
Prediction
predictions and Bottom cow the FNO predicaoos. snined oa x 64 x 20 daisect and evaltated on 312 512 deusee.
Trining envinomment [b1, b2, bā,.. b90). and [b]l] b12)
Figure 4: Left. shot super-resolution and environment generalization experiments on Prometheus. Right. Zero-shot super-resolution experiments on the Navier-Stokes equations.
PURE significantly reduces prediction errors at different resolutions, with an MSE of 0.0471655 256 512 resolution. For the Navier-Stoker equations. the FNO model combined with PURE significantly reduces prediction errоrs on high-resolution datasets and performs better in handling complex flow patterns and boundary layers, especially in capturing details the last five time steps. Overall, PURE significantly improves model prediction accuracy and generalization ability in zero- shot super-resolution and environmental generalization tasks.
4.4 Qualitative Analysis & Ablation Study
this section, we evaluate the effectiveness of the PURE method and the importance of its compo- nents through qualitative analysis and ablation studies.
Qualitative Analysis. The Figure uses t-SNE to perform clustering analysis on FNO prediction results. (a) represents the ground truth, (b) shows the predictions of the original FNO, and (c) shows the predictions of FNO combined with PURE. is evident that the FNO combined with PURE is closer to the labels in clustering effect, with more tightly distributed data point cluster. This demonstrates that PURE significantly improves the prediction accuracy the FNO model.
Ablation Study. To evaluate the contribution and importance of each component in the proposed PURE, design ablation experiments based the default backbone model in this paper, and use Relative L2 error as metric. Our model variants are as follows: (1) PURE w/o Graph ODE, we
Table 4: Ablation Studies on S-SWE.
VARIANTS
S-SWE
PURE w/o GRAPH ODE
0.1882
PURE w/O INTERPOLATION
0.1696
(6)
PURE W/O MI
0.1588
Figure t-SNE clustering. (a) Ground truth, (b) FNO predictions, (c) FNO +PURE predictions.
PURE w/O FFT PURE
0.1602
0.1357

---

remove the Graph ODE module and use static prompt embeddings. (2) PURE w/o Interpolation. we remove interpolation and use only Eqn. 7. (3) PURE w/o MI, we remove the mutual information minimization. (4) PURE w/o FFT, we remove the frequency domain enhancement (FFT). Table 4 shows the results of our ablation study. Removing Graph ODE, interpolation, mutual information minimization, and FFT results in Relative 1.2 errors of 0. 081882, 0.1696, 0.1588, and 0.1602, respec- tively. The complete PURE method has an error of 0.1357. The results of the ablation experiments show that removing any component results in decrease in predictive performance, further proving the critical role of these components in the PURE method. More results in Appendix H.
5 Conclusion
In this paper, we study practical problem of cittr--distribttion fluid dynamics modeling and propose a novel approach named PURE for this problem. The high-level idea of our PURE is to learn time-evolving prompts using graph ODEs, which can effectively adapt spatio-temporal forecasting models to different scenarios. Our PURE first initializes prompt embeddings by exploring multi-view context information from spatio-temporal data and system parameters. Then, PURE incorporates the interpolation observation sequences into the graph ODE, which helps capture the temporal evolution of prompt embeddings to mitigate temporal distribution shifts. In future works, we will extend our PURE to more real-world scenarios such as rigid dynamics modeling and traffic flow forecasting.
References
[1] Md Ashiqur Rahman, Zachary E Ross, and Kamyar Azizzadeneshelli U-no: U-shaped neural operators. arXiv e-prints, pages arXiv-2204, 2022.
[2] Ainesh Bakshi, Allen Liu, Ankur Moitra, and Morris Yau. Tensor decompositions meet control theory: learning general mixtures of linear dynamical systems. In International Conference on Machine Learning, pages 1549-1563. PMLR, 2023.
[3] Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, and Christian Wolf. Cophy: Counterfactual learning of physical dynamics. arXiv preprint arXiv:1909.12000, 2019.
[4] Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian, Accurate medium-range global weather forecasting with 3d neural networks. Nature, 619(7970)5533-533, 2023.
[5] Michael S Branicky. Continuity of ode solutions. Applied Mathematics Letters, 7(5):57-60, 1994.
[6] Lanlan Chen, Kai Wu, Jian Lou, and Jing Liu. Signed graph neural ordinary differential equation for modeling continuous-time dynamics. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 8292-8301, 2024.
[7] Ailin Deng and Bryan Hooi. Graph neural network-based anomaly detection in multivariate time series. AAAI, 2021.
[8] Xun Deng, Wenjie Wang, Fuli Feng, Hanwang Zhang, Xiangnan He, and Yong Liao. Counter- factual active learning for out-of-distribution generalization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11362 1137771377, 2023.
[9] Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Hai-Tar Zheng, and Maosong Sun. Openprompt: An open- framework for prompt-learnngg arXiv preprint arXiv:2111.01999. 2021.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlshy. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
10

---

[11] Huawei Fan, Junjie Jiang, Chun Zhang, Xingang Wang, and Ying-Cheng Lai. Long-term prediction of chaotic systems with machine learning. Physical Review Research, 2(1):012080, 2020.
[12] Zhiwei Fang. A high-efficient hybrid physics-ifformed neural networks based on convolutional neural network. IEEE Transactions on Neural Networks and Learning Systems, 33(10):5514- 5526, 2021.
[13] Stathi Fotiadis. Mario Lino Valencia, Shunlong Hu, Stef Garasto, Chris D Cantwell, and Anil Anthony Bharath. Disentangled generative models robust prediction of system dynamics. In International Conference on Machine Learning, pages 10222-10248. PMLR, 2023.
[14] Joseph Galewsky, Richard K Scott, and Lorenzo M Polvani. An initial- problem for testing numerical models of the global shallow- equations. Tellus A: Dynamic Meteorology and Oceanography, 56(5):429 440, 2004.
[15] Saurabh Garg, Sivaraman Balakrishnan, and Zachary Lipton. Domain adaptation under open set label shift. Advances in Neural Information Processing Systems, 1--22531-22546, 2022.
[16] Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. Ppt: Pre-trained prompt tuning for few- shot learning. arXiv preprint arXiv:2109.0453: 2021.
[17] Xiaoxiao Guo, Wei Li, and Francesco Iorio. Convolutional neural networks for steady flow approximation. In KDD, pages 481-490, 2016.
[18] Jiaqi Han, Wenbing Huang, Hengbo Ma, Jiachen Li, Joshua B Tenenbaum, and Chuang Gan. Learning physical dynamics with subequivariant graph neural networks. arXiv preprint arXiv:2210.06876, 2022.
[19] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang. An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence, 45(1):87-110, 2022.
[20] Xu Han, Han Gao, Tobias Pffaf, Jian-Xun Wang, and Li-Ping Liu. Predicting physics in mesh- reduced space with temporal attention. arXiv preprint arXiv:2201.09115 2022.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 770-778, 2016.
[22] Dan Hendrycks. Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli. Mike Guo, et al. The many faces of robustness: critical analysis of International Conference on Computer Vision, pages 8340 8349, 2021.
generalization. In Proceedings of the IEEE/CVF
[23] Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz- Sabater, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, et al. The era5 global reanalysis. Quarterly Journal of the Royal Meteorological Society, 146(730): 1999-99999,,99 2020.
[24] Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision lasgua-anguuage models. arXiv preprint ar arXiv:2204.0334 2022.
[25] Xinquan Huang, Wenlei Shi, Qi Meng, Yue Wang, Xiaotian Gao, Jia Zhang, and Tie- Van Liu. Neuralstagger: accelerating physics-constrate neural pdc solver with spatial- -empaal-emmpoal decomposition. In ICML, 2023.
[26] Zijie Huang, Yizhou Sun, and Wei Wang. Coupled graph ode for learning interacting system dynamics. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 705-715, 2021.
[27] Zijie Huang, Yizhou Sun, and Wei Wang. Generalizing graph ode for learning complex system dynamies across environments. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 798-809, 2023.
11

---

[28] Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, and Xiang Ren. A good prompt is worth millions of parameters: Low-resouree prompt-based learning for Vasion-language models. arXiv preprini arXiv:21 10.00444 2021.
[29] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 19113-19122, 2023.
[30] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional
networks. In ICLR, 2017.
[31] Matthieu Kirchmeyer, Yuan Yin, Jérémie Donå, Nicolas Baskiotis, Alain Rakotomamonjy, and Patrick Gallinari. Generalizing to new physical systems via context-informed dynamics model. In International Conference on Machine Learning, pages 1283-11301. PMLR, 2022.
[32] Dmitrii Kochkov, Jamie A Smith, Ayya Alieva, Qing Wang, Michael Brenner, and Stephan Hoyer. Machine learning-acceleran computational fluid dynamics. Proceedings of the National Academy of Sciences, 1 : 2021.
[33] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, al. Wilds: A benchmark of in-the-wild distribution shifts. In International conference on machine learning, pages 5637-5664. PMLR, 2021.
[34] Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems, 34:26548-26566 2021.
[35] Jogendra Nath Kundu, Naveen Venkat, Ambareesh Revanur, R Venkatesh Babu, et al. Towards inheritable models for open-set domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12376-1223 12385, 2020.
[36] Thorsten Kurth, Shashank Subramanian, Peter Harrington, Jaideep Pathak, Morteza Mardani, David Hall, Andrea Micle, Karthik Kashinath, and Anima Anandkumar. Fourcastnet: Accel- erating global high-resolution weather forecasting using adaptive fourier neural operators. In Proceedings of the platform for advanced scientific computing conference, pages 1-11, 2023.
[37] Vangipuram Lakshmikantham. Method of variation of parameters for dynamic systems. Rout- ledge, 2019.
[38] Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, al. Learning skillful medium-range global weather forecasting. Science, 382(6677)
2023.
[39] Soledad Le Clainche, Esteban Ferrer, Sam Gibson, Elisabeth Cross, Alessandro Parente, and Ricardo Vinuesa. Improving aircraft performance using machine learning: review. Aerospace Science and Technology, page 108354, 2023.
[40] Ang Li, Yixiao Duan, Huanrui Yang, Yiran Chen, and Jianlei Yang. Tiprde: task-independent privacy-respecting data crowdsourcing framework for deep learning with anonymized interme- diate representations. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 824 832, 2020.
[41] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In CVPR, pages 5400-5409, 2018.
[42] Haoliang Li, YuFei Wang, Renjie Wan, Shiqi Wang, Tie-Qiang Li, and Alex Kot. Domain generalization for medical imaging classification with linear-dependency regularization. In NeurlPS, pages 31 18-3129, 2020.
[43] Haoyang Li and Lei Chen. Cache-based gnn system for dynamic graphs. In Proceedings of the 30th ACM International Conference on Information Knowledge Management, pages 937- 946, 2021.
12

---

[44] Jinxi Li, Ziyang Song. and Bo Yang. Nvfi: Neural velocity fields for 3d physics learning from dynamic videos. In NeurlPS, 2023.
[45] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differen- tial equations. arXiv preprint arKiv:2010.0885 2020.
[46] Zongyi Li, Nikola Borislavov Kovachki, Chris Choy, Boyi Li, Jean Kossaifi, Shourya Prakash Otta, Mohammad Amin Nabian, Maximilian Stadler, Christian Hundt, Kamyar Azizzadenesheli, et al. Geometry-informe neural operator for large-scale 3d pdes. In NeurlPS, 2023.
[47] Phillip Lippe, Bastiaan S Veeling, Paris Perdikaris, Richard E Turner, and Johannes Brandstetter. Pde-refiner: Achieving accurate long rollouts with neural pde solvers. In NeurlPS, 2023.
[48] Yajing Liu, Yuning Lu, Hao Liu, Yaozu An, Zhuoran Xu, Zhuokun Yao, Baofeng Zhang, Zhiwei Xiong, and Chenguang Gui. Hierarchical prompt learning for multi-task learning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10888-10898, 2023.
[49] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF international conference on computer vision, pages 10012-10022, 2021.
[50] Xinwei Long, Shuzi Niu, and Yucheng Li. Position enhanced mention graph attention network for dialogue relation extraction. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1985-1989, 2021.
[51] Renze Lou, Kai Zhang, and Wenpeng Yin. Is prompt all you need? no. comprehensive and broader view of instruction learning. arXiv preprint arXiv:2303./0475. 2023.
[52] Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5206-5215, 2022.
[53] Xiao Luo, Yiyang Gu, Huiyu Jiang, Hang Zhou, Jinsheng Huang, Wei Ju, Zhiping Xiao, Ming Zhang, and Yizhou Sun. Pgode: Towards high-quality system dynamics modeling. In Forty-first International Conference on Machine Learning, 2024.
[54] Xiao Luo, Haixin Wang, Zijie Huang, Huiyu Jiang, Abhijeet Gangan, Song Jiang, and Yizhou Sun. Care: Modeling interacting dynamics under temporal environmental variation. Advances in Neural Information Processing Systems, 36, 2024.
[55] Xiao Luo, Jingyang Yuan, Zijie Huang. Huiyu Jiang, Yifang Qin, Wei Ju, Ming Zhang, and Yizhou Sun. Hope: High-order graph ode for modeling interacting dynamics. In International Conference Machine Learning, pages 23124-23133 PMIRR 2023.
[56] Xihaier Luo, Wei Xu, Balu Nadiga, Yihui Ren, and Shinjae Yoo. Continuous field reconstruction from sparse observations with implicit neural networks. In The Twelfth International Conference on Learning Representations, 2024.
[57] Nima Mohajerin and Steven L Waslander. Multistep prediction of dynamic systems with recur- rent neural networks. IEEE transactions on neural networks and learning systems, 30(11):3370- 3383, 2019.
[58] Octavi Obiols-Sales, Abhinav Vishnu, Nicholas Malaya, and Aparna Chandramowliswharaa. Cfdnet: A deep learning- based accelerator for fluid simulations. In Proceedings of the 34th ACM international conference on supercomputing, pages 1- 2020.
[59] Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W Battaglia. Learning mesh-based simulation with graph networks. arXiv preprint arXiv:2010.03409, 2020.
[60] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informet neural networks: deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
13

---

[61] Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Multistep neural networks for data-driven discovery of nonlinear dynamical systems. arXiv preprint arXiv:1801.01255 2018.
[62] Chengping Rao, Pu Ren, Qi Wang, Oral Buyukozturk, Hao Sun, and Yang Liu. Encoding physics to learn reaction-diffusion processes. Nature Machine Intelligence, 5(7):765-779, 2023.
[63] Bogdan Raonic, Roberto Molinaro, Tim De Ryck, Tobias Rohner, Francesca Bartolucci, Rima Alaifari, Siddhartha Mishra, and Emmanuel de Bézenac. Convolutional neural operators for robust and accurate learning of pdes. Advances in Neural Information Processing Systems, 36, 2024.
[64] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted CnaveenttonMMMCCAA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241 Springer, 2015.
[65] Alvaro Battaglia. Learning to simulate complex physics with graph networks. In ICML, pages 8459- 8468, 2020.
Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter
[66] Yidi Shao, Chen Change Loy, and Bo Dai. Transformer with implicit edges for particle-based physics simulation. In ECCV, pages 549-564, 2022.
[67] Janny Steeven, Nadri Madiha, Digne Julie, and Wolf Christian. Space and time continuous physics simulation from partial observations. In ICLR, 2024.
[68] Onder Tutsoy. Graph theory based large-scale machine learning with multi-dimensional con- strained optimization approaches for exact epidemiological modelling of pandemic diseases. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[69] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez,
E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, 2017.
[70] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.
[71] Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In NeurIPS, 2018.
[72] Haixin Wang, Hao Wu, Jinan Sun, Shikun Zhang, Chong Chen, Xian-Sheng Hua, and Xiao Luo. Idea: An invariant perspective for efficient domain adaptive image retrieval. Advances in Neural Information Processing Systems, 36:57256-57275, 2023.
[73] Kun Wang, Yuxuan Liang, Xinglin Li, Guohao Li, Bernard Ghanem, Roger Zimmermann, Huahui Yi, Yudong Zhang, Yang Wang, et al. Brave the wind and the waves: Discovering robust and generalizable graph lottery tickets. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
[74] Kun Wang, Hao Wu, Yifan Duan, Guibin Zhang, Kai Wang. Xiaojiang Peng. Yu Zheng, Yuxuan Liang, and Yang Wang. Nuwadynamics: Discovering and updating in causal spatio-temporal modeling.
[75] Yuqing Wang, Xiangxian Li, Zhuang Qi. Jingyu Li, Xuelong Li, Xiangxu Meng, and Lei Meng. Meta-causal feature learning for out-of-distribution generalization. In ECCV, pages 530-545, 2022.
[76] Zifeng Wang, Zizhao Zhang, Chen Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Leamning to prompt for continual learning. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pages 139 149, 2022.
[77] Ziqi Wang, Marco Loog, and Jan van Gemert. Respecting domain relations: Hypothesis invariance for domain generalization. In ICPR, pages 9756-9763, 2021.
14

---

[78] Haixu Wu, Tengge Hu, Huakun Luo, Jianmin Wang, and Mingsheng Long. Solving high- dimensional pdes with latent spectral models. arXiv preprint arXiv:2501.12664, 2023.
[79] Hao Wu, Yuxuan Liang, Wei Xiong, Zhengyang Zhou, Wei Huang, Shilong Wang, and Kun Wang. Earthfarsser: Versatile spatio-temppll dynamical systems modeling in one model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15906-15914, 2024.
[80] Hao Wu, Huiyuan Wang, Kun Wang, Weiyan Wang. Changan Ye, Yangyu Tao, Chong Chen, Xian-Sheng Hua, and Xiao Luo. Prometheus: Out-of-distribution fluid dynamics modeling with disentangled graph ode. In Proceedings of the 41 International Conference on Machine Learning, page PMLR 235, Vienna, Austria, 2024. PMLR.
[81] Hao Wu, Wei Xion, Fan Xu, Xiao Luo, Chong Chen, Xian-Sheng Hua, and Haixin Wang. Pastnet: Introducing physical inductive biases for spatio-temporal video prediction. arXiv preprint wwxXiv22305.14222 2023.
[82] Hao Wu, Shuyi Zhou, Xiaomeng Huang, and Wei Xiong. Neural manifold operators for learning the evolution of physical dynamics, 2024.
[83] Qitian Wu, Hengnui Zhang, Junchi Yan, and David Wipf. Handling distribution shifts on graphs: An invariance perspective. arXiv preprint arXiv:2202.02-48 2022.
[84] JiaLu Xing, JianPing Liu, Jian Wang, LuLu Sun, Xi Chen, XunXun Gu, and YingFei Wang. survey of efficient fine-tuning methods for vision-language models ppomppt and adapter. Computers & Graphics, 2024.
[85] Chenxiao Yang, Qitian Wu, Qingsong Wen, Zhiqiang Zhou, Liang Sun, and Junchi Yan. Towards out-of-distribution sequential event prediction: A causal treatment. arXiv preprint arXiv:2210.11 3300,, 2022.
[86] Sixuan Yang. Pengje Tang, Hanli Wang. and Qinyu Li. Position embedding fusion on trans- former for dense video captioning. In Developments of Artificial Intelligence Technologies in Computation and Robotics: Proceedings of the 14th International FLINS Conference (FLINS 2020), pages 792-799. World Scientific, 2020.
[87] Yuan Yin, Ibrahim Ayed, Emmanuel de Bézenac, Nicolas Baskiotis, and Patrick Gallinari. Leads: Learning dynamical systems that generalize across environments. Advances in Neural Information Processing Systems, 14:7561-75733 2021.
[88] Hong- Hong-Xing Yu, Yang Zheng, Yuan Gao, Yitong Deng, Bo Zhu, and Jiajun Wu. Inferring hybrid neural fluid fields from videos. In NeurlPS, 2023.
[89] Youn-Yeol Yu, Jeongwhan Choi, Woojin Cho, Kookjin Lee, Nayong Kim, Kiseok Chang. ChangSeung Woo, Iloo Kim, Seok Woo Lee, Joon Young Yang, et al. Learning flexible body collision dynamics with hierarchical contact mesh transformer. In ICLR, 2024.
[90] Yaohua Zha, Jinpeng Wang, Tao Dai, Bin Chen, Zhi Wang, and Shu-Ta Xia. Instance- dynamic prompt tuning for pre-trained point cloud models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14161-14170, 2023.
[91] Zeyang Zhang, Xin Wang, Ziwei Zhang, Haoyang Li, Zhou Qin, and Wenwu Zhu. Dynamic graph neural networks under spatio-temporal distribution shift, Advances in neural information processing systems, 35:6074-6088 2022.
[92] Zizhuo Zhang and Bang Wang. Prompt learning for news recommendation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 227-237, 2023.
[93] Xi Zhao, Linghsuan Meng, Xu Tong, Xiaotong Xu, Wenxin Wang, Zhongrong Miao, Dapeng Mo, et al. A novel computational fluid dynamic method and validation for assessing distal cerebrovascular microcirculatory resistance. Computer Methods and Programs in Biomedicine. 230:107338. 2023.
15

---

[94] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1681 (6816-16825, 2022.
16

---

A Proofs of Theorem 3.1 and Theorem 3.2
Proof of Theorem 3.1. We first introduce lemma as follows.
Lemma A.I. Suppose that i Mix b(t), Io. Then, the solution can be written
x(t) zoe + S. b(s)c
(25)
The proof of Lemma A.I from Method Variation of Parameters [37]. With Lemma A.1, we can prove Theorem 3.1.
By Lemma 1. we have
x(1) zie + S f( ((s)eMil
(26)
and
Noe Mel + S(¹ 8' ('(s)eali(too)
v()) = voe
(27)
follows that
=(1) v(1) S (b(s) (())
(28)
By the Mean Value Theorem for Integrals, we have
/ ddiilcss
xtt) - u(t) = (b(s') - -b(s')-b'(s))
(29)
Thus, we obtain that
[x(()) v(() 2 1o 1)/M2,
(30)
which we complete the proof.
Proof of Theorem 3.2. To prove the theorem, need the following lemma, which can be found in [5].
Lemma A.2. Given the following ODEs in R"
f = A(t) + t) r(0) = To
(31)
ý A(ty + G(y,t), y(0) D0.
assume that F is globally Lipschitz continuous and "close to G. In other words, there exist I > 0 and such that
F(x, () - F(y.() |]| S L: : || vll. F(x, G(x, 1) Se,
for all x, E RD and E (0,T), for all RD and E (0, T),
(32)
and assume that
(1L,, s)(l: S (t--s) for all S S SICT,
(33)
which || lI denotes the induced matrix norm associated with vector norm
I1, (t, s) denotes the
transition matrix for A(t), and 2 1. Then for all E (0,T), if + cL +
[|x(() - ((t)) -
ec
(c()teL)11)
(34)
17 + cL
In this case, set 1,7) 17 0. Then, Theorem 3.2 is clear from the Lemma A.2.
17

---

B Our Basic Forecasting Mode Details
Our base forecasting model combines two parallel modules [81]: the Fourier Neural Operator (FNO) [45] and a Vision Transformer (ViT)-based convolution [10]. The FNO processes the input observation embeddings in the frequency domain with Fast Fourier Transform (FFT) and inverse FFT (iFFT), capturing frequency features. The VIT module uses multi-head attention mechanism to process the spatial features of the input data. The PURE framework generates time-evolving prompt embeddings using Graph ODE, capturing dynamic changes in spatio-temporal features. These embeddings, along with the features from the FNO and ViT modules, are integrated using skip connections and Multi-Layer Perceptron (MLP) to produce the final predictions. The model optimizes by reducing the mean squared error (MSE) between predictions and the ground truth, and by enhancing robustness through reducing mutual information between prompt and observation embeddings. This approach ensures high accuracy in -ustt--diitiiuttion fluid dynamics forecasting.
C Related work
Dynamical System Modeling. The field combining machine learning with dynamical systems aims to use machine learning methods to model, predict, and control the behavior of dynamical systems [45, 80, 59, 55, 2, 54, 81]. Key techniques include using neural networks to extract patterns from spatiotemporal data, such as Convolutional Neural Networks (CNN) [63, 61], Graph Neural Networks (GNN) [59, 43, 38], and Transformer models [79, 4, 36). Additionally, Physics-Informed Neural Networks (PINN) [34, 60] embed physical laws into neural networks to enhance the model's physical consistency. These methods apply both short-term and long-term predictions of dynamical systems and optimize control strategies in areas like robotics and autonomous driving [11, 57]. To address the challenge of -iittrddiittiittionn data, researchers develop new datasets and benchmarks to evaluate model performance under different data distributions [80]. This field finds wide applications in aerospace, biomedical, and meteorological domains [93, 39]. In this work, we propose framework named PURE, which uses prompt learning and graph neural ODE to address complex distribution shifts fluid dynamics due to parameter and temporal changes.
Out-of- distribbtibt Generalization Out-of-distribution (OOD) [71, 73, 83, 85, 27] generalization means model performs well on new, unseen data. The core goal in this field is improve model performance when training and test data come from different distributions. Models that excel in OOD scenarios should be robust and adaptable. Researchers have proposed several methods, such as data augmentation [74, 8]. invariant feature learning [42, 77, 41, 72], adversarial training [75, 8]. and domain adaptation [35, 15]. These methods are widely used in areas like autonomous vehicles, medical diagnosis, financial forecasting, and dynamical systems modeling [80, 13, 53]. In this work, we propose PURE, which uses prompt learning and graph neural ODE to adapt spatio-temporal forecasting models to address distribution shifts in fluid dynamics.
Prompt Learning. Prompt learning [48, 16, 24] has recently gained significant attention a strategy for adapting pre- pre-trained models to various downstream tasks by leveraging the power of prompt- based fine-tuning [9, 29, 94, 76, 52]. In the domain of large language models, prompt learning aims to incorporate optimal tokens into the input sequence, which can effectively improve performance without extensive retraining [28, 92, 90]. In the context of fluid dynamics modeling, prompt learning means a supplementary hint to indicate the context, which is incorporated into the input (observation embedding) for better generalization. Although it shares a similar meaning as prompt tuning in language models, our prompt refers to the current environment, which determines the future evolution with better generalization.
D
The Proposed PURE Algorithm
The whole learning algorithm of PURE is summarized in Algorithm 1.
E Detailed description of datasets
We evaluate our proposed PURE on five physical benchmarks.
18

---

Algorithm PURE Framework
Require: Historical observations {s! (ooo N physical parameters &
Ensure: Future observations {s; To+ 1:76+77 2=1-
Ji=1
1: Initialize prompt embeddings using Multi-view Context Exploration
2: for each sensor i do
3: Map location at; and initial observation se to embeddings p; and qi using FFNS (.))) and
(.)
4: Aggregate embeddings: e p;
@ qi
5: end for
6: Stack initial embeddings E = { and apply self-attention blocks &f (SA,(())
7: Retrieve representations u for each query position Bi using attention mechanism
8: Generate 3D representation tensor U and integrate with parameter embedding u² 0² (E) to obtain Ü
9: Enhance representation in frequency domain: H = iPFT(TeN(FT(()))
Flatten tensor H and retrieve initial prompt embeddings =
11: Learn Time-evolving Prompts with Graph ODE
for each timestamp do
13: Interpolate observations sf from historical sequences
Update prompt embeddings 2 using continuous graph ODE 15: Incorporate interpolated observations into graph ODE with attention mechanism
end for
17: Model Adaptation with Prompt Embeddings
18: for each sensor i do
19: Concatenate observation embeddings nl and prompt embeddings of to obtain ç
20: end for
21: Incorporate ziद into spatio-temporal forecasting model
22: Optimize framework by minimizing MSE loss LySE and mutual information loss CMI
23: return Predicted future observations {s] 1o =0
Prometheus [80] is large-scale fluid dynamics dataset focused on studying out-of-distributio (OOD) generalization. This dataset simulates tunnel and pool fire scenarios, generating 4.8TB of raw data compressed to 340GB. The tunnel fire simulation takes place in tunnel 100 meters long, 6 meters wide, and meters high. It adjusts the heat release rate (HRR) and ventilation speed to create 30 different environmental combinations. The pool fire simulation occurs in 150x 100 meter area with tanks and buildings, creating 25 different environmental combinations by adjusting HRR and ventilation speed. Each scenario includes a high-density sensor network measure temperature and gas concentration. The dataset integrates advanced engineering methods, focusing on precise and efficient data analysis and inference irregular grid structures. Prometheus provides rich data resources and benchmarks for OOD generalization research in fluid dynamics.
Klvver-SStokes equations [45] depict the motion of viscous, incompressible fluid. The equations as follows vorticity form on the unit torus:
0, 1) u((, 1) Vw(x, 1) vAw(x, f(x), E (0,1)), LE (0,7)
V V. v-u(z,t 0, x (0. 1)², te [0.7]
(35)
w(x, 0) wo(>), x E (0, 1)2
We solve these equations using the stream function formulation and pseudo-spectral approach. In particular, we first solve the Poisson equation in order identify the velocity field. Afterward, we differentiate vorticity, compute the nonlinear terms, and apply de-aliasing. We use the Crank- Nicolson scheme for time-stepping, recording the solution at time intervals of = 1 on a 256x256 grid followed by downsampling. For the Bayesian inverse problem, the timestep during data generation le-4, and MCMC, 2e-2. We simulate the Navier-Stokes equations with varying viscosity coefficients, adjusting to study its impact on fluid flow and vorticity distribution.
19

---

Spherical Shallow Water Equations (Spherical-SWE) [14] describe the large-scale atmospheric and oceanic fluid motion on Earth's 's surface. The equations are:
0,h + (hu)= = 0 d,u + (u + -kx== vAu
(36)
Here, / represents fluid thickness, u is the fluid velocity vector, / represents the Coriolis parameter, represents the unit vertical vector, 9 represents the gravitational acceleration, / represents the viscosity coefficient, and A represents the Laplacian operator. The first equation (continuity equation) represents mass conservation, describing changes in fluid thickness. The second equation (momentum equation) represents momentum conservation, including advection, Coriolis force, pressure gradient force, and viscous diffusion. We simulate the Spherical Shallow Water Equations with different viscosity coefficients v, adjusting v to study its effects on fluid motion.
3D Reaction- Reaction-Diffusion Equations [62] describe the diffusion and reaction of chemical substances in space. The general form of these equations is:
0,u 0u= (u,
(37)
0,0 D,Av + Ru(
Here, u and v represent the concentrations of the chemical substances, Du and D, represent the diffusion coefficients for and v, respectively, A is the Laplacian operator, and R., v) and R. v) denote the reaction terms that represent the reaction rates between u and v. Diffusion terms, i.e., Au and Av denote the diffusion process of the chemicals in space. The diffusion coefficients D,, and D, determine the rate of diffusion. Reaction terms, i.e., Ru (v, v) and Rs (u, v) describe the reaction rates of the chemical substances. These terms depend the concentrations of u and v, and can include linear reactions, nonlinear reactions, and complex dynamic processes. We simulate the 3D -affiinn equations with different diffusion coefficients D. and D. study the effects of diffusion rates on the distribution and reaction rates of the chemical substances.
ERA5 [23] is global atmospheric reanalysis dataset produced by ECMWF, providing weather data from 1979 to the present with high spatial (31 km) and temporal (hourly) resolution. includes surface height, and two-meter temperature. ERA5 data supports applications in weather forecasting, climate research, environmental monitoring, energy management, and agriculture. Accessible via the Copernieus Climate Data Store, ERA5 is crucial for analyzing and predicting meteorological and climate phenomena.
variables like surface pressure, surface temperature,
Table 5:
The table presents the In-Domain and Adaptation environments for various benchmarks. Training and testing in the In-Domain environment is called w/o OOD experiment, while training in the In-Domain environment and testing in the Adaplation environment is called w/ OOD experiment.
BENCHMARKS
IN DOMAIN ENVIRONMENTS
ADAPTATION ENVIRONMENTS
PROMETHEUS
Ogynn Ras (b, by 62g)
(020, O2ri 0an) (ba ba bas
2D NAVIER-STOEES EQUATION
{le
le
(1e
le
SPHERICAL SHALLOW WATER EQUATION
(le 1. le 2,.., le
le 10
[le
u. le 12)
3D REACTION DIFFUSOONOO EQUATIONS ERA5
D = [2.1 x 10-2, 16 10-9,6.1 6.1 10-3)
D [2.03 x 10 1.96 x 10-2)
V SST,
V (85R
F Details of Compared Approaches
The compared approaches involved in this study is as follows:
U-Net [64] is a convolutional neural network initially used for biomedical image segmentation. has symmetric U-shaped structure and uses skip connections to link the encoder and decoder, enabling efficient feature fusion.
ResNet [21] introduces residual blocks to solve the degradation problem in deep networks. It allows the network to be deeper and easier to train by using skip connections to directly pass information.
ViT [10] applies the Transformer model to image recognition. It divides the image sample into patches and uses self-attention mechanisms to process these patches, balancing computational efficiency and performance.
20

---

Swin'T [49] introduces a sliding window mechanism for effective local and global feature extraction. It is suitable for various computer vision tasks.
FNO [45] uses Fourier transforms for global feature extraction, suitable for processing continuous field data and efficiently solving PDEs.
UNO [1] combines the U-Net architecture with optimization methods to enhance feature extraction and fusion capabilities, improving model performance.
CNO [63] combines convolution operations with operator learning. focusing on high-dimensional continuous data and modeling complex dynamic systems.
NMO [82] enhances the modeling capability for multi-scale dynamic systems by combining neural networks with manifold learning algorithms.
CGODE [26] neural ODE model that aims to capture the dynamics of both nodes and edges jointly.
DGODE [80] addresses the challenge of out-of-distribution (OOD) generalization in fluid dynamics modeling by learning disentangled representations using temporal GNN and a frequency net- work. It minimizes mutual information between node and environment representations to mitigate distribution shifts and employs coupled graph ODE framework for robust modeling.
G
Metrics details
Mean Squared Error (MSE). Mean Squared Error measures the gap between the predicted and ground truth. The formula is:
MSE
U1s -;))
(38)
==1
in which ys denotes the actual value, 5: denotes the predicted value, and n denotes the number of data points.
Relative L2 Error. Relative L2 Error evaluates the relative accuracy of the model's 's predictions. The Relative L2 Error
formula is:
|ly 8112
(39)
||yl|
where
|| ||2 denotes the L2 norm, y is the vector of actual values, and y is predicted values.
H More experiment results
Sparse Reconstruction Experiments. The experimental setup includes sparse reconstruction experiments on the Prometheus and ERA5 datasets. For the Prometheus dataset, the sparsity rates are set to 25%, 50%, and 75%. For the ERA5 dataset, the sparsity rates set 5% and 25%. Each set of experimental results includes the original data, sparse input data, results from our method, and results from MMGNet [56]. The experiments evaluate the performance of each method by comparing the reconstruction results at different sparsity rates.
Figure 6 shows two sets of sparse reconstruction experiment results from the Prometheus and ERA5 datasets. The sparsity rates Prometheus are 25%, 50%, and 75%, while for ERA5, they are 5% and 25%. Each set displays the Ground Truth, Sparse Input, our reconstruction method (Ours), and MMGNet's results in order. As the sparsity rate increases, the quality of the reconstruction decreases. At low sparsity rates, both our method and MMGNet effectively restore image details. At high sparsity rates, our method performs better in preserving details and recovering overall structure.
b More Ablation Experiments. Table show the ablation study outcomes on the Navier-Stokes equations. We use MSE assess the contribution of each component. We remove different com- ponents from the PURE method and compare them with the original FNO model and the complete FNO + PURE method. The complete FNO PURE method achieves the lowest MSE of 0.0987, while the original FNO model has error of O. 157.. Removing Graph ODE, interpolation, mutual information minimization, and FFT increases the error 282, 1097. 0. 182, and 1266, respec- tively. These results clearly show that each component of the PURE method significantly improves
21

---

Sparse cate 5 25%
Sparse sate S 50%
Spurse ease S 75%
5.5%
$'25%
Topet
U
T
Ground Iruth
Spaces Inpur. 25%
Ouue
MMGNet
Oure Errot
SEGGNNe Ereor
Figure 6: The figure shows sparse reconstruction results using the Prometheus and ERA5 datasets at various sparsity rates. Each group displays the ground truth, sparse input, our reconstruction method, and MMGNet's results. As the sparsity rate increases, the quality of the reconstruction decreases. U and V represent velocity components, and represents temperature.
Table 6: Ablation Studies Navier-Stokes equations (with viscosity coefficient of v {)e-3).
VARIANTS
EQUATIONS
FNO PURE w/o GRAPH ODE
0.1282
FNO + PURE w/o INTERPOLATION
0.1097
FNO + PURE w/O MI
0.1182
FNO PURE W/O FFT
0.1266
FNO
0.1567
FNO PURE
0.0987
the model's predictive performance. Removing any component leads to performance degradation, proving the importance of these components in enhancing prediction accuracy.
b Performance with respect to Different Difficulty Levels. Here, we demonstrate the performance of our PURE with varying difficulty levels. In particular, we measure the difficulty levels based on the distance between Pimin (E) and P (E) and generate three levels on the Prometheus dataset. The compared results are shown in Table 7. From the results, can observe that all the model performs worse in hard scenarios and our method has consistently outperformed these baselines. The potential reason is that (1) our model enhances model invariance across different distributions through decoupling prompt embeddings and observation embeddings via mutual information which results in high generalizationability to different environments; (2) our model utilizes multi-view context mining and graph ODE to extract prompt embeddings, which capture environment information accurately.
Table 7: Performance comparison across varying levels of OOD generalization difficulty. The values represent the Mean Squared Error (MSE) for each method.
METHOD
U-NET
RESNET
VIT
SwIN-T
FNO
CGODE
PURE
EASY
0.0945
0.0682
0.0654
0.0676
0.0452
0.0772
0.0325
MID
0.1063
0.0922
0.0902
0.0912
0.0544
0.0863
0.0341
HARD
0. 0412
0.1234
0.1076
0.1123
0.0623
0.0921
0.0354
Robustness to Noisy Data. We have also experimented with noisy data to evaluate the robustness of our method. The results are shown in Table 8. From the results, the performance of both ResNet and NMO models degrades significantly when noise is introduced. However, the integration of our PURE with these models substantially mitigates the impact of noise, leading to much lower MSE values compared to their basclines.
> Expanded Evaluation OOD Generalization in Dynamical Systems. To further validate the effectiveness of our PURE, we include three additional models specifically designed for out- of-distribution generalization in dynamical systems: LEADS [87], CODA [31], and NUWA [74]. Table 9 shows the performance of each method across different datasets in both in-distribution (ID) and onttt--distriuution (OOD) scenarios. The results indicate that our proposed method outperforms these basclines on all datasets, especially in OOD scenarios.
22

---

Table 8: Performance comparison under noisy data conditions. Squared Error (MSE) for each method with and without noise.
The values represent the Mean
Dataset
ResNet/Noise
ResNet-PURE/Nois
NMO/Noise
NMO+PURE/Nose
PROMETHEUS
0.0674 0.3422
0.0542 0.0586
0.0397 0.1287
0.0281 0.0309 0.0876 0.0892
NS
0.1823 0.6572
0.1492 / 0.1537
0.1021 0.2542
Table 9: Performance comparison of our method (PURE) against additional baselines on vari- ous datasets. The values represent Mean Squared Error (MSE) in-distribution (ID) and out-of- distribution (OOD) scenarios.
Prometheus
ERA5
SSWE
Dataset
ID
OOD
ID
OOD
ID
OOD
LEADS
0.0374
0,0403
0.2367
0,4233
0.0038
0.0047
CODA
0.0353
0.0372
0.1233
0.2367
0.0034
0.0043
NUWA
0.0359
0,0398
0.0645
0.0987
0.0032
0.0039
PURE (Ours)
0.0323
0.0328
0.0398
0.0401
0.0022
0.0024
I Limitations of This Study
Although the PURE method shows superiority multiple benchmark datasets, it has some limitations. First, we assume that training and test data are independent identically distributed (IID). This may be not true in some extreme physical scenarios due to environmental changes causing significant data distribution shifts. Second, while the PURE method performs well addressing distribution shifts, may still face challenges when dealing with high-ddmeensionl and complex fluid dynamics systems. Additionally, the PURE method has high computational complexity, requiring more computational resources and time in practical applications. Future research focus optimizing the algorithm to improve computational efficiency and extending it to more real-world scenarios such as rigid dynamics modeling and traffic flow forecasting.
J Borader Impact
The PURE method significantly impacts out-of-dissributin fluid dynamics modeling. It adapts to different scenarios, improving the model's performance in handling distribution changes. This is helpful for climate prediction, epidemic spread, aerospace, and biomedical fields. In future works, we will extend our PURE to more real-world scenarios such as rigid dynamics modeling and traffic flow forecasting.
23

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatinn right after your answer (even for NA).
The checklist answers
integral part of your paper submission. They visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist one of the factors in their evaluation.
While "[Yes] is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). In general, answering "[No] " or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
We have made clear claims about contributions and scopes in the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
24

---

Justification:
We have the seperated limitation section in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We provide detailed theoretical proofs in the main paper and appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe the details of our model and training strategy in the paper for reproduction.
Guidelines:
25

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We offer the implementation code reproduce work.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
Beteiii/guudde//ooeeeue wttpsuuiiissiooooooooooooioie for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //n cts.pss.cc/uubic/uuuuuu. gesds//////eeeeeee/e /CodeSubmian ss ierionnoolliei onoolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
26

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperpa- rameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: we have introduced experiment settings and details, and even add more details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:[No
Justification: We do not report specific error bars for the following reasons:
(a) In fluid dynamies modeling, using a fixed random seed shows consistent performance, and the performance difference with different seeds is small [78, 45].
(b) Related work [45, 80, 55] in this field does not report error bars.
(c) To ensure fairness, we fix all random seeds and conduct experiments on the same machine
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: we have discussed the information about computer resources.
27

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httppp////arrp...ppppppc..// cet.ea/eeeeeciiieeleeeeeeeeeelee cechhiciccieeeeenneeecc
Answer: [Yes]
Justification:
We follow the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The PURE method not only makes significant contributions to academic research but also shows broad potential in practical applications. It helps address distribution changes in complex dynamic systems.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
28

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do release any data or model that has high risks for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us usel-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited the original paper that produced the code package and dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcee thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide documentation with code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.

---

14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does involve crowdsoureing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "q2thl2PsL7", "reviewer": "Reviewer_Sh2w", "summary": "The paper proposes a graph ODE-based approach for OOD fluid dynamics modeling. PURE aims to learn time-evolving prompts via graph ODE for adaptation of spatio-temporal forecasting models on OOD scenarios. To address temporal distribution shifts, the interpolation of obersvation sequences are combined into graph ODE framework to learn evolution of prompt embeddings.", "questions": "Please address the questions in the Weaknesses.", "limitations": "The limitations are explained in Appendix I.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes a new approach that connects prompt learning and dynamical system modeling which addresses OOD shifts.\n- By learning time-evolving prompts that adapt to changes in system parameters and temporal evolution, the approach can enhance model robustness.\n- The paper provides theoretical analysis on incorporating observations during evolution.\n- Experiments on diverse benchmarks show generalization ability to OOD and different prediction length.", "weaknesses": "As I am not an expert in this field, I am unable to find major concerns or weakness of the approach.\n- As the method is based on attention, the proposed approach may have limited scalability and take long computation time. Is there a comparison on these with the previous works?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful for the time you have taken to review our paper, your insightful comments and support. Your positive feedback is incredibly encouraging for us! In the following response, we would like to address your major concern and provide additional clarification.\n\n> Q1. As the method is based on attention, the proposed approach may have limited scalability and take long computation time. Is there a comparison on these with the previous works?\n\nA1. Thank you for your comment. We have added a comparison of computational costs below. From the results, we can observe that our method has a competitive computation cost with huge performance increasement. Without OOD, it improves performance by an average of 4.44%, and with OOD, it improves performance by an average of 25.90%. We will include it in our revised version. \n\n| Method       | UNet | ResNet | VIT   | SwinT | FNO  | UNO  | CNO  | NMO  | DGODE | PURE (Ours) |\n|--------------|-------|--------|-------|--------|------|------|------|------|-------|------------|\n| Training time (h) | 11.2  | 9.76   | 14.5  | 12.3   | 6.9  | 7.8  | 13.4 | 6.3  | 13.2  | 7.2        |\n| Inference time (s) | 1.34  | 0.93   | 1.32  | 1.13   | 0.54 | 0.67 | 0.12 | 0.52 | 1.23  | 0.69       |\n\nWe will also add your suggestion about future works into our revised version. Thanks again for appreciating our work and for your constructive suggestions. Please let us know if you have further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tQZ3GEfz5x", "reviewer": "Reviewer_MBSt", "summary": "This paper pioneers the connection of prompt learning with dynamical system modeling to address the challenge of out-of-distribution shifts. The proposed PURE method initializes prompt embeddings by learning from historical observations and system parameters.", "questions": "Q1. There might be misuses of symbols in the paper, such as, Change \"xi\" to \"si\" in line 65, Change \"xq\" to \"xq\" in line 96, \"Zero-shot Experiments\" and \"Generalization Experiments\" in line 215 should be treated equally and placed on separate lines.\nQ2. Is there an issue with the second-best data in Table 2? For example, in the column w/OOD of SPHERICAL-SWE, the second-best should be DGPDE 0.0028. The corresponding improvement results also need to be modified.\nQ3. What does the clustering in Figure 5 represent? Could you provide a detailed explanation?\nQ4. The paper utilizes mutual information to decouple different prompt embeddings and observation embeddings, reducing the sensitivity of observation embeddings to different distributions. However, I don't quite understand the purpose of decoupling. Observational embeddings are related to the environment, and prompt embeddings are related to the environment as well; they are inherently correlated. Please provide further explanation.", "limitations": "This method does not apply to real-world scenarios, such as rigid dynamics modeling and traffic flow forecasting.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper is easy to follow. \n2.The proposed method is sound and innovative.\n3. The authors provide theoretical proof and show comprehensive experimental comparisons.", "weaknesses": "1. Some results may be incorrectly labeled as suboptimal in table, and there are errors in the use of some symbols.\n2. The explanation of the experimental results is not detailed enough, making some experiments difficult to understand.\n3. The proposed method is aimed at OOD (Out-Of-Distribution), but the experiments lack comparison and discussion with methods specifically targeting OOD, such as [1] and [2].\nReference:\n[1] Kirchmeyer, Matthieu, et al. \"Generalizing to new physical systems via context-informed dynamics model.\" International Conference on Machine Learning. PMLR, 2022.\n[2] Yin, Yuan, et al. \"LEADS: Learning dynamical systems that generalize across environments.\" Advances in Neural Information Processing Systems 34 (2021): 7561-7573.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful for the time you have taken to review our paper, your insightful comments and support. Your positive feedback is incredibly encouraging for us! In the following response, we would like to address your major concern and provide additional clarification.\n\n\n\n> Q1. Some results may be incorrectly labeled as suboptimal in table, and there are errors in the use of some symbols. There might be misuses of symbols in the paper, such as, Change \"xi\" to \"si\" in line 65, Change \"xq\" to \"xq\" in line 96, \"Zero-shot Experiments\" and \"Generalization Experiments\" in line 215 should be treated equally and placed on separate lines. Is there an issue with the second-best data in Table 2? For example, in the column w/OOD of SPHERICAL-SWE, the second-best should be DGPDE 0.0028. The corresponding improvement results also need to be modified.\n\n\nA1. Thank you for pointing this out. We will correct all the typos carefully in the revised version.\n\n\n> Q2. The explanation of the experimental results is not detailed enough, making some experiments difficult to understand. What does the clustering in Figure 5 represent? Could you provide a detailed explanation? \n\nA2. Thank you for your comment. We demonstrate the t-SNE visualization of the predicted trajectories for compared methods. From the results, we can observe that our method outperforms baselines, which validate the superiority of our method. Here, clustering refers to the technique of reducing the dimensions in t-SNE. We will include more details in our revised version. \n\n\n\n> Q3. The proposed method is aimed at OOD (Out-Of-Distribution), but the experiments lack comparison and discussion with methods specifically targeting OOD, such as [1] and [2]. Reference: [1] Kirchmeyer, Matthieu, et al. \"Generalizing to new physical systems via context-informed dynamics model.\" International Conference on Machine Learning. PMLR, 2022. [2] Yin, Yuan, et al. \"LEADS: Learning dynamical systems that generalize across environments.\" Advances in Neural Information Processing Systems 34 (2021): 7561-7573.\n\n\nA3. Thank you for your comment. We have added two baselines LEADS [1] and CODA [2] for performance comparison. The results below show that our method is superior to these baselines, which validates the effectiveness of our method when addressing OOD problems. We will include it in our revised version. \n\n\n| Dataset     | Prometheus ID | Prometheus OOD | ERA5 ID | ERA5 OOD | SSWE ID | SSWE OOD |\n|-------------|----------------|----------------|---------|----------|---------|----------|\n| LEADS       | 0.0374         | 0.0403         | 0.2367  | 0.4233   | 0.0038  | 0.0047   |\n| CODA        | 0.0353         | 0.0372         | 0.1233  | 0.2367   | 0.0034  | 0.0043   |\n| Ours | 0.0323         | 0.0328         | 0.0398  | 0.0401   | 0.0022  | 0.0024   |\n\n\n\n\n> Q4. The paper utilizes mutual information to decouple different prompt embeddings and observation embeddings, reducing the sensitivity of observation embeddings to different distributions. However, I don't quite understand the purpose of decoupling. Observational embeddings are related to the environment, and prompt embeddings are related to the environment as well; they are inherently correlated. Please provide further explanation.\n\nA4. Thank you for your comment. Our decoupling aims to build invariance of our observation embeddings to environments for better generalization. Here, both observation embeddings and prompt embeddings are combined for the final prediction, and prompt embeddings are utilized to provide the environment information. In other words, observational embeddings are not correlated with the environment for better generalization. We will include it in our revised version. \n\n> Q5. This method does not apply to real-world scenarios, such as rigid dynamics modeling and traffic flow forecasting.\n\nA5. Thank you for your comments.  We have added four baselines, i.e., EGNN [1], SGNN [2], SimVP [3], PastNet [4] for performance comparison on the RigidBall and TaxiBJ datasets. The table shows our method outperforms the baseline models in rigid dynamics and traffic flow forecasting. Please see the PDF for visual results. We will include it in our revised version.、\n\n\n**Rigid dynamics (MSE)**\n\n| Method | PL 10 $\\downarrow$ | PL 20 $\\downarrow$ | PL 30 $\\downarrow$ | PL 40 $\\downarrow$ | PL 50 $\\downarrow$ |\n|--------|-------|-------|-------|-------|-------|\n| EGNN   | 1.37  | 1.89  | 3.77  | 5.66  | 7.87  |\n| SGNN   | 0.64  | 0.72  | 1.23  | 2.44  | 4.96  |\n| PURE   | **0.59** | **0.68**  | **0.97**  | **1.45**  | **3.99**  |\n\n**Traffic flow**\n\n| Metric | MSE $\\downarrow$   | MAE $\\downarrow$   | SSIM $\\uparrow$ | PSNR$\\uparrow$  |\n|--------|--------|--------|-------|-------|\n| SimVP  | 0.4332 | 16.897 | 0.9822| 39.29 |\n| PastNet| 0.4293 | 16.405 | 0.9876| 39.42 |\n| PURE   | **0.3982** | **15.434** | **0.9971**| **41.23** |\n\n\n**Reference**\n\n[1] Satorras, Vıctor Garcia, Emiel Hoogeboom, and Max Welling. \"E (n) equivariant graph neural networks.\" ICML2021\n\n\n[2] Han, Jiaqi, et al. \"Learning physical dynamics with subequivariant graph neural networks.\" NeurIPS2022\n\n\n[3] Gao, Zhangyang, et al. \"Simvp: Simpler yet better video prediction.\" CVPR. 2022.\n\n[4] Wu, Hao, et al. \"Pastnet: Introducing physical inductive biases for spatio-temporal video prediction.\" MM2024\n\nWe will also add your suggestion about future works to our revised version. Thanks again for appreciating our work and for your constructive suggestions. Please let us know if you have further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0i7XHiuD2S", "reviewer": "Reviewer_vRkX", "summary": "- The paper aims to improve the out-of-distribution (OOD) generalization of fluid dynamics modeling.\n\n- Two types of OOD scenarios are targeted: OOD across different systems and OOD within the same system across different timestamps.\n\n- The paper proposes a framework named PURE, composed of modules including:\n    - Multi-view Context Exploration, which explores spatio-temporal data using both the attention mechanism and the frequency domain;\n    - Time-evolving Prompt Learning, which incorporates the interpolation of observation sequences;\n    - Model Adaptation with Prompt Embeddings, which leverages time-evolving prompts to mitigate temporal distribution shifts.\n\n- Extensive experiments on a range of fluid dynamics datasets support the claim.", "questions": "NA", "limitations": "NA", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Significant topic: OOD generalization in fluid dynamics modeling.\n\n- Well-motivated, as OOD generalization is a crucial challenge in this field.\n\n- The presentation effectively delivers the message.\n\n- Extensive experiments have been conducted.", "weaknesses": "- My major concern with the paper is that the OOD challenge in dynamics modeling is not well-formulated. The paper describes the OOD scenario verbally as \"*different dynamical systems could involve different parameters in underlying rules*\" and \"*during long-term auto-regressive forecasting, the input data distribution could vary hugely during temporal evolution,*\" which is straightforward and easy to understand. However, the mathematical formulation of these scenarios is absent. This formulation should be the foundational basis of the topic, as we need to clearly define the problem before addressing it.\n\n- Given the lack of mathematical formulation of the challenge, I find myself lost in the proposed approach section, unsure of the necessity for specific components. While I understand the function of each component, I cannot see why it is needed or which gaps it aims to bridge in the absent mathematical framework.\n\n- Why is the proposed method termed \"prompt\"? Is there a connection to prompt tuning in large language models?\n\n- How do you quantify the distribution shift in dynamics modeling? Can you rank the 'difficulty level' of OOD generalization in your experiments and analyze in which scenarios your method stands out and why?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful for the time you have taken to review our paper and your insightful review. Here we address your comments in the following.\n\n>Q1. My major concern with the paper is that the OOD challenge in dynamics modeling is not well-formulated. The paper describes the OOD scenario verbally as \"different dynamical systems could involve different parameters in underlying rules\" and \"during long-term auto-regressive forecasting, the input data distribution could vary hugely during temporal evolution,\" which is straightforward and easy to understand. However, the mathematical formulation of these scenarios is absent. This formulation should be the foundational basis of the topic, as we need to clearly define the problem before addressing it.\n\n\nA1. Thank you for your comment. In dynamical systems, the OOD problem studies the prediction performance of models under parameter distributions or environments not seen during training. In formulation, the evolution of dynamical systems is defined by $\\frac{du}{dt} = F(u, \\xi)$, where $u$ is the observation, and $\\xi$ is the system parameter. If these parameters come from a distribution $\\xi \\sim P(\\xi)$, the state trajectory comes from the distribution $u^{1:T_0} \\sim P(u^{1:T_0} | \\xi)$. Assume we learn a state mapping $f$ from time $u^{1:T_0}$ to $u^{T_0+1:T_0+T}$, i.e., $u^{T_0+1:T_0+T}= f(u^{1:T_0})$. We could have different distributions across training and test datasets, i.e., $P_{\\text{train}}(\\xi)\\neq P_{\\text{test}}(\\xi)$, which results in $P_{\\text{train}}(u^{1:T_0})\\neq P_{\\text{test}}(u^{1:T_0})$. Moreover, when conducting rollout prediction, we need to feed the output back to the model, i.e., $u^{T_{start}:T_{start}+T-1}= f(u^{T_{start}-T_0:T_{start}-1})$ with $P(u^{1:T_0} | \\xi)\\neq P(u^{T_{start}-T_0:T_{start}-1}|\\xi, T_{start})$, which demonstrates temporal distribution shift. We will include our explanation in the revised version. \n\n> Q2. Given the lack of mathematical formulation of the challenge, I find myself lost in the proposed approach section, unsure of the necessity for specific components. While I understand the function of each component, I cannot see why it is needed or which gaps it aims to bridge in the absent mathematical framework.\n\nA2: Thank you for your comment. From our definition, the mapping $u_{output}=f(u_{input})$ could suffer from a serious distribution shift result from different $\\xi$ and $T_{start}$, i.e., $P(u_{input}|\\xi, T_{start})$. To reduce the impact of distribution shift, we aim to learn invariant observation embeddings $\\mu^t$ to environments, i.e., $\\xi$ and $T_{start}$, and utilize prompt embeddings $z^t$ to indicate the current environment. The basic idea of our method is to ensure the invariance of observation embeddings for better generalization, i.e., $z^t\\perp \\mu^t$. Then, the prompt embeddings would be combined to generate the future predictions, i.e., $u_{output} = \\phi ([\\mu^t,z^t])$. In our framework, a basic model is adopted to generate observation, i.e., $\\mu^t=Basic Model(u_{input})$. We adopt context mining and graph ODE to learn time-varying prompt embeddings., i.e., $z^0= Context Mining (u_{input})$, $z^t = GraphODE (z^0,t)$, which can explore the temporal evolution of environments. We will include our explanation in the revised version. \n\n> Q3 Why is the proposed method termed \"prompt\"? Is there a connection to prompt tuning in large language models?\n\nA3. Thank you for your comment. \"Prompt\" means a supplementary hint to indicate the context, which is incorporated to the input (observation embedding) for better generalization. Generally, it shares the basic idea with prompt tuning, which also aims to incorporate optimal tokens into the input sequence. However, prompt tuning usually utilizes prompts to indicate different tasks in NLP scenarios, our prompt refers to the current environment in fluid dynamical modeling, which determines the future evolution with better generalization. We will include it in our revised version. \n\n\n> Q4. How do you quantify the distribution shift in dynamics modeling? Can you rank the 'difficulty level' of OOD generalization in your experiments and analyze in which scenarios your method stands out and why?\n\nA4. Thank you for your comment. We have added experiments to demonstrate the performance with varying difficulty levels. In particular, we measure the difficulty levels based on the distance between $P_{\\text{train}}(\\xi)$ and $P_{\\text{test}}(\\xi)$ and generate three levels on the Prometheus dataset. The compared results are shown below. We can observe that although all the model performs worse in hard scenarios, our method consistently outperforms these baselines. The potential reason is that (1) our model enhances model invariance across different distributions through decoupling prompt embeddings and observation embeddings via mutual information which results in high generalizationability to different environments; (2) our model utilizes multi-view context mining and graph ODE to extract prompt embeddings, which capture environment information accurately. We will include it in our revised version. \n\n\n| Method      | U-NET  | ResNet | VIT    | Swin-T | FNO    | CGODE  | PURE(Ours) |\n|-------------|--------|--------|--------|--------|--------|--------|------------|\n| **Easy**    | 0.0945 | 0.0682 | 0.0654 | 0.0676 | 0.0452 | 0.0772 | **0.0325**    |\n| **Mid**     | 0.1063 | 0.0922 | 0.0902 | 0.0912 | 0.0544 | 0.0863 | **0.0341**    |\n| **Hard**    | 0.1432 | 0.1234 | 0.1076 | 0.1123 | 0.0623 | 0.0921 |  **0.0354**     |\n\n\n**Reference**\n\n[1] Wu, et al. \"Prometheus: Out-of-distribution Fluid Dynamics Modeling with Disentangled Graph ODE.\" ICML2024\n\nIn light of these responses, we hope we have addressed your concerns, and hope you will consider raising your score. If there are any additional notable points of concern that we have not yet addressed, please do not hesitate to share them, and we will promptly attend to those points.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RMPUztzEcQ", "reviewer": "Reviewer_MFD5", "summary": "The paper presents a new approach called Prompt Evolution with Graph ODE (PURE) for non-distributed fluid dynamics modeling. PURE first learns from historical observations and system parameters in the frequency domain to explore multi-view contextual information, which can efficiently initialize the cue embedding. Interpolations of the observation sequences are then merged into the graph ODE so that the time evolution of the model-adaptive cue embeddings can be captured. These time-evolving cue embeddings are then incorporated into the underlying predictive model to overcome spatio-temporal distributional variations. In addition the paper minimizes the mutual information between the cue embeddings and the observation embeddings to enhance the robustness of the model to different distributions. Finally, extensive experiments conducted on various kinds of benchmark datasets validate the superiority of the proposed PURE compared to various baselines.", "questions": "1.  What are the advantages of PURE over previous efforts to address OOD? \n\n2. Some of the symbols in the method description section are not defined, e.g., what do P and N in Equation 9 refer to?", "limitations": "The authors list limitations in the appendix but do not mention them in the main text. It is recommended that the author make a description in the main text.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The idea of the paper is novel. It is the first to link prompt learning to dynamic system modeling for out-of-distribution problems. \n \n2. This paper is technically sound. PURE first learns initialized prompt embeddings from historical observations and system parameters, and then employs a graph ODE with interpolated observation sequences to capture the continuous evolution of their model adaptation under out-of-distribution changes.  \n\n3. The experimental results show the effectiveness of PURE in different challenging environments.", "weaknesses": "1. The contribution of the proposed method in dealing with the OOD problem needs to be further clarified since the advantages of PURE over the previous efforts, such as Refs. [7, 67, 14, 72], etc., to address the OOD problem are not listed.\n\n2. The writing of the paper needs to be improved. Some of the symbols in the method description section are not defined, e.g., what do P and N in Equation 9 refer to?\n\n3. The experiment is not comprehensive enough. (a) The reasons for selecting baselines are not explained. Data augmentation [66, 7], invariant feature learning [39, 69, 38], adversarial training [67, 7], and domain adaptation [32, 14] are mentioned in the paper in related work for solving the OOD problem, but they are not be compared as baselines in the experiment. (b) The experiments in this paper do not state whether noisy data are considered. (c) The authors just give a brief description of the results without analyzing the reasons behind the high performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful for the time you have taken to review our paper, your insightful comments and support. Your positive feedback is incredibly encouraging for us! In the following response, we would like to address your major concern and provide additional clarification.\n\n> Q1. The contribution of the proposed method in dealing with the OOD problem needs to be further clarified since the advantages of PURE over the previous efforts, such as Refs. [7, 67, 14, 72], etc., to address the OOD problem are not listed.\n\nA1. Thanks for your comment. Compared with previous OOD methods, our contribution can be listed as follows:\n- **Underexplored Scenarios**. Our work studies an underexplored yet practical problem of OOD generalization in fluid dynamic modeling while previous OOD methods usually study the OOD problem in image classification scenarios.\n- **Innovative Methodlogy**. Our work not only learns time-evolving prompts using context mining and a graph ODE but also decouples prompt embeddings and observation embeddings via mutual information for invariant learning to address the OOD problem.\n- **Theoretical Analysis**. We provide a comprehensive theoretical analysis to support our designs, which makes our framework more solid.\n- **Superior Performance.** Comprehensive experiments validate the effectiveness of our method in different challenging settings. In particular, it improves performance by an average of 25.90% in OOD scenarios. \n\n> Q2. The writing of the paper needs to be improved. Some of the symbols in the method description section are not defined, e.g., what do P and N in Equation 9 refer to?\n\nA2. Thank you for your comment. $P$ denotes the set collecting all the positive pairs (i.e., from the same trajectory) of observation embeddings and prompt embeddings while $N$ denotes the set collecting all the possible pairs of observation embeddings and prompt embeddings in the dataset. We will include it in our revised version. \n\n\n> Q3. The experiment is not comprehensive enough. (a) The reasons for selecting baselines are not explained. Data augmentation [66, 7], invariant feature learning [39, 69, 38], adversarial training [67, 7], and domain adaptation [32, 14] are mentioned in the paper in related work for solving the OOD problem, but they are not be compared as baselines in the experiment. \n\nA3. Thank you for your comment. We have added three more baselines LEADS [1], CODA [2] and NUWA [3] for performance comparison. The results show that our method is superior to these baselines, which validates the effectiveness of our method when addressing OOD problems. In addition, the other references focus on the image classification problem, which cannot be adopted to solve our fluid dynamics modeling problem. We will include it in our revised version. \n\n| Dataset     | Prometheus (ID) | Prometheus (OOD) | ERA5 (ID) | ERA5 (OOD) | SSWE (ID) | SSWE (OOD) |\n|-------------|----------------|----------------|---------|----------|---------|----------|\n| LEADS       | 0.0374         | 0.0403         | 0.2367  | 0.4233   | 0.0038  | 0.0047   |\n| CODA        | 0.0353         | 0.0372         | 0.1233  | 0.2367   | 0.0034  | 0.0043   |\n| NUWA        | 0.0359         | 0.0398         | 0.0645  | 0.0987   | 0.0032  | 0.0039   |\n| PURE (ours) | 0.0323         | 0.0328         | 0.0398  | 0.0401   | 0.0022  | 0.0024   |\n\n> Q4. (b) The experiments in this paper do not state whether noisy data are considered. \n\nA4. Thank you for your comment. We have added experiments with noisy data to evaluate the robustness of our method. The results are shown below, which validate that our method is more robust to interference from noisy data. We will include it in our revised version. \n\n|            | ResNet/Noise | ResNet+PURE/Noise | NMO/Noise   | NMO+PURE/Noise |\n| ---------- | ------  | ----------------- | ------ | -------------- |\n| PROMETHEUS | 0.0674/0.3422 | 0.0542/0.0586     | 0.0397/0.1287 | 0.0281/0.0309  |\n| NS         | 0.1823/0.6572 | 0.1492/0.1537     | 0.1021/0.2542 | 0.0876/0.0892  |\n\n\n\n>Q5. (c) The authors just give a brief description of the results without analyzing the reasons behind the high performance.\n\nA5. Thank you for your comment. The potential reason for our high performance is that (1) our model enhances model invariance across different distributions through decoupling prompt embeddings and observation embeddings via mutual information which results in high generalizationability to different environments; (2) our model utilizes multi-view context mining and graph ODE to extract prompt embeddings, which capture environment information accurately. We will include it in our revised version. \n\n**Reference**\n\n[1] Kirchmeyer, Matthieu, et al. \"Generalizing to new physical systems via context-informed dynamics model.\" ICML 2022. \n\n[2] Yin, Yuan, et al. \"LEADS: Learning dynamical systems that generalize across environments.\" NeurIPS2021.\n\n[3] Wang, Kun, et al. \"NuwaDynamics: Discovering and Updating in Causal Spatio-Temporal Modeling.\" ICLR2024.\n\nWe will also add your suggestion about future works to our revised version. Thanks again for appreciating our work and for your constructive suggestions. Please let us know if you have further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

Thanks for your time and valuable feedbacks. We acknowledge **three reviewers** (Reviewer Sh2w, Reviewer MBSt, and Reviewer MFD5) comments that **our work is novel or new**. We acknowledge the positive comments such as "a new approach" (Reviewer Sh2w), "enhance model robustness" (Reviewer Sh2w), "provides theoretical analysis " (Reviewer Sh2w), "show generalization ability" (Reviewer Sh2w), "easy to follow" (Reviewer MBSt), "sound and innovative" (Reviewer MBSt), "theoretical proof" (Reviewer MBSt), "comprehensive experimental comparisons" (Reviewer MBSt), "significant topic" (Reviewer vRkX), "well-motivated" (Reviewer vRkX), "the effective presentation" (Reviewer vRkX), "extensive experiments" (Reviewer vRkX), "novel idea" (Reviewer MFD5), "technically sound" (Reviewer MFD5), and "effectiveness" (Reviewer MFD5). We have also responded to your concerns in the following. The figures are included in the pdf file for your reference.

Please let us know if you have any additional questions or concerns. We will try our best to address them.

Best regards,

the Authors

Meta-review: This paper proposes a new approach for out-of-distribution fluid dynamics modeling, PURE. It can learn time-evolving prompt embedding, using a graph ODE, for model adaptation, to overcome temporal distribution. Experiments show its effectiveness.

Firstly, the paper studies an important topic in fluid dynamics modeling. Secondly, the idea is new, and the techniques are sound. Finally, the paper provides extensive empirical results to support its claims. However, the presentation is not clear enough. For example, the studied problem is not well-formulated. And there are quite a few typos.

The authors should further improve the paper carefully!
Decision: Accept (poster)