Title: PURE: Prompt Evolution with Graph ODE for Out-of-distribution Fluid Dynamics Modeling
Venue: NeurIPS 2024
Accepted: yes
Keywords: AI for Science
Abstract: This work studies the problem of out-of-distribution fluid dynamics modeling. Previous works usually design effective neural operators to learn from mesh-based data structures. However, in real-world applications, they would suffer from distribution shifts from the variance of system parameters and 
temporal evolution of the dynamical system. In this paper, we propose a novel approach named \underline{P}rompt Evol\underline{u}tion with G\underline{r}aph OD\underline{E} (\method{}) for out-of-distribution fluid dynamics modeling. The core of our \method{} is to learn time-evolving prompts using a graph ODE to adapt spatio-temporal forecasting models to different scenarios. In particular, our \method{} first learns from historical observations and system parameters in the frequency domain to explore multi-view context information, which could effectively initialize prompt embeddings. More importantly, we incorporate the interpolation of observation sequences into a graph ODE, which can capture the temporal evolution of prompt embeddings for model adaptation. These time-evolving prompt embeddings are then incorporated into basic forecasting models to overcome temporal distribution shifts. We also minimize the mutual information between prompt embeddings and observation embeddings to enhance the robustness of our model to different distributions. Extensive experiments on various benchmark datasets validate the superiority of the proposed \method{} in comparison to various baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new approach called Prompt Evolution with Graph ODE (PURE) for non-distributed fluid dynamics modeling. PURE first learns from historical observations and system parameters in the frequency domain to explore multi-view contextual information, which can efficiently initialize the cue embedding. Interpolations of the observation sequences are then merged into the graph ODE so that the time evolution of the model-adaptive cue embeddings can be captured. These time-evolving cue embeddings are then incorporated into the underlying predictive model to overcome spatio-temporal distributional variations. In addition the paper minimizes the mutual information between the cue embeddings and the observation embeddings to enhance the robustness of the model to different distributions. Finally, extensive experiments conducted on various kinds of benchmark datasets validate the superiority of the proposed PURE compared to various baselines.\n\nSTRENGTHS:\n1. The idea of the paper is novel. It is the first to link prompt learning to dynamic system modeling for out-of-distribution problems. \n \n2. This paper is technically sound. PURE first learns initialized prompt embeddings from historical observations and system parameters, and then employs a graph ODE with interpolated observation sequences to capture the continuous evolution of their model adaptation under out-of-distribution changes.  \n\n3. The experimental results show the effectiveness of PURE in different challenging environments.\n\nWEAKNESSES:\n1. The contribution of the proposed method in dealing with the OOD problem needs to be further clarified since the advantages of PURE over the previous efforts, such as Refs. [7, 67, 14, 72], etc., to address the OOD problem are not listed.\n\n2. The writing of the paper needs to be improved. Some of the symbols in the method description section are not defined, e.g., what do P and N in Equation 9 refer to?\n\n3. The experiment is not comprehensive enough. (a) The reasons for selecting baselines are not explained. Data augmentation [66, 7], invariant feature learning [39, 69, 38], adversarial training [67, 7], and domain adaptation [32, 14] are mentioned in the paper in related work for solving the OOD problem, but they are not be compared as baselines in the experiment. (b) The experiments in this paper do not state whether noisy data are considered. (c) The authors just give a brief description of the results without analyzing the reasons behind the high performance.\n\nQUESTIONS:\n1.  What are the advantages of PURE over previous efforts to address OOD? \n\n2. Some of the symbols in the method description section are not defined, e.g., what do P and N in Equation 9 refer to?\n\nLIMITATIONS:\nThe authors list limitations in the appendix but do not mention them in the main text. It is recommended that the author make a description in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The paper aims to improve the out-of-distribution (OOD) generalization of fluid dynamics modeling.\n\n- Two types of OOD scenarios are targeted: OOD across different systems and OOD within the same system across different timestamps.\n\n- The paper proposes a framework named PURE, composed of modules including:\n    - Multi-view Context Exploration, which explores spatio-temporal data using both the attention mechanism and the frequency domain;\n    - Time-evolving Prompt Learning, which incorporates the interpolation of observation sequences;\n    - Model Adaptation with Prompt Embeddings, which leverages time-evolving prompts to mitigate temporal distribution shifts.\n\n- Extensive experiments on a range of fluid dynamics datasets support the claim.\n\nSTRENGTHS:\n- Significant topic: OOD generalization in fluid dynamics modeling.\n\n- Well-motivated, as OOD generalization is a crucial challenge in this field.\n\n- The presentation effectively delivers the message.\n\n- Extensive experiments have been conducted.\n\nWEAKNESSES:\n- My major concern with the paper is that the OOD challenge in dynamics modeling is not well-formulated. The paper describes the OOD scenario verbally as \"*different dynamical systems could involve different parameters in underlying rules*\" and \"*during long-term auto-regressive forecasting, the input data distribution could vary hugely during temporal evolution,*\" which is straightforward and easy to understand. However, the mathematical formulation of these scenarios is absent. This formulation should be the foundational basis of the topic, as we need to clearly define the problem before addressing it.\n\n- Given the lack of mathematical formulation of the challenge, I find myself lost in the proposed approach section, unsure of the necessity for specific components. While I understand the function of each component, I cannot see why it is needed or which gaps it aims to bridge in the absent mathematical framework.\n\n- Why is the proposed method termed \"prompt\"? Is there a connection to prompt tuning in large language models?\n\n- How do you quantify the distribution shift in dynamics modeling? Can you rank the 'difficulty level' of OOD generalization in your experiments and analyze in which scenarios your method stands out and why?\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper pioneers the connection of prompt learning with dynamical system modeling to address the challenge of out-of-distribution shifts. The proposed PURE method initializes prompt embeddings by learning from historical observations and system parameters.\n\nSTRENGTHS:\n1.The paper is easy to follow. \n2.The proposed method is sound and innovative.\n3. The authors provide theoretical proof and show comprehensive experimental comparisons.\n\nWEAKNESSES:\n1. Some results may be incorrectly labeled as suboptimal in table, and there are errors in the use of some symbols.\n2. The explanation of the experimental results is not detailed enough, making some experiments difficult to understand.\n3. The proposed method is aimed at OOD (Out-Of-Distribution), but the experiments lack comparison and discussion with methods specifically targeting OOD, such as [1] and [2].\nReference:\n[1] Kirchmeyer, Matthieu, et al. \"Generalizing to new physical systems via context-informed dynamics model.\" International Conference on Machine Learning. PMLR, 2022.\n[2] Yin, Yuan, et al. \"LEADS: Learning dynamical systems that generalize across environments.\" Advances in Neural Information Processing Systems 34 (2021): 7561-7573.\n\nQUESTIONS:\nQ1. There might be misuses of symbols in the paper, such as, Change \"xi\" to \"si\" in line 65, Change \"xq\" to \"xq\" in line 96, \"Zero-shot Experiments\" and \"Generalization Experiments\" in line 215 should be treated equally and placed on separate lines.\nQ2. Is there an issue with the second-best data in Table 2? For example, in the column w/OOD of SPHERICAL-SWE, the second-best should be DGPDE 0.0028. The corresponding improvement results also need to be modified.\nQ3. What does the clustering in Figure 5 represent? Could you provide a detailed explanation?\nQ4. The paper utilizes mutual information to decouple different prompt embeddings and observation embeddings, reducing the sensitivity of observation embeddings to different distributions. However, I don't quite understand the purpose of decoupling. Observational embeddings are related to the environment, and prompt embeddings are related to the environment as well; they are inherently correlated. Please provide further explanation.\n\nLIMITATIONS:\nThis method does not apply to real-world scenarios, such as rigid dynamics modeling and traffic flow forecasting.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
The paper proposes a graph ODE-based approach for OOD fluid dynamics modeling. PURE aims to learn time-evolving prompts via graph ODE for adaptation of spatio-temporal forecasting models on OOD scenarios. To address temporal distribution shifts, the interpolation of obersvation sequences are combined into graph ODE framework to learn evolution of prompt embeddings.

STRENGTHS:
- The paper proposes a new approach that connects prompt learning and dynamical system modeling which addresses OOD shifts.
- By learning time-evolving prompts that adapt to changes in system parameters and temporal evolution, the approach can enhance model robustness.
- The paper provides theoretical analysis on incorporating observations during evolution.
- Experiments on diverse benchmarks show generalization ability to OOD and different prediction length.

WEAKNESSES:
As I am not an expert in this field, I am unable to find major concerns or weakness of the approach.
- As the method is based on attention, the proposed approach may have limited scalability and take long computation time. Is there a comparison on these with the previous works?

QUESTIONS:
Please address the questions in the Weaknesses.

LIMITATIONS:
The limitations are explained in Appendix I.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13