SPARK: Physics-Guided Quantitative Augmentation for Dynamical System Modeling

Authors: Fan Xu, Penghao Zhao, Zhipeng Xu, XINLIANG ZHOU, Xinping Yi, Qingsong Wen, Hao Wu, Kun Wang
Primary area: learning on time series and dynamical systems
Keywords: dynamical system, augmentation

Abstract:
In dynamical system modeling, traditional numerical methods have a solid theoretical foundation but are limited by high computational costs and sensitivity to initial conditions. Current data-driven approaches use deep learning models to capture complex spatiotemporal features, but they rely heavily on large amounts of data and assume a stable data distribution, making them ineffective against data scarcity and distribution shifts. To address these challenges, we propose SPARK, a physics-guided quantized augmentation plugin. SPARK integrates boundary information and physical parameters, using a reconstruction autoencoder to build a physics-rich discrete memory bank for data compression. It then enhances selected samples for downstream tasks with this pre-trained memory bank. SPARK then utilizes an attention mechanism to model historical observations and combines fourier-enhanced graph ODE to efficiently predict long-term dynamical systems, enhancing robustness and adaptability to complex physical environments. Extensive experiments on benchmark datasets show that our approach significantly outperforms various baseline methods in handling distribution shifts and data scarcity.

Decision: reject
Reviewer scores: 8, 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1730439859769

thread_type:
official_review

<thread_object>
creator: reviewer_ph3k

<content>
summary:
This paper proposes SPARK to address the challenges of data scarcity and distribution shifts in dynamical system modeling. SPARK integrates boundary information and physical parameters by using an autoencoder, and then a pre-trained memory bank is obtained. It further combines Fourier-enhanced graph ODE to efficiently predict long-term dynamical systems. The experimental results have demonstrated the superiority of the proposed method against the baseline models across many dynamical systems under distribution shifts and limited data conditions.

strengths:
- This paper presents an interesting idea for handling data scarcity and OOD, which are important topics in scientific machine learning. 

- This paper is well-written and has a detailed presentation of methods, experimental setup, and results discussion.

- This paper has tested multiple challenging datasets, such as ERA5 and 3D systems.

weaknesses:
- The motivation for using each component in SPARK can be further clarified. The paper will benefit from discussing the interconnection between each network component.  

- It would also be good to have ablation studies on incorporated physics. The authors may consider reducing physical information (i.e., boundary information and physical parameters) for pre-training. Then, we can see the contribution of each physical component.

questions:
- On Page 6, for RQ2, could you be more specific on what challenging tasks?

- What is the setup for OOD experiments?

- How do you compute PSNR and SSIM for scientific data? Image data has a fixed range of [0,255] but scientific data doesn’t. 

- Energy Spectrum is a common metric for fluid dynamics. Is it also commonly used for reaction-diffusion equations? How does this paper compute the energy spectrum?


- Some minor typos: 

    - On Page 2, “effectively long-term prediction” should be “effective …”.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730537189234

thread_type:
official_review

<thread_object>
creator: reviewer_bjtd

<content>
summary:
The paper introduces SPARK, a physics-guided augmentation framework for modeling dynamical systems that overcomes the limitations of traditional numerical and data-driven methods. By incorporating a unique compression and augmentation plugin, along with an attention mechanism and Fourier-enhanced graph ODE, SPARK improves model generalization and robustness, especially in data-scarce situations and distribution shifts. Experimental results highlight SPARK's strong performance in accurately predicting complex spatiotemporal dynamics, particularly in challenging cases like sea ice evolution, effectively capturing intricate physical phenomena.

strengths:
1. By incorporating boundary information and physical parameters, SPARK enhances the model's ability to generalize across different physical scenarios, which is crucial for real-world applications.
2. The paper provides extensive experimental results across various benchmark datasets, demonstrating SPARK's superior performance compared to existing models, particularly in handling out-of-distribution scenarios.

weaknesses:
1. The symbols and formulas appear to be somewhat disorganized, which makes it difficult for readers to understand the meaning.  Clear definitions and a more structured presentation of the equations would greatly enhance the paper's accessibility and overall readability.
2. The lack of novelty. This paper claims to be the first to use physics-guided compression and augmentation. But there has been a paper [1] doing like this. The techniques of the two papers are very similar, including : (1) using VQ-VAE to compress information (2) augmenting training set by the top-K discrete embeddings.
3. The proposed methodology, may be complex to implement in practice. The paper could provide more guidance or examples on how to effectively apply SPARK in different contexts.

[1] Wu, Hao, et al. "BeamVQ: Aligning Space-Time Forecasting Model via Self-training on Physics-aware Metrics." arXiv preprint arXiv:2405.17051 (2024).

questions:
1. In line 163, it needs references for those methods which simply concatenate boundary information with node features.
2. What does boundary information refer to? Give some examples please.
3. In abstract, what's the meaning of "stable data distribution"? Provide explanations about it and why does it can cause ineffectiveness of data scarcity and distribution shifts.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730597151923

thread_type:
official_review

<thread_object>
creator: reviewer_89br

<content>
summary:
Data-driven methods for dynamical systems often face distribution shift challenges. To tackle this, this paper proposes SPARK, a physics-guided plugin to address both environmental distribution shift (due to changes in boundary conditions and physical parameters) and temporal distribution shift.
SPARK achieves this by incorporating the boundary information and physical parameters into a discrete memory bank constructed through solution reconstructions.
By embedding these physical priors, the memory bank can then be used to augment data samples in downstream tasks, thereby increasing model generalizability.
To handle the temporal distribution shift, SPARK encodes historical information into initial states through attention and uses Fourier-enhanced graph ODE for long-term prediction.
In the end, the paper evaluates SPARK on several benchmarks.

strengths:
1. The idea of increasing model generalizability by augmenting data in downstream tasks with a pre-trained memory bank that contains boundary information and physical parameters is nice.
2. The paper evaluates the method on a good number of benchmarks.

weaknesses:
1. The degree of originality is not high. It shares quite some similarities with the DGODE model in (Prometheus by Wu et al. 2024, cited by the paper), which proposed the idea of "codebank" to include the environmental factor for OOD and graph ODE for future predictions.
2. The algorithm is not clearly presented. The paper presented reasonable ideas but without enough technical details to tell a clear story. 
Mathematical notation is not clearly defined, which makes it hard to follow the method. For example, how is the "real boundary" ("p^{boun}")represented? Is it a list of spatial coordinates of discrete boundary nodes? Time index does not make sense in section 3.3. For example, T in equation 8 is the length of history observations but then it is also used in loss function in equation 11 to represent the number of future predictive steps, which is confusing. Index notation does not make sense in the pretraining loss equation (6).
3. It's unclear how the discrete memory bank is built. What are the e_i in E and how are they constructed ?

questions:
1. How are the with and without OOD datasets constructed in the experiments? Is there an explanation for why SPARK achieved better performance on OOD cases even than other models did on non-OOD cases? 
2. Given the big accuracy difference, what is the training cost comparison?
3. What are the training details, e.g., model architecture, optimizer, training devices?  Is boundary information injected at two places, i.e., through node features and directly through the boundary latent vector B?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730746676442

thread_type:
official_review

<thread_object>
creator: reviewer_rucb

<content>
summary:
The paper proposes a new approach to spatiotemporal surrogate modeling. Their approach aims to target some of the limitations of data-driven models as they pertain to distribution shift. The framework, SPARK, combines physics-guided data augmentation and compression to enhance generalization. Key architectural innovations include a discrete memory bank for storing previous physical samples, physical prior and BC incorporation with graph neural nets, and a curriculum learning strategy to incorporate augmented data progressively. SPARK's superior performance is then evaluated on a relatively large suite of benchmark datasets.

strengths:
- The incorporation of a data bank to progressively generate augmented samples is an intelligent design choice, enabling the model to store physical information for use in OOD prediction. Combined with the work on storing physical parameters and boundary conditions, the authors have presented many useful tricks for physical surrogate modeling. 
- Extensive tests across numerous datasets and benchmarks conclusively demonstrate the superior performance of this training strategy, particularly in OOD scenarios. The benchmarks are extensive as well, and helpful in framing the work.
- The theoretical framework is helpful, providing solid support for the model's architecture and approach.
- The paper is well-written and well-presented.

weaknesses:
- Many new architectural design choices are proposed (handling of physical parameters, boundary conditions, data banks, curriculum learning, etc.). However, it is unclear how much each strategy contributes to the success of the model, and some ablation studies would be useful.

questions:
- Can the data bank be used for direct retrieval-augmentation?
- How well does the model perform in the very low-data regime (just a few samples for transfer learning)?
- Have the authors explore generalization to 1-D or 3-D data at all?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732184390994

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear Reviewers,

Thanks for your time and valuable feedbacks. We acknowledge three reviewers (Reviewer Rucb, Reviewer 89BR, and Reviewer pH3K) comment that the work is novel or nice. We acknowledge the positive comments such as "a new approach" (Reviewer Rucb), "superior performance" (Reviewer Rucb), "an intelligent design choice" (Reviewer Rucb), "many useful tricks" (Reviewer Rucb), "benchmarks are extensive" (Reviewer Rucb), "the theoretical framework is helpful" (Reviewer Rucb), "well-written and well-presented", "the idea is nice" (Reviewer 89BR), "a good number of benchmarks" (Reviewer 89BR), "crucial for real-world applications" (Reviewer bjTD), "extensive experimental results" (Reviewer bjTD), "an interesting idea" (Reviewer pH3K), "important topics" (Reviewer pH3K), "well-written and has a detailed presentation" (Reviewer pH3K). We have also responded to your concerns in the following.

Please let us know if you have any additional questions or concerns. We will try our best to address them.

Best regards,

the Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732366009730

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Response

comment:
Dear Reviewers,

Thank you for your thorough and insightful reviews. We sincerely appreciate your feedback, which has significantly enhanced our paper! Below, we summarize the key concerns raised and our corresponding responses:

- **Implementation details of SPARK** (Reviewer 89BR, bjTD, pH3K)

    We have presented our model details in tabular form and used the Navier-Stokes equation as an example to show dimensional changes and related experiments.

- **Problem about novelty** (Reviewer 89BR, bjTD)

    We have clarified the differences between our SPARK and the mentioned models. Furthermore, we have conducted comprehensive experiments to verify this.

- **Contribution of each component or strategy** (Reviewer Rucb, pH3K)

    We have added ablation experiments to confirm the effectiveness of each component or strategy. 

- **Details of boundary information and OOD experimental setup** (Reviewer 89BR, bjTD, pH3K)

    We have included a more detailed description of the boundary information, and an intuitive figure is shown in Appendix I. For OOD experimental setup, we have provided a clear table to present.

Once again, we are truly grateful for your valuable feedback and are happy to address any further concerns or questions!

Sincerely,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces SPARK, a physics-guided quantized augmentation plugin for dynamical system modeling. While traditional methods are computationally expensive and sensitive to initial conditions, and current deep learning models depend on large datasets and stable distributions, SPARK overcomes these issues. It uses a reconstruction autoencoder to build a physics-rich memory bank and applies attention mechanisms and Fourier-enhanced graph ODEs for efficient long-term predictions. Extensive experiments show that SPARK outperforms baseline methods in handling data scarcity and distribution shifts.

Strengths:

The incorporation of a data bank to progressively generate augmented samples is an insightful design choice.

The paper presents extensive experimental results across a variety of benchmark datasets.

Weaknesses:

The degree of originality and novelty is relatively low when compared to methods like DGODE and BeamVQ.

The motivation behind the approach and the algorithm itself are not clearly articulated.

It is unclear how much each individual strategy contributes to the overall success of the model. Some ablation studies would provide valuable insights here.


While the proposed approach is interesting and the experimental results cover a broad range of benchmarks, the novelty of the methodology is only marginally significant. Long-term prediction is a common metric in dynamical system modeling, yet the experiments in this paper are limited to 10-50 steps. For PDE modeling, it would be beneficial to evaluate rollout errors for more than 100 ane even 1000 steps [1]. Given these concerns, I suggest a borderline reject but encourage the authors to address the reviewers' feedback and resubmit to a top conference.