PIMRL: Physics-Informed Multi-Scale Recurrent Learning for Spatiotemporal Prediction

Authors: Wan Han, Qi Wang, Hao Sun
Primary area: learning on time series and dynamical systems
Keywords: PDEs, physics encoding, data-driven modeling

Abstract:
Simulation of spatiotemporal systems governed by partial differential equations is widely applied in fields such as biology, chemistry, aerospace dynamics, and meteorology. Traditional numerical methods incur high computational costs due to the requirement of small time steps for accurate predictions. While machine learning has reduced these costs, long-term predictions remain challenged by error accumulation, particularly in scenarios with insufficient data or varying time scales, where stability and accuracy are compromised. Existing methods often neglect the effective utilization of multi-scale data, leading to suboptimal robustness in predictions. To address these issues, we propose a novel multi-scale learning framework, namely, the Physics-Informed Multi-Scale Recurrent Learning (PIMRL), to effectively leverage multi-scale data for spatiotemporal dynamics prediction. The PIMRL framework comprises two modules: the micro-scale module embeds physical knowledge into neural networks via pretraining, and the macro-scale module adopts a data-driven approach to learn the temporal evolution of physics in the latent space. Experimental results demonstrate that the PIMRL framework consistently achieves state-of-the-art performance across five benchmark datasets ranging from one to three dimensions, showing average improvements of over 9\% in both RMSE and MAE evaluation metrics, with maximum enhancements reaching up to 80\%.

Decision: reject
Reviewer scores: 8, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730012741089

thread_type:
official_review

<thread_object>
creator: reviewer_bvvk

<content>
summary:
This paper focuses on the temporal multi-scale property of spatiotemporal systems and proposes a multiscale recurrent learning framework, named PIMRL. In this model, the micro-scale module learns to embed multiscale data and the macro-scale module captures the data-driven evolution. PIMRL performs well in five different tasks.

After rebuttal: I think most of my concerns have been resolved. Thus, I have raised my score to 6 and changed the score of presentation and soundness.

strengths:
This paper focuses on an important problem: the temporal multiscale property, which is mainly overlooked in previous methods.

The experiments are relatively comprehensive. The authors test PIMRL in five typical tasks.

weaknesses:
1.	About the writing.

I believe that this paper needs a significant revision to ensure a clear presentation. Here are some examples.

(1)	Introduction: There are many important concepts without explanation. For instance, the authors claim that the micro-scale is “pretrained” in the abstract and introduction. However, in the method part, there are no descriptions of how this pretraining be implemented.

(2)	Related work: The authors fail to provide a clear organization of the related work. For example, ConvLSTM is just one classical (maybe old) baseline in spatiotemporal learning. There are extensive papers that weren’t included, such as PredRNN [1] or TrajGRU [2].

[1] PredRNN: a recurrent neural network for spatiotemporal predictive learning

[2] Deep Learning for Precipitation Nowcasting: A Benchmark and A New Model

(3)	Method: I think Eq. (2) and (3) do not provide a clear and right formalization for the overall framework. The descriptions of the prediction protocol are self-contradictory. Given that each micro module can conduct a \delta t stepsize transition and the macro module is \Delta t, why the output of Eq.(3) is just t+k\delta t. Besides, as they presented in Figure 2, I cannot tell which part is “history information” or if the micro-scale module is applied to the model prediction or not. A flowchart or pseudocode can be helpful.

The low quality of writing seriously affects the contribution of this paper.

2.	Compare with more advanced baselines.

As the authors mentioned in Lines 125-133, there are many advanced Neural Operators such as MWT and U-NO, which should be compared.

3.	About the novelty.

As the proposed method is more like a multiscale ensemble method, I cannot score high for the novelty.

questions:
Why did the authors name the micro module as “physics-informed”? Actually, I think the formalization of Eq. (4) is just a convolution neural work, which is far away from the conventional definition of “physics-informed neural networks”.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730202474885

thread_type:
official_review

<thread_object>
creator: reviewer_qubx

<content>
summary:
This paper introduces a data-driven model to simulate a spatiotemporal system. It proposes to leverage multi-scale data by embedding physical knowledge into a micro-scale module and employing a data-driven approach for its macro-scale module. 
The method is then tested on various fluid dynamics and reaction-diffusion systems equations, reaching impressive results on a challenging dataset.

strengths:
- the paper is beautifully illustrated, making the methods easy to read understand and the results easy to read and understand
- the proposed method outperforms the state of art on a variety of PDEs

weaknesses:
- Some method details lack clarity, especially regarding the physics operator.
- the related work and baselines seem to lack a few strong methods. For example, there is extensive literature on improving FNO (including some published here last year: [https://openreview.net/pdf?id=tmIiMPl4IPa]. Is there a reason for using FNO and not its extensions?
[https://arxiv.org/abs/2204.11127] also seems a like a strong baseline
- The ablation study lacks depth: more details on each module's contribution, the micro-scale module's pre-training, and some parameters would help understand the contribution of the method.

questions:
- It is not clear how the boundary conditions are encoded in each baseline: is boundary padding also used? If not it seems like an unfair advantage as it has access to additional information
- why is the Physics operator optional? Does it depend on the dataset? If not, it should be added to the ablation study.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730300073893

thread_type:
official_review

<thread_object>
creator: reviewer_m929

<content>
summary:
This work introduces a framework for handling multi-timescale data by modeling dynamics across two time scales. A micro module manages the high-frequency observations, while a macro module captures long-term dependencies. The micro module operates directly on physical states, leveraging PerCNN, whereas the macro module learns in a latent space and uses a ConvLSTM block for temporal propagation. Initially, the micro module is pretrained, and later fine-tuned jointly with the macro module using mean-squared error (MSE) supervision. The experiments appear to yield promising results.

strengths:
The paper presents a clear and well-formulated motivation that connects nicely with existing literature.
Experimental results are well-discussed and appear conclusive, demonstrating notable improvements over comparable methods.

weaknesses:
- The model seems to require aligned micro- and macro-scale training data, which may limit its applicability.
- While the model presentation is well-structured, its technical novelty is somewhat limited, as it could be considered a specific implementation of Rubanova et al. [1].

Would the authors consider working with unaligned datasets, such as those with only micro-scale or only macro-scale trajectories?

[1]: Rubanova et al., Latent ODEs for Irregularly-Sampled Time Series

questions:
1. Despite Figure 2, it remains challenging to understand the inference process. I suggest clarifying the figure or including a detailed description in the manuscript.
2. Is the RMSE computation performed on the micro- or macro-scale data?
3. Is model training robust to the number of training trajectories? Exploring the model’s scalability in relation to both the micro and macro modules would be insightful.
4. Does the model require consistency in the underlying physical parameters during training ? said otherwise, is it robust to out of distribution trajectories ?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730658140075

thread_type:
official_review

<thread_object>
creator: reviewer_nbme

<content>
summary:
This paper introduces physics informed multi-scale recurrent learning (PIMRL), which is a novel architecture for learning both micro and macro time scales commonly present in real world data. Specifically, PIMRL introduces a micro and macro architecture, along with corresponding training schemes, that provides an implicit bias for the network to learn both micro and macro timescales.


# Post rebuttal

The authors did a great job addressing my concerns. As such, I am raising my score.

strengths:
I want to preface this by saying that I'm not an expert in PDEs nor in deep-learning based approaches for solving PDEs. Nonetheless, I am a big fan of the proposed approach. The proposed architecture is something I have never seen before and is very nifty! I am also very impressed by the experiments section! Specifically, I LOVE that the authors look at multiple different metrics which allows one to look at the results from multiple different angles, which is very important.

weaknesses:
I again want to preface this section by saying that I am not an expert in this field. I hope that a non-expert's viewpoint will provide criticism that will make this paper stronger.

There are two big weaknesses that I think limit the paper from being truly great. The first is the writing. There are A LOT of interesting things in this paper that are not adequately explained nor properly described. Moreover, there is a lot of text and figures that are confusing and don't add much to the text; they could be removed to allow for the other aspects of the paper to shine. I will list my qualms with the writing below

- To start, the authors begin by stating that "direct numerical simulation requires a deep understanding of physics" but it is not clear what this means. Do the authors mean that one needs an understanding of physics to come up with PDEs that describe physical phenomena? This is independent of numerical simulating equations as a solver does not need to know what physical phenomena an equation is trying to describe. Moreover, since the name of the proposed approach is physics-informed multi-scale recurrent learning, does this not imply that a deep understanding of physics would also be required to effectively use PIMRL?

- Next, Figure 1 takes up a lot of space but does not add much. Personally, given its vagueness it confused me on what PIMRL is. Figure 2 does a much better job explaining the method. I would recommend removing figure 1.

- A big part of PIMRL is PerCNN, as the micro-scale architecture is exactly the PerCNN architecture. To me this is not an issue as science is built upon previous works, but I would have preferred the authors explicitly stated this, i.e., 'We introduce PIMRL, which combines the micro-scale power of PerCNN, with a novel macro scale architecture to...". Next, it is also not clear why PerCNN needed an extension in the first place (this evidence is lacking both in the text and in the experiments section). In the related works section, the authors state the following about PerCNN "However, the model suffers from error accumulation during long rollout predictions. To address this issue, a straightforward approach is to reduce the frequency of iterations by increasing the time stepping. Nonetheless, for hard-embedded methods like PeRCNN, accuracy is limited by the time stepping, making simple modifications unfeasible". This doesn't make any sense to me. In PerCNN, one can choose both the number of iterations in a the $\Pi$ block as well as the time step, $\delta t$; thus, it seems it would be rather straightforward to train PerCNN on macro-scale data. Moreover, the comparison to PerCNN doesn't seem fair. First, similar to what the authors did for FNO, it seems straight forward to also include a PerCNN coarse and fine scale. One could even just train one PerCNN on both the micro and macro scale data, where $k$ is dataset dependent. 

- To me the most interesting part of the approach is how the macro and micro modules interact. Sadly, this is not given time to shine in the paper! For instance, why was this interaction between the macro and micro module chosen? What role does the mirco-module serve (i.e., does it serve as error correction for the initial condition?)? There are so many interesting design choices here that I think are fascinating but sadly are missing.

- In the paper, the authors use the term physical embedding where they state "... physical embedding methods, where explicit physics knowledge is embedded into the model to fully leverage physical principles...". They also state that PIMRL uses this but the entire approach is data-driven. If by physical embedding they mean the *Optional* physical-based FD conv then they should explicitly state this. Also, I think the fact this is *optional* weakens the statement considerably.


The next weakness is the experiments section. While the results are compelling and I love that they look at the results from multiple different angles, I think there are some major things missing.
- There are no details on how the datasets are constructed. What solver was used? How were the initial conditions chosen?
- The authors state that they create a micro and macro dataset for training PIMRL. For the other baselines, it seems like they were either trained on the micro or macro dataset. On a first read, it seems like PIMRL was trained on more data. Was the dataset size equalized for the baselines?
- Error bars are missing in the error propagation plots and Table 2!
- The ablation study is missing a lot of details. For instance, what does the NoConnect model mean? Is it just purely the micro module or the macro module?
- Without knowing the solver, the inference time figure is a little underwhelming.

questions:
I think most of my questions were listed in the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732255890800

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General reply

comment:
Dear Reviewers:

We deeply appreciate the constructive comments from you. Thank you for the time and effort dedicated to reviewing our paper. **Comprehensive revisions and adjustments** (indicated in red color) have also been made in the revised paper (please see ***the updated .pdf file***). In particular, we have thoroughly proofread our paper, corrected typos and grammar mistakes, and re-organized our writing to improve the clarity of the paper.

We are pleased that the reviewers recognized our work. In particular, we thank the reviewers for recognizing the ***thorough experiments*** (Nbme, M929, QUbX, BVvK), ***novelty*** (Nbme, M929) and ***beautifully illustrated paper*** (M929, QUbX).

We have summarized a detailed reply to common questions and addressed other concerns in each individual rebuttal. 

**Q1: New Baselines.**

**Reply:** Following your suggestion regarding the use of more advanced baseline models, we supplemented the experimental results for the **U-NO** [1], **F-FNO** [2] and **MWT** [3] models in the FN case, enhancing our comparative analysis. The results are shown in **Table A** below, which is also provided in Table S3 (Appendix H, Page 17) in the revised paper.

**Table A: The Results of U-NO, F-FNO, MWT and FNO in the FN Case with our method**
|Metrics|U-NO|F-FNO|MWT|FNO|PIMRL|
|-|-|-|-|-|-|
|RMSE|0.3675|0.2280|0.3494|0.1878|**0.1349**|
|MAE|0.1465|0.1350|0.2228|0.1634|**0.0990**|

***References:***

[1] Rahman et al., U-NO: U-shaped neural operators. arXiv, 2022.

[2] Tran et al., Factorized Fourier Neural Operators. ICLR, 2023.

[3] Gupta et al., Multiwavelet-based operator learning for differential equations. NeurIPS, 2021.

**Q2: Details of PIMRL.**

**Reply:** To address these concerns and clarify ambiguities, we have provided a detailed exposition of the framework's application scenarios, major contributions, and procedures for model training and inference.

1. **Application scenarios:** 
    - The PIMRL framework is tailored for multi-scale data akin to that acquired via multi-scale Burst sampling, ensuring efficiency while retaining detailed physical variations. PIMRL effectively exploits the data to learn fine-grained physical patterns and to deliver rapid and precise predictions.
2. **Our main contributions:** 
    - We proposed a new PIMRL framework that effectively leverages information from multi-scale data for long-term spatiotemporal dynamics prediction. The concept of reducing the accumulation of errors is achieved through the integration of the macro-scale and micro-scale modules.
    - We designed a novel message passing mechanism between micro- and macro-scale modules, which effectively transmits physical information, enhances the micro-module's correction capability, and reduces the number of micro-scale rollout iterations through the macro-scale module.
3. **Training workflow of PIMRL:**
    - First, pretrain the micro module using fine-grained data, independent of the macro module. This step maximizes the micro module's learning capacity and ensures effective capture of physical information.
    - Next, the micro module, initialized with pre-trained weights, participates in the training of the entire PIMRL framework, which utilizes coarse-grained data.
4. **Inference workflow of PIMRL:** 
    - Firstly, the micro-module loop is a simple autoregressive process with time step $\delta t$, where the output at the previous time step serves as the input for the next time step.
    - Secondly, the macro-module loop performs self-cycles. When the micro-module is involved in the prediction, for every $k$ steps of micro-module with $\delta t$, the final output of micro-module is passed to the macro-module, and at this point, the output from the macro-module serves as the output of the entire PIMRL model. When the micro-module is not involved in the prediction, the macro-module loop is a simple autoregressive process with time step $\Delta t$. 
    - Finally, there is a PIMRL loop that operates in conjunction with the macro-module loop. After every $N$ cycles of the macro-module loops, the micro-module stops participating in the prediction, and the macro-module performs $N$ steps of autoregressive prediction. This completes a total of $2N$ cycles. Each output from the macro-module during these $2N$ cycles serves as the output of PIMRL.

The application scenarios are discussed in the Introduction section (Page 1, lines 50-56) in the revised paper, while the main contributions are outlined in the same section (Section 1, Page 2, lines 87-98). The training workflow is detailed in the Training Strategies subsection of Methodology (Section 3.4, Page 4, lines 209-248), and the inference workflow is covered in the Network Architecture subsection of Methodology (Section 3.3, Page 4, lines 179-206).

Thank you very much.

Best regards,

The Authors of the Paper
</content>
</thread_object>

</thread>
<thread>
creation date:
1733215207502

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Special thanks to all reviewers

comment:
Dear Reviewers,

We would like to extend our sincerest gratitude to you for your thorough and insightful reviews. The discussion has been both productive and fruitful, significantly contributing to the overall quality and clarity of our paper.

Once again, thank you very much for your time, effort, and insightful comments. 

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a surrogate model for solving PDEs designed for multi-timescale data, modeling dynamics across two-time scales by leveraging aligned data. A “micro-module,” which implements an existing physics-informed model (PerCNN), captures temporal data at a high resolution, while a “macro-module” operates autoregressively at a lower time resolution and interacts with the micro-module. The micro-module is first pretrained, after which the entire system is trained end-to-end. A key motivation for this design is to reduce the accumulation of errors typically associated with neural autoregressive surrogates. Experiments are conducted on the KdV equation, 2D Burgers' equation, and a series of reaction-diffusion equations.
The reviewers appreciated the motivations but raised several concerns related to the clarity of the technical description, the lack of appropriate baselines and analyses, and the incremental nature of the technical contribution. The authors made significant efforts during the rebuttal, including adding new comparisons and providing further explanations. However, discussions revealed that some of the initial misunderstandings stemmed from the imprecise technical description and the lack of detailed explanation regarding the performance of the proposed methods.

Even with the revisions, the paper still lacks clarity and precision, and some reviewers remain concerned about the validity and relevance of the experiments. Furthermore, the discussions highlighted potential flaws in the experimental comparisons. Overall, given the significant changes made and the remaining uncertainties, it would be better to revise and resubmit. The authors should clarify their positioning, specifically that the paper addresses IC/BC problems with prior knowledge of the underlying PDE, and ensure comparisons are made within a relevant setting. Direct comparisons with purely data-driven methods in the IC/BC setting, with no prior knowledge of the PDE, might be inappropriate.