MaD-Scientist: AI-based Scientist solving Convection-Diffusion-Reaction Equations Using Massive PINN-Based Prior Data

Authors: Mingu Kang, Dongseok Lee, Woojin Cho, Jaehyeon Park, Kookjin Lee, Anthony Gruber, Youngjoon Hong, Noseong Park
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: in-context learning, scientific foundation model, zero-shot, PINN-prior

Abstract:
Large language models (LLMs), like ChatGPT, have shown that even trained with noisy prior data,  they can generalize effectively to new tasks through in-context learning (ICL) and pre-training techniques.
Motivated by this, we explore whether a similar approach can be applied to scientific foundation models (SFMs). Our methodology is structured as follows: (i) we collect low-cost physics-informed neural network (PINN)-based approximated prior data in the form of solutions to partial differential equations (PDEs) constructed through an arbitrary linear combination of mathematical dictionaries; (ii) we utilize Transformer architectures with self and cross-attention mechanisms to predict PDE solutions without knowledge of the governing equations in a zero-shot setting; (iii) we provide experimental evidence on the one-dimensional convection-diffusion-reaction equation, which demonstrate that pre-training remains robust even with approximated prior data, with only marginal impacts on test accuracy. Notably, this finding opens the path to pre-training SFMs with realistic, low-cost data instead of (or in conjunction with) numerical high-cost data. These results support the conjecture that SFMs can improve in a manner similar to LLMs, where fully cleaning the vast set of sentences crawled from the Internet is nearly impossible.

Decision: reject
Reviewer scores: 6, 8, 5, 5

Reviews and discussion:
<thread>
creation date:
1730316718711

thread_type:
official_review

<thread_object>
creator: reviewer_qaiw

<content>
summary:
This paper introduces an in-context learning architecture for scientific foundation models. The proposed method incorporates Bayesian inference into the prediction by pre-training on PINN-based low-cost and noisy approximated data, which is referred to as the PINN-prior. The PINN is initialized as a general PDE with derivatives up to a certain degree. Initializing the PINN with a general PDE helps to overcome the fact that the underlying governing equations of physical systems are often unknown; instead, only the observations of the underlying systems are available. At the base, the method uses a transformer architecture, where a self-attention mechanism is used to capture relationships among known data points, and a cross-attention mechanism is used to extrapolate and infer solutions for unseen points without fine-tuning.

strengths:
1. This paper integrates several non-trivial concepts of the scientific foundation model, approximated prior data, Bayesian inference, and pre-training that are already well-established in scientific machine learning. 
2. The concept of using PDE dictionaries in the PINN-prior for pre-training is original. 
3. This paper tries to address the challenges of the inherent noisiness of the data used for training.

weaknesses:
According to me, some parts of the paper need further clarification. Please see the comments below:
1. The underlying concept of using approximated prior data is a concept that has been introduced previously in scientific machine learning. This concept already exists in multi-fidelity learning, where a low-fidelity solver is first trained on a large number of available low-fidelity data, and the approximate predictions of low-fidelity solver are used in conjunction with the small high-fidelity data to train a fast high-fidelity emulator [1,2]. Except for the difference between the purely data-driven nature of the multi-fidelity learning methods and the partial physics-constrained nature of the proposed framework, I do not see any significant improvement.  Further clarification is required to understand the novelty of the proposed method.
2. The authors mention the "scientific foundation model" at several places in the paper. The purpose of the foundation models is to pre-train on a large number of datasets, which in this case is analogous to multiple PDEs, and then efficiently fine-tune or directly predict on unseen PDEs without sacrificing prediction accuracy. However, the authors consider only a simple CDR equation for pre-training and special cases of the CDR equation like convection, diffusion, and reaction equations for testing. This raises concerns about whether the methods can genuinely aid in foundation modeling, highlighting the need for more rigorous case studies to properly assess the effectiveness of the proposed PINN-prior framework. To be able to clearly appreciate the method's potential as a foundation model, in addition to the CDR equation, it is necessary to simultaneously pre-train on a few of more complex PDEs like the Kuramoto–Sivashinsky equation, Schrodinger equation, Korteweg–De Vries equation, Navier-Stokes equation, Burgers' equation, etc. and test on remaining PDEs.
3. In the PINN dictionary, the authors consider the exact derivative terms of the CDR equation. This is unlikely to be available in a practical scenario, which is also mentioned by the authors in the motivation. Thus, the performance of the PRIOR in the presence of a large number of dictionary functions needs to be inspected to understand further the potential of the proposed framework for scientific foundation modeling. The authors need to conduct additional experiments with a larger and more diverse set of dictionary functions to better simulate real-world scenarios where exact terms may not be known.
4. Many scientific foundation models have already been developed for pre-training and testing on multiple physical systems [3-6]. However, the authors do not mention them in this paper. Given the existing literature on foundation models, the authors should consider comparisons with a few of the existing scientific foundation models. This will further improve the contents of the paper.

[1] Meng, Xuhui, and George Em Karniadakis. "A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse PDE problems." Journal of Computational Physics 401 (2020): 109020.\
[2] Howard, Amanda A., et al. "Multifidelity deep operator networks for data-driven and physics-informed problems." Journal of Computational Physics 493 (2023): 112462.\
[3] McCabe, Michael, et al. "Multiple physics pretraining for physical surrogate models." arXiv preprint arXiv:2310.02994 (2023).\
[4] Tripura, Tapas, and Souvik Chakraborty. "A foundational neural operator that continuously learns without forgetting." arXiv preprint arXiv:2310.18885 (2023).\
[5] Hao, Zhongkai, et al. "Dpot: Auto-regressive denoising operator transformer for large-scale pde pre-training." arXiv preprint arXiv:2403.03542 (2024).\
[6] Herde, Maximilian, et al. "Poseidon: Efficient Foundation Models for PDEs." arXiv preprint arXiv:2405.19101 (2024).

questions:
Please see below questions on the paper content:
1. line 078. The authors claim that their goal is "to predict solutions from observed quantities, such as velocity and pressure, without relying on governing equations". However, any data-driven ML model learns solutions from observed quantities without relying on governing physics. Therefore, why use a PINN with approximate PDE dictionaries while robust methods like neural operators are already there? 
2. line 269. In Eq. (12), define $\tilde{u}(x_T^{(j)},t_T^{(j)})$. Is it the same as $\tilde{u}(\alpha)$, or does it denote the observed solution field?
3. line 271. "...our model only needs observed quantities". The correct derivatives, as in the original CDR equation, are considered in Eq. (8). Thus, this sentence seems to be vague to me. Consider a large number of derivatives and their polynomials in Eq. (8) and then generate prior data to see the performance of the proposed method. 
4. line 302. it is written $D$ requires the solution u. Is $u$ the true observed quantity or the PINN-approximated data?
5. line 323. What value of $m$ is taken in the examples?
6. line 372. Is it supposed to be section 4.2.1?
7. Table 3. For 100% PINN-prior, the error in the convection equation is as high as 16%. This raises questions about the framework's ability to generalize effectively over a large number of unseen systems. 
8. Is there any reason similar to the above why the "TIME DOMAIN EXTRAPOLATION" study is not carried out on diffusion and reaction equations?
9. Fig. 3. This is a continuation of comment 7. See that the time extrapolation results for the convection equation is more than 50%, which again raises questions about the framework's generalization ability. 
10. Fig. 4. The solution field on the right is too smooth to draw a conclusion about the extrapolation ability.
11. line 480. Why is the $\beta$ = $\nu$ = 0 taken? I think a non-zero value will make the problem more challenging.
12. line 483. It is not fair to say "it can generalize to unseen PDEs" since the model is pre-trained on the CDR PDE, which is nothing but a mixture of convection, diffusion, and reaction.
13. line 487. How many training samples are taken? 
14. Table 4. A central goal of the paper is to achieve zero-shot inference for predicting PDE solutions. However, when the model is pre-trained on a mixture of reactions but tested on different reactions separately, the error increases significantly beyond 6% in most of the cases. This indicates that the proposed framework may also need fine-tuning or few-shot learning.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730405535388

thread_type:
official_review

<thread_object>
creator: reviewer_wwxl

<content>
summary:
The paper presents a scientific foundation model to solve convection-diffusion-reaction equations in a zero-shot setting. Its main novelty is that the PDE solution data used to train the model is itself generated by a PINN solver, and is therefore subject to noise and measurement error. The authors demonstrate the the model can outperform baselines and make accurate predictions under parameter distribution shifts and time-domain extrapolation.

strengths:
1. This work validates the hypothesis that foundation models are able to generalize even under noisy prior data, which has been thoroughly demonstrated in NLP but has been largely unexplored in scientific ML.

2. The model performs well even under substantial distribution shifts, which suggests that it is really learning to learn as opposed to memorizing the distribution of the training data.

weaknesses:
1. The authors provide very few details on their architecture, other than that they use a transformer with self-attention and cross-attention. This makes it very difficult to meaningfully compare their model with other baselines. I would suggest adding a detailed description of the architecture to the appendix.

2. The class of PDEs considered is quite restrictive, since it is a family of 1D equations parameterized by relatively few variables. It doesn't seem too surprising that a sufficiently complex foundation model can learn this family of equations.

Typos: In Theorem 2.1, I think the first inequality should say that $\mathbb{E}[KL(p_{\widehat{\theta}}(\cdot | x, D_n) \parallel \pi(\cdot | x, D_n))] < \epsilon$ rather than $\widehat{\theta} < \epsilon$.

questions:
1. Can you clarify specifically what you mean by 'zero-shot inference'? In Figure 2 (Inference Phase) it looks like you prompt your model with pairs $(x_i,t_i,u_i)$, and thus require data from the operator to be inferred in order to make predictions.

2. Can you explain your rationale for using the L^2 norm as an evaluation metric, as opposed to the H^1 norm or a more PDE-specific metric?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693259077

thread_type:
official_review

<thread_object>
creator: reviewer_szqg

<content>
summary:
This paper presents a scientific foundation model to solve PDEs with scarce data by training on low-cost data. This methodology is proven to be amenable to zero-shot learning with high performance, as evidenced by benchmark experiments across various kinds of PDEs.

strengths:
- The paper is well organized and written.
- Encouraging results across datasets.
- To have a genearlizable zero-shot PDE solver trained from low-cost data would have high impact on the physical modeling community.

weaknesses:
- Figure 2. used a lot of page real estate to demonstrate concepts not unique to the paper. I would perhaps highly the innovation of this paper here.
- I feel like to brand this as a _scientific_ foundation model is slightly overselling it. To solve PDEs is hardly equal to _science_.

questions:
- I feel like the idea behind Theorem 2.1 is rather intuitive. And it is a little out-of-place here. I don't see it referenced at all in the text. Could you motivate this theorem a bit more?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696645631

thread_type:
official_review

<thread_object>
creator: reviewer_fgqz

<content>
summary:
This paper introduced MaD-Scientist, a novel approach to solving partial differential equations (PDEs) using Transformer-based models trained on Physics-Informed Neural Network (PINN) generated prior data. The key innovation is demonstrating that scientific foundation models (SFMs) can learn effectively from approximate, low-cost PINN-generated data, similar to how large language models (LLMs) learn from noisy internet data. The authors focus on convection-diffusion-reaction (CDR) equations and show impressive zero-shot inference capabilities.

strengths:
- The parallel drawn between LLMs learning from noisy internet data and SFMs learning from approximate PINN-generated data is insightful and well-motivated. Similarly the zero-shot inference approach without requiring knowledge of governing equations is equally important. 
- Clear mathematical formulation of the problem and the methodology. 
- Extensive experiments covering multiple scenarios. The baselines are sound, and the results are excellent. Especially notable is the "superconvergence" phenomenon observed (Remark 4.1)

weaknesses:
- The scope of the paper is very limited. Focus solely on 1D CDR equations may limit the broader applicability. Furthermore, no real-world applications are thoroughly explored (albeit a brief mention of semi-conductors). 
- Limited discussion of computational efficiency and scaling properties, with not much variations of the transformer architecture explored. 
- The appendices contain important information that should be in the main text like efficiency comparison (above point), training algorithm and some PINN failure modes. 

Based on the questions/suggestions below, I am willing to increase my score.

questions:
- Can the authors investigate the method's performance on higher-dimensional PDEs?
- How does the computational cost scale with problem complexity?
- Will including physical constraints in the Transformer architecture improve performance?
- A discussion of potential real-world applications would strengthen the paper
</content>
</thread_object>

</thread>
<thread>
creation date:
1732208464154

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response (edited)

comment:
Thank you to all the reviewers for their reviews and constructive feedback. Before addressing each review individually, we will clarify some commonly raised points in this general response. All revised parts in the revised paper have been marked in red.

1. **Experiments on 2D CDR**    
We anticipated questions about experiments in higher dimensions, so we began these experiments immediately after submission. As a result, we are able to present the 2D experiment results promptly. In the revised paper, we include not only the original 1D CDR but also 2D CDR experiments. Additionally, we incorporate neural operator baselines and evaluate the model using not only the previous metrics but also the $L_{\infty}$ norm. 2D CDR experiments have been added in Appendix I of the revised paper. 

2. **Experiments on Biharmonic Equation**    
In response to Reviewer QAiw's question 3, we conduct additional experiments to evaluate our model's performance on PDEs with higher-order derivative terms. Specifically, we perform an interpolation task on the biharmonic equation, a PDE involving fourth-order derivative terms. Detailed information about this experiment has been added in Appendix J of the revised paper.

3. **Experiments on Burgers' Equation**    
In response to Reviewer QAiw's question 10, we conduct additional experiments to evaluate our model's performance on PDE, which has a solution profile with singular behavior. Specifically, we perform an extrapolation task on the Burgers equation, a PDE involving shock formation when the viscosity parameter is small. Detailed information about this experiment has been added in Appendix K of the revised paper.

4. **Experiments on diverse reaction term with fine-tuning**    
In response to Reviewer QAiw's question 14, we conduct additional experiments to fine-tune the model from Section 4.3, adapting it to each target reaction system. Detailed information about this experiment has been added in Appendix L.1 of the revised paper.

5. **Experiments on 1D CDR with diverse reaction term**    
In response to Reviewer QAiw's question 11, we conduct additional experiments on 1D CDR with diverse reaction terms using fine-tuning. We evaluate our model on an interpolation task with nonzero parameters $\beta$ and $\nu$, where $\beta, \nu, \rho_1, \rho_2, \rho_3 \in [1, 5] \cap \mathbb{Z}$. We demonstrate that a model pre-trained over a broad parameter space can be effectively adapted to specific equations. Detailed information about this experiment has been added in Appendix L.2 of the revised paper.

6. **Experiments on Sensitivity to given data number**    
In response to Reviewer sZQg's question 1, we conduct additional experiments on the model's sensitivity to the number of the given data $\widetilde{D}$ during testing. The results of this experiment provide empirical evidence for the findings proved in Theorem 2.1. Detailed information about this experiment has been added in Appendix M of the revised paper.

7. **Experimental details**    
For all experiments, detailed hyperparameter settings have been added in Appendix N of the revised paper, and the information about computational costs has been added in Appendix C.

8. **Additional information about detail model structure**    
In response to Reviewer weakness pointed out by sZQg and WwXL, we have added a detailed description of our model and a simplified figure focusing solely on the model structure in Appendix O of the revised paper. Additionally, we have incorporated a detailed illustration of the in-context learning mechanism during the inference phase into Figure 2. The refined version has been included in the revised paper.

9. **Typo**    
In our original paper, we stated that we used the $L_2$​ absolute error for evaluation, but this was a typo. We actually used the $L_1$​ absolute error as our metric. We have fixed it all in the revised paper.
Additionally, Reviewer WwXL kindly pointed out a typo in Theorem 2.1 and Reviewer QAiw kindly pointed out a typo in line 372. We have corrected them in the revised paper. Furthermore, we conduct a thorough review of the manuscript to identify and correct additional typos throughout the paper to ensure overall clarity and accuracy.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732449198275

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
To all reviewers

comment:
To all reviewers:    

First, we would like to express our gratitude for your insightful and valuable reviews.    
We have made every effort to provide the best possible responses to all the feedback we received.    
Please review our responses and let us know if you have any follow-up questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732673446438

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Additional general response

comment:
We have prepared responses to the additional experiments that required more time and were not included in the initial rebuttal, specifically addressing General Response 5 and Reviewer QAiw's Question 11.    
In particular, we conducted experiments on the 1D CDR equation with various reaction terms using fine-tuning. Our results demonstrate that a model pre-trained over a broad parameter space can be effectively adapted to specific equations. Detailed information about this experiment has been added to Appendix L.2 of the revised paper.    
The rebuttal has been updated to include this information, and the related experiments have been incorporated into Section L.2 of the revised paper. As a result, the content previously in Section L has been renumbered as Section L.1.


Lastly, please note that the revised paper cannot be updated after November 27. Therefore, we kindly ask reviewers to submit any additional questions before this deadline.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper applies LLMs to scientific foundation models (SFMs) for solving PDEs. Technically, the algorithm collects low-cost physics-informed neural network (PINN)-based approximated prior data in the form of solutions to partial differential equations (PDEs) constructed through an arbitrary linear combination of mathematical dictionaries, and utilize Transformer architectures with self and cross-attention mechanisms to predict PDE solutions without knowledge of the governing equations in a zero-shot setting. Experimental results were provided for one-dimensional convection-diffusion-reaction equations, supporting the conjecture that SFMs can improve in a manner similar to LLMs.

There were adequate discussions during the rebuttal, and the reviewers acknowledged the notable efforts from the authors during the review and AC discussion period. However, there are still general concerns:

1) The class of PDEs considered is restricted, and the results of the paper seem to be only a marginal improvement compared to previous works on SciFMs. The claim to introducing a scientific foundation model that is pre-trained on PINN-simulated data is an overclaim: while the idea of foundation models is to learn multimodel features from multiple pre-training physical systems, each involving thousands of samples with varying PDE parameters, the proposed framework is trained on only a handful of PDEs of similar types.

2) The superiority in terms of prediction accuracy over the existing scientific foundation models is not provided. Besides, the results based on the PINN prior appear to be poorer than those based on numerical priors, often showing more than 10% relative error. This raises concerns about whether the use of low-cost PINN-prior data is justified given the low prediction accuracy and can, therefore, actually outperform existing scientific foundation models.

Therefore, given the competitiveness this year at ICLR 2025, the decision is to reject this paper. Hopefully this doesn't disappoint the authors - the paper has been improving during the rebuttal, but more efforts are needed to address the aforementioned concerns for future venues. There are also many appendices newly added to the rebuttal that the authors may consider reorganizing for future versions of the paper.