MaD-Scientist: AI-based Scientist solving Convection-Diffusion-Reaction Equations Using Massive PINN-Based Prior Data

## Abstract

Abstract Large language models (LLMs), like ChatGPT, have shown that even trained with noisy prior data, they can generalize effectively to new tasks through in-context learning (ICL) and pre-training techniques.
Motivated by this, we explore whether a similar approach can be applied to scientific foundation models (SFMs). Our methodology is structured as follows: (i) we collect low-cost physics-informed neural network (PINN)-based approximated prior data in the form of solutions to partial differential equations (PDEs) constructed through an arbitrary linear combination of mathematical dictionaries; (ii) we utilize Transformer architectures with self and cross-attention mechanisms to predict PDE solutions without knowledge of the governing equations in a zero-shot setting; (iii) we provide experimental evidence on the one-dimensional convection-diffusion-reaction equation, which demonstrate that pre-training remains robust even with approximated prior data, with only marginal impacts on test accuracy. Notably, this finding opens the path to pre-training SFMs with realistic, low-cost data instead of (or in conjunction with) numerical high-cost data. These results support the conjecture that SFMs can improve in a manner similar to LLMs, where fully cleaning the vast set of sentences crawled from the Internet is nearly impossible.

## 1 Introduction

In developing large-scale models, one fundamental challenge is the inherent noisiness of the data used for training. Whether dealing with natural language, scientific data, or other domains, large datasets almost inevitably contain noise. Large language models (LLMs), such as ChatGPT, present an interesting paradox: despite being trained on noisy datasets, they consistently produce remarkably clean and coherent output. This observation raises an important question for the development of scientific foundation models (SFMs): Can an SFM, like an LLM, learn from noisy data and still generate accurate, dynamic results for complex scientific tasks?

Figure: Figure 1: An end-to-end schematic diagram of our model. Our model performs in-context learning based on the given observations, i.e., context, to infer the solution. Even when trained with noisy PINN-prior, our model can obtain clean solutions due to its Bayesian inference capability.
Refer to caption: extracted/5911933/Figure/Figure1.png

In recent years, LLMs have revolutionized the field of natural language processing by introducing highly flexible and scalable architectures (Brown et al., ; Kaplan et al., ; Touvron et al., ; Frieder et al., ; Chowdhery et al., ). Notably, the in-context learning (ICL) paradigm has demonstrated powerful generalization capabilities, enabling LLMs to adapt to new tasks without explicit fine-tuning (Brown et al., ; Radford et al., ; Dai et al., ; Gruver et al., ). This success has motivated the application of such foundation models across a variety of domains (Xu et al., ; Xie et al., ; Yang et al., ).
Scientific machine learning (SciML) is one such emerging domain which merges physics-based models with machine learning methodologies (Raissi et al., ; Willard et al., ; Subramanian et al., ; Kim et al., ; ; Choi et al., ). SciML aims to leverage the power of machine learning to solve complex scientific problems, including those governed by partial differential equations (PDEs). Recent efforts in this direction have led to the development of foundation models specifically designed for scientific tasks, called SFMs (Yang et al., ; Xie et al., ; Yang et al., ; Moor et al., ; Bodnar et al., ; Herde et al., ). These models aim to generalize across a wide range of scientific problems using prior data, much like how LLMs generalize across various language tasks.
For example, the versatility of in-context operator networks (ICONs), as illustrated in studies like Yang & Osher () and Yang et al. (), underscores their generalization capabilities in various PDE-related tasks, particularly in the context of few-shot learning. Moreover, the integration of in-context operator learning into multi-modal frameworks, as demonstrated by ICON-LM (Yang et al., ), has pushed the boundaries of traditional models by combining natural language with mathematical equations. Additionally, several other studies have focused on solving a family of PDEs with a single trained model (Cho et al., ). However, all these studies are limited in their ability to fully harness the capabilities of large foundation models.
Our methodology, called Massive prior Data-assisted AI-based Scientist (MaD-Scientist), addresses these limitations and offers significant advantages in the following four aspects.

##### No prior knowledge of physical laws

Our goal is to predict solutions from observed quantities, such as velocity and pressure, without relying on governing equations, a common challenge in many real-world scenarios (Lee & Cant, ; Nicolaou et al., ; Rouf et al., ; Beck & Kurz, ; Chien et al., ). In complex systems, such as those governing semiconductor manufacturing, the exact governing equations are often unknown and may change over time (Chien et al., ; Quirk & Serda, ). Therefore, excluding these equations from the model input is a strategic choice aimed at enhancing the applicability of our method across various domains.

##### Zero-shot inference

Our goal is to achieve zero-shot inference for predicting PDE solutions.
For instance, ICON-LM requires few-shot “demos”(^1^11In ICON and ICON-LM, a demo means a set of (input, output) pairs of an operator to infer.) for an unknown target operator before making predictions. In contrast, our foundation model eliminates the need for such demos, as collecting them implies that inference cannot occur until these few-shot examples are available; see e.g., Figure 1. Our approach is designed to enable immediate inference as soon as the model is queried.

##### Bayesian inference

We incorporate Bayesian inference into the prediction process by leveraging prior knowledge obtained from numerical solutions in PDE dictionaries. This approach allows the model to make more accurate and well-informed predictions by defining a prior distribution over unseen PDE coefficients. During training, the model learns to capture relationships among known data points using self-attention mechanisms, while cross-attention enables it to extrapolate and infer solutions for new, unseen points. When tested, the model utilizes this prior data to generalize effectively to novel data points, achieving zero-shot predictions without the need for additional fine-tuning.

##### Approximated prior data

For LLMs, one of the most challenging steps is collecting prior data, which typically involves crawling and cleaning sentences from the Internet. However, this process is far from perfect due to two key issues: (i) the Internet, as a data source, is inherently unreliable; (ii) cleaning such vast amounts of data requires significant manual effort. Consequently, LLMs are often trained on incomplete or imperfect prior data. Remarkably, this realistic yet critical issue has been largely overlooked in the literature on SFMs, despite their similarities to LLMs. For example, when generating data using numerical solvers for PDEs without known analytical solutions, numerical errors inevitably arise, manifesting as a form of measurement noise
— for this, we conducted preliminary experiments for training SFMs with noisy data in Appendix G, which shows the possibility of training SFMs with data inexact to some degree.

Moreover, numerical solvers running on large-scale servers are frequently expensive and time-consuming and they are typically optimized towards certain types of PDEs, e.g., the finite-difference time-domain (FDTD) method for Maxwell’s equations. In this work, we are the first to explore the potential of pre-training SFMs with PINN-based low-cost/noisy/approximated data.

##### CDR Study

For our empirical studies, we use a family of the convection-diffusion-reaction (CDR) equation with various types of reaction, which serves as a paradigm problem representing generic elliptic equations. By solving the CDR equation, our approach can be extended to a wide range of other problems. We compare our method with two state-of-the-art machine learning techniques for solving parameterized PDEs. Additionally, we introduce three different types of noise into the numerical solutions of the CDR equation. Our approach not only outperforms the two baseline methods but also demonstrates stable performance, even when noise is added to the prior data during pre-training.

## 2 Background

Consider a sequence of pairs $(X_{1},Y_{1}),(X_{2},Y_{2}),\ldots$, each within the measurable space $(\mathcal{X}\times\mathcal{Y},\mathcal{A})$, where $X_{i}$ represents the spatiotemporal coordinate, $Y_{i}$ denotes the corresponding solution in this paper’s context and $\mathcal{A}$ denotes the Borel $\sigma$-algebra on the measurable space $\mathcal{X}\times\mathcal{Y}$. For simplicity, we adopt this notation in this section. These pairs are drawn from a family of probability density distributions $\{p_{q}:q\in\mathcal{Q}\}$, commonly referred to as the statistical model, where $\mathcal{Q}$ represents the parameter space equipped with a $\sigma$-algebra $\mathcal{B}$ ensuring that the mappings $q\mapsto p_{q}(x,y)$ are measurable. The true underlying density function $\pi$ is a member of $\mathcal{Q}$, and the pairs $(X_{i},Y_{i})$ are sampled according to $p_{\pi}$.
Lacking information about $\pi$, we adopt a Bayesian framework to establish a prior distribution $\Pi$ which is defined as probability measure on $(\mathcal{Q},\mathcal{B})$. Then we have, for any measurable set $A\in\mathcal{B}$,

$$ $\Pi(A\mid X,Y)=\frac{\int_{A}p_{q}(X,Y)\mathrm{d}\Pi(q)}{\int_{\mathcal{Q}}p_{ q}(X,Y)\mathrm{d}\Pi(q)}.$ (1) $$

Let us adopt the notation $p_{q}=q$. This prior is updated with the observed data to form the posterior distribution, which is defined as

$$ $\Pi(A\mid D_{n})=\frac{\int_{A}L_{n}(q)\mathrm{d}\Pi(q)}{\int_{\mathcal{Q}}L_{ n}(q)\mathrm{d}\Pi(q)},$ (2) $$

where $L_{n}(q)=\prod_{i=1}^{n}\frac{q(X_{i},Y_{i})}{\pi(X_{i},Y_{i})}$ for $A\subset\mathcal{Q}$ and $D_{n}=\{(X_{i},Y_{i})\}_{i=1}^{n}$.
The resulting posterior density is

$$ $q_{n}(X,Y\mid D_{n})=\int_{\mathcal{Q}}q(X,Y)\mathrm{d}\Pi(q\mid D_{n}),$ (3) $$

and the posterior predictive distribution (PPD) is formulated as

$$ $\pi(y\mid x,D_{n})=\int_{\mathcal{Q}}q(y\mid x)\,\mathrm{d}\Pi(q\mid D_{n}).$ (4) $$

The behavior of $D_{n}$ plays a crucial role in this formulation.
As noted by Walker (; ); Blasi & Walker (); Walker (); Nagler (), for a well-behaved prior, the PPD converges toward $\pi$ as $n$ increases. This aligns with findings in Blasi & Walker (), demonstrating that in well-specified scenarios, strong consistency is achieved as

$$ $\Pi^{n}\left(\{q:H(\pi,q)>\epsilon\}\right)\to 0\quad\text{almost surely},$ (5) $$

for any $\epsilon>0$,
where $\Pi^{n}(A)=\int_{A}d\Pi(q\mid D_{n})$ is the posterior measure and $H$ is the Hellinger distance defined by

$$ $H(p,q)=\left(\int_{\mathcal{X}\times\mathcal{Y}}(\sqrt{p}-\sqrt{q})^{2}\, \right)^{1/2}.$ $$

###### Theorem 2.1 .

Suppose that for any $\epsilon>0$, there exists a Transformer parameterized by $\hat{\theta}$ such that

$$ $\hat{\theta}=\underset{\theta}{\arg\min}\;\mathbb{E}_{x}\left[KL\left(p_{ \theta}(\cdot\mid x,D_{n}),\pi(\cdot\mid x,D_{n})\right)\right]<\epsilon,$ $$

for any realization of $D_{n}$.
If the posterior consistency condition equation 5 holds, and for any $q\in\mathcal{Q}$, $q(x)=\pi(x)$ almost everywhere on $\mathcal{X}$, then the following holds almost surely (see Appendix A for proof):

$$ $\mathbb{E}_{x}\left[H\left(p_{\hat{\theta}}(\cdot\mid x,D_{n}),\pi(\cdot\mid x )\right)\right]\xrightarrow{n\to\infty}0.$ $$

This result demonstrates that as the amount of data increases, the neural network close to the posterior distribution converges to the expected value under the prior distribution, highlighting the consistency and robustness of the Bayesian inference process. Based on this observation, our model performs the Bayesian inference with prior data. Ultimately, our model’s goal is to read some ground-truth spatiotemporal points and infer an appropriate PDE solution that accurately describes the dynamics under the given spatiotemporal conditions.

###### Theorem 2.1 .

## 3 Methods

Suppose the dataset $D_{n}=\{(X_{i},T_{i},Y_{i})\}_{i=1}^{n}$ is independently and identically distributed (i.i.d.) and sampled from a distribution $q_{\alpha}$, where $\alpha$ is the parameter vector representing the coefficients governing the PDE dynamics, including convection, diffusion, and reaction terms. Specifically, $Y_{i}\sim u(X_{i},T_{i}\mid\alpha)+\text{noise}$, where the noise represents the difference between the PINN-predicted solution $\tilde{u}(\alpha)$ and the true solution $u(\alpha)$. The PPD of the solutions given the dataset can be expressed as

$$ $q(y\mid x,t,D_{n})=\int_{H}q_{\alpha}(y\mid x,t)\,\mathrm{d}\Pi(q_{\alpha}\mid D _{n}),$ (6) $$

which represents the likelihood distribution of $y$ given $D_{n}$, capturing the most probable solution distribution for the given parameter $\alpha$. In this work, we aim to predict the solution from $D_{n}$ by minimizing the mean squared error (MSE) between the PPD-derived solution and the true solution, even in the presence of noise. This requires constructing a prior over the PDE solution space, which is detailed next.

##### Benchmark PDE

The following one-dimensional convection-diffusion-reaction (CDR) equation is used for the benchmark PDE,

$$ $\displaystyle\textnormal{1D CDR: }u_{t}+\beta u_{x}-\nu u_{xx}-\rho f(u)=0,\; \;\;\;x\in[0,2\pi],t\in[0,1],$ (7) $$

where $f:\mathbb{R}\rightarrow\mathbb{R}$ is a reaction term such as Fisher, Allen-Cahn and Zeldovich. This equation consists of three key terms with distinct properties: convective, diffusive, and reactive, making it an ideal benchmark problem. It is commonly used in the PINN literature due to the diverse dynamics introduced by its three parameters: $\beta,\nu,$ and $\rho$, which include various failure modes (Krishnapriyan et al., ).
To our knowledge, however, our work is the first predicting all those different reaction terms with a single model.

In this paper, we use the following dictionary of CDR-related terms, incorporating a linear combination of $J$ nonlinear reaction terms, for generating prior data.

$$ $\displaystyle u_{t}=\mathcal{N}(\cdot),$ $\displaystyle\mathcal{N}(t,x,u,\beta,\nu,\rho_{1},\cdots,\rho_{J})=-\beta u_{x }+\nu u_{xx}+\sum_{j=1}^{J}\rho_{j}f_{j}(u),$ (8) $$

where each $f_{j}$ represents specific reaction term. This expansion allows for the introduction of diverse reaction dynamics.
One can solve CDR equations with numerical solvers. In this work, however, we are interested in building low-cost PINN-based prior data. In the future, one may need to build prior data for not only CDR but also many other equations for which none of analytical/numerical solutions are obtainable in a low-cost manner, e.g., Naiver-Stokes equations. We think our PINN-based prior data will play a crucial role in such a case.

##### PINN-Prior of PDE Solution Space

To approximate the solution space for PDEs,
we construct a parameter space, $\Omega$, which is the collection of coefficients in equation 8:

$$ $\displaystyle\Omega=\{\alpha$ $\displaystyle:=(\beta,\nu,\rho_{1},\cdots,\rho_{J})\},$ (9) $$

which has a dictionary form. Consequently, the target exact prior $\mathcal{U}$ represents the collection of solutions $u(\alpha)$ at equation 8 for each parameter $\alpha\in\Omega$, where $\mathcal{X}$ and $\mathcal{T}$ correspond to the spatial and temporal domains of interest, respectively

$$ $\displaystyle\mathcal{U}=\bigcup_{\alpha\in\Omega}\{u(\alpha)\,|\,u_{t}= \mathcal{N}(t,x,u,\alpha)\},\;\;\;\;\mathcal{U}:\mathcal{X}\times\mathcal{T} \rightarrow\mathbb{R}.$ (10) $$

Since the target exact prior data $\mathcal{U}$ is hard to obtain, we instead use a PINN-prior $\mathcal{D}$ that closely approximates $\mathcal{U}$ as follows. Suppose $\tilde{u}(\alpha)$ is the prediction by PINN (Appendix E) which is trained to predict the PDE $u_{t}=\mathcal{N}(\cdot)$. The PINN-prior $\mathcal{D}$ is a collection of approximated solutions $\tilde{u}(\alpha)$ for each $\alpha\in\Omega$,

$$ $\displaystyle\mathcal{D}=\bigcup_{\alpha\in\Omega}\{\tilde{u}(\alpha)\},\;\;\; \;p(\mathcal{D})\sim p(\mathcal{U}).$ (11) $$

Subsequently, the model learns the PPD of the generated prior $p(\mathcal{D})$ through ICL.

Figure: Figure 2: A schematic diagram of Transformer. (Left) The Transformer $\tilde{u}_{\theta}$ takes prior of solution-known $D$ and querying task $T$ drawn from the prior distribution $\mathcal{D}$ and infers solutions of the queried points in the training phase. ICL is leveraged with a self-attention among $D$ (blue rods) and a cross-attention from $T$ to $D$ (red rods). (Right) In the testing phase, $\tilde{u}_{\theta}$ takes an input of unseen data $\widetilde{D}$ and $\widetilde{T}$ drawn from the ground truth distribution $\mathcal{U}$, and the model predicts the queried points $\widetilde{T}$.
Refer to caption: extracted/5911933/Figure/Figure2.png

##### Training

From a given parameter space $\Omega$, the parameter $\alpha$ is randomly drawn i.i.d. from $\Omega$. This method is adopted from meta learning (Finn et al., ) which optimizes the model parameter to adapt to various tasks, in our case the prediction over wide prior space $\mathcal{D}$ expressed as a dictionary over $\alpha$. After that, the previous $\tilde{u}(\alpha)$ is then given as an input to Transformer $\tilde{u}_{\theta}$ to minimize the mean square error (MSE) at the predicted points, see equation 12. The MSE loss criterion is proposed as the Transformer’s task is to perform regression of the solution over the spatial and temporal domain for given $\tilde{u}(\alpha)$,

$$ $L_{\alpha}=\frac{1}{N_{T}}\sum_{j=1}^{N_{T}}\left[\tilde{u}_{\theta}(x_{T}^{(j )},t_{T}^{(j)}\mid D_{n})-\tilde{u}(x_{T}^{(j)},t_{T}^{(j)})\right]^{2}.$ (12) $$

##### Evaluation

To illustrate the model’s zero-shot learning capability in scenarios commonly encountered in practical applications, we assess the model’s performance using data $\tilde{D}$ sampled i.i.d. from $\mathcal{U}$, not overlapping with the training set $D\cup T$. For evaluation, we employ both absolute and relative $L_{2}$ errors between the model’s predicted solutions for test queries and the numerically computed ground truth. These errors are then averaged over the target parameter space $\Omega$ used during training.

## 4 Experiments

Our experiment section is divided into two phases: in the first phase, we conduct a focused study with the basic reaction term, Fisher, to understand the base characteristics of SFMs, and in the second phase we conduct comprehensive studies with various reaction terms.

### 4.1 Experimental Setup

##### Baseline methods

We compare our model with 2 baselines: Hyper-LR-PINN (Cho et al., )
and P^2INN without fine tuning (Cho et al., ). Both models are parametrized PINNs designed to learn parameterized PDEs. Hyper-LR-PINN emphasizes a low-rank architecture with a parameter hypernetwork, while P^2INN focuses on a parameter-encoding scheme based on the latent space of the parameterized PDEs.

Following this, as shown in Figure 2, the model takes $D\cup T\sim p(\mathcal{D})$ in training phase and $\widetilde{D}\cup\widetilde{T}\sim p(\mathcal{U})$ in testing phase. In addition, the dataset $D\cup T$ requires the prior $\widetilde{u}$, and $\widetilde{D}$ requires the solution $u$. For a fair comparison, we use $D$, $T$, and $\widetilde{D}$ as the training dataset for both Hyper-LR-PINN and P^2INN. Notably, while Hyper-LR-PINN and P^2INN do not rely on solution points during training and testing, our model operates without any knowledge of the governing equation $\mathcal{N}(\cdot)$. This setup ensures a valid and balanced comparison (Table 1). The additional comparison details are elaborated in Appendix C, providing further insights into the distinctions between these models.

**Table 1: Major comparisons between Hyper-LR-PINN, P^2INN, and our model. While both Hyper-LR-PINN and P^2INN require the knowledge of governing equation, our model only needs observed quantities. The notations used in the table are fully aligned with those in Figure 2.**
| Properties | Hyper-LR-PINN | P^2INN | Ours |
| --- | --- | --- | --- |
| Target function | $u(x,t;\alpha\in\Omega)$ | $u(x,t;\alpha\in\Omega)$ | $u(x,t)|_{\mathcal{D}}$ |
| Governing equation $\mathcal{N}(\cdot)$ given | $\checkmark$ | $\checkmark$ | ✗ |
| Train dataset | $D\cup T\cup\widetilde{D}$ | $D\cup T\cup\widetilde{D}$ | $D\cup T$ |
| Test dataset | $\widetilde{T}$ | $\widetilde{T}$ | $\widetilde{D}\cup\widetilde{T}$ |
| Dataset with a solution | None | None | $D,T,\widetilde{D}$ |

##### Training algorithm

The concrete flow of training phase is described in Algorithm at Appendix F.

### 4.2 Focused Study to Better Understand SFMs’ base characteristics

In this section, we employ six different dynamics derived from the 1D CDR equation with a Fisher reaction term, $u_{t}+\beta u_{x}-\nu u_{xx}-\rho u(1-u)=0$ (Appendix D). We begin with an in-depth study using the Fisher reaction term, chosen for its simplicity among the reaction terms, which has been extensively studied in population dynamics (Al-Khaled, ). This allows us to better understand the core characteristics of the SFM, facilitating a more effective analysis of the model’s behavior.

#### 4.2.1 Time Domain Interpolation for Seen PDE Parameters with a Numerical Prior

We first verify the ICL capability of Transformer with a numerical prior, i.e., $\tilde{u}(\alpha)$ equals to the solution $u$ of the PDE $u_{t}=\mathcal{N}(t,u,x,\alpha)$, before we dive into using a PINN prior. For each equation, we set the parameter space $\Omega$ with three different coefficient ($\beta,\nu,\rho$) range: $\left([1,5]\cap\mathbb{Z}\right)^{m},\left([1,10]\cap\mathbb{Z}\right)^{m}$, and $\left([1,20]\cap\mathbb{Z}\right)^{m}$, where $m$ is the number of nonzero coefficients. The Transformer $\widetilde{u}_{\theta}$ is trained with $D\cup T\subseteq u(\alpha)$ where $\alpha\in\Omega$ is selected uniformly at random for each epoch. After that, we test $\widetilde{u}_{\theta}$ with $\widetilde{D}\cup\widetilde{T}\subseteq u(\alpha)$ for all $\alpha\in\Omega$ and evaluate average $L_{2}$ absolute and relative error (Table 2).

We highlight two key observations: First, our model outperforms baseline models applied to diffusion, reaction, reaction-diffusion, and convection-diffusion-reaction systems. Second, it demonstrates stable performance across a wide range of coefficient values.
For instance, all baselines show difficulties in predicting accurate solutions for high coefficients, especially in diffusion and reaction systems, while ours do not. When we measure the standard deviation of $L_{2}$ relative error over three coefficient range for diffusion system, ours have $9.1\times 10^{-4}$ while others show $10^{-2}$ scale value. These observations not only verify the effectiveness of the Transformer’s ICL capability, but also suggest its potential to handle larger parameter space $\Omega$.

Table: Table 2: The relative and absolute $L_{2}$ errors over the 1D-CDR equation using a numerical prior. P^2INN is tested without fine-tuning, and *-marked cases are evaluated with a reduced number of parameters due to the extensive computational requirements.

#### 4.2.2 Time Domain Interpolation for Seen PDE Parameters with a PINN-Priors

The Transformer has demonstrated strong ICL capabilities when trained with numerical priors. Our main focus now is to determine if this same success can be achieved using a PINN-prior. As outlined in Appendix C, our preliminary results show that the Transformer remains robust even when numerical priors are subject to various types of noise.
Building on this, we examine how the model performs when mixing low-cost PINN-priors with numerical priors in different proportions, assessing its stability and robustness when incorporating PINN-priors.

Specifically, we train the model using the convection, diffusion, and Fisher reaction equations with integer coefficients ranging from 1 to 20. For each equation, we evaluate the model with a prior that is a mixture of PINN-prior and numerical prior in varying ratios: 0%, 20%, 40%, 60%, 80%, and 100% PINN-priors. Table 3 indicates the absolute and relative errors $L_{2}$ for each setup compared to the baseline results in Section 4.2. Furthermore, the average $L_{2}$ error of the PINN-prior, compared to the numerical solution, is presented to demonstrate the quality of the PINN-prior.

As a result, mixing PINN-priors with numerical priors does not significantly impact performance, as the $L_{2}$ absolute and relative errors remain consistent with other baselines. This indicates that a Transformer can maintain ICL capability even when trained with PINN-prior data.
Also, this finding confirms that the model can effectively infer solutions from limited observed data $\tilde{D}$, even in the presence of inaccurate PINN-priors.

###### Remark 4.1 .

Notably, in diffusion case, the prior exhibits an error of $14\%$, yet our prediction error stands at just $2.6\%$. This discrepancy not only highlights the Transformer model’s strong ICL capability but also demonstrates a form of superconvergence, where the model significantly outperforms expectations given the inaccurate prior. Such a result underscores the robustness and adaptability of our approach, reinforcing the idea that even with flawed prior information, the Transformer can extract meaningful insights and achieve high accuracy in predictions.

Table: Table 3: $L_{2}$ error for convection, diffusion, and reaction equations measured at seen parameters, where the parameter values range from 1 to 20. Results are provided for each PINN-Prior Ratio, and for comparison, the results of baseline models are also included.

###### Remark 4.1 .

#### 4.2.3 Time Domain Interpolation for Unseen PDE Parameters

From this point, we train our model using only PINN-priors and further explore the base characteristics of SFMs’. In this section, we test our model with unseen parameters at convection, diffusion, and reaction systems. For each system, the model is trained with $[1,20]\cap\mathbb{Z}$ range coefficients and tested with unseen coefficient $1.5,2.5,\cdots,19.5$ which is included in interval $[1,20]$ and $20.5,21.5,22.5,\cdots,30.5$ which is not in range of $[1,20]$.
The $L_{2}$ relative error measured for each coefficient value is plotted in Figure 3, along with the baselines Hyper-LR-PINN and the non-fine-tuned P^2INN.

Figure: (a) convection
Refer to caption: extracted/5911933/Figure/Figure3_label.png

Over the trained coefficient range, our model effectively interpolates the coefficients $\beta$, $\nu$, and $\rho$, achieving performance comparable to that seen with known coefficients. Moreover, the model demonstrates stable extrapolation in diffusion and reaction systems. Compared to the baselines, our model significantly outperforms it, particularly in diffusion and reaction systems. This result indicates that the Transformer can effectively learn the PPD of the prior space $\mathcal{D}$, even without observing the complete prior.

#### 4.2.4 Time Domain Extrapolation for Seen PDE Parameters

One major limitation of the PINN is an extrapolation at the temporal domain that infer solutions at unknown points. Our model demonstrates extrapolation capability in the 1D convection equation, where the solution exhibits wave-like fluctuations in the inference region. In particular, the model trained with the PINN-prior $\mathcal{D}$ over the coefficient range $\beta\in[1,20]\cap\mathbb{Z}$ can predict $\beta$ values in ${1.5,2.5,\cdots,16.5}$ for equations where the test points $\widetilde{T}$ fall within $t\in(0.6,1.0]$, even though $\widetilde{D}$ is only distributed within $t\in[0.0,0.6]$. We then evaluate the relative $L_{2}$ error and plot for each coefficient $\beta$ with our baselines. Both baselines are not fine-tuned for each test $\beta$ to make a fair comparison with our zero-shot model.

Figure: Figure 4: (Left) The $L_{2}$ relative error is evaluated for each convection coefficient $\beta=1.5,2.5,\cdots,16.5$ as an extrapolation task. (Right) The graph illustrates the extrapolation of convection equation with $\beta=10.5$ at $0.6\leq t\leq 1.0$.
Refer to caption: extracted/5911933/Figure/Figure4.png

As a result, our model demonstrates effective extrapolation capabilities in convection equation (Figure 4, Left). In addition, our model outperforms both Hyper-LR-PINN and P^2INN across most values of $\beta$, while maintaining a stable $L_{2}$ relative error over a wider range. The diagram in Figure 4, Right presents the detailed performance at $\beta=10.5$. This capability emphasizes our model’s potential for advancing solutions to PDEs in unknown spatial regions and for enhancing time series predictions.

### 4.3 Comprehensive Study with Various Reaction Terms

In this section, we expand the parameter space to following $\Omega$ using three different reaction terms: Fisher ($f_{1}$), Allen-Cahn ($f_{2}$), and Zeldovich ($f_{3}$),

$$ $\displaystyle u_{t}=\mathcal{N}(\cdot),\;\;\;\;\mathcal{N}(t,x,u,\alpha)$ $\displaystyle=-\beta u_{x}+\nu u_{xx}+\sum_{j=1}^{3}\rho_{j}f_{j}(u),$ (13) $\displaystyle f_{1}:=u(1-u),\;\;f_{2}:=u$ $\displaystyle(1-u^{2}),\;\;f_{3}:=u^{2}(1-u),$ $\displaystyle\Omega=\{\alpha:=(\beta,\nu,$ $\displaystyle\;\rho_{1},\rho_{2},\rho_{3})\}.$ $$

To justify the expansion, we train the Transformer with $\beta=0$, $\nu=0$, and $\rho_{j}\in[1,5]\cap\mathbb{Z}$ for $j=1,2,3$ to evaluate its ICL capability in handling linear combinations of the reaction terms. The model is then tasked with inferring the solutions of the PDEs $u_{t}=\rho_{1}f_{1}$, $u_{t}=\rho_{2}f_{2}$, and $u_{t}=\rho_{3}f_{3}$ to test whether it can generalize to unseen PDEs and accurately distinguish between each component.

**Table 4: The Transformer is trained using a linear combination of Fisher, Allen-Cahn, and Zeldovich reaction terms with a given train parameter range. The $L_{2}$ absolute and relative errors for inferring each reaction term are then averaged over the given test parameter range. For comparison, the results for the Fisher reaction term tested in Section 4.2.1 are also included.**
| Train Parameter Range | $\rho_{1}\in[1,5]\cap\mathbb{Z}$ | $\rho_{1},\rho_{2},\rho_{3}\in[1,5]\cap\mathbb{Z}$ |  |  |
| --- | --- | --- | --- | --- |
| Test Parameter Range | $\rho_{1}\in[1,5]\cap\mathbb{Z}$ | $\rho_{1}\in[1,5]\cap\mathbb{Z}$ | $\rho_{2}\in[1,5]\cap\mathbb{Z}$ | $\rho_{3}\in[1,5]\cap\mathbb{Z}$ |
| $\mathbf{L_{2}}$ Abs Err. | 0.0102 | 0.0755 | 0.0381 | 0.0611 |
| $\mathbf{L_{2}}$ Rel Err. | 0.0154 | 0.1098 | 0.0734 | 0.0830 |

According to the result at Table 4, the $L_{2}$ absolute and relative errors are comparable to those obtained when trained with $\rho_{1}\in[1,5]\cap\mathbb{Z}$, suggesting the potential for expanding the parameter space. Specifically, the results demonstrate that our model can accurately distinguish between the three different reaction terms, even when trained with their linear combination. Although the model was trained using arbitrary linear combinations of terms commonly found in real-world applications, it is capable of effectively solving PDEs composed of meaningful combinations of these terms during testing. This demonstrates the model’s ability to generalize beyond its training data and infer significant governing relationships from complex systems.

## 5 Related Works

##### In-context learning

Transformers have shown remarkable ICL abilities across various studies. They can generalize to unseen tasks by emulating Bayesian predictors (Panwar et al., ) and linear models (Zhang et al., ), while also efficiently performing Bayesian inference through Prior-Data Fitted Networks (PFNs) (Müller et al., ). Their robustness extends to learning different classes of functions, such as linear and sparse linear functions, decision trees, and two-layer neural networks even under distribution shifts (Garg et al., ). Furthermore, Transformers can adaptively select algorithms based on input sequences, achieving near-optimal performance on tasks like noisy linear models (Bai et al., ). They are also highly effective and fast for tabular data classification (Hollmann et al., ).

##### Foundation model

Recent studies have advanced in-context operator learning and PDE solving through Transformer-based models. Ye et al. () introduces PDEformer, a versatile model for solving 1D PDEs with high accuracy and strong performance in inverse problems. In-context operator learning has also been extended to multi-modal frameworks, as seen in Yang et al. (), where ICON-LM integrates natural language and equations to outperform traditional models. Additionally, Yang & Osher () and Yang et al. () demonstrate the generalization capabilities of In-Context Operator Networks (ICON) in solving various PDE-related tasks, highlighting ICON’s adaptability and potential for few-shot learning across different differential equation problems.
Several other studies have addressed the problem of solving various PDEs using a single trained model (Hang et al., ; Herde et al., ) . However, many of these approaches rely on symbolic PDE information, true or near-true solutions and/or do not support zero-shot in-context learning, making their objectives different from ours.

## 6 Conclusion and Limitations

In this work, we presented MaD-Scientist for scientific machine learning that integrates in-context learning and Bayesian inference for predicting PDE solutions. Our results demonstrate that Transformers, equipped with self-attention and cross-attention mechanisms, can effectively generalize from prior data, even in the presence of noise, and exhibit robust zero-shot learning capabilities. These findings suggest that foundation models in SciML have the potential to follow the development trajectory similar to that of natural language processing foundation models, offering new avenues for further exploration and advancement in the field.

The Transformer used in our study clearly demonstrates the ICL capability, when trained with PINN-based prior. However, it is limited to the CDR equations in our paper. We will consider other types of PDE and more diverse initial and boundary conditions in the future, enhancing its adaptability to real-world scenarios and its role as a foundation model.

Acknowledgements. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525. This paper describes objective technical results and analysis. Any subjective views or opinions that might be expressed in the paper do not necessarily represent the views of the U.S. Department of Energy or the United States Government. This article has been co-authored by an employee of National Technology & Engineering Solutions of Sandia, LLC under Contract No. DE-NA0003525 with the U.S. Department of Energy (DOE). The employee owns all right, title and interest in and to the article and is solely responsible for its contents. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this article or allow others to do so, for United States Government purposes. The DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan https://www.energy.gov/downloads/doe-public-access-plan. K. Lee acknowledges support from the U.S. National Science Foundation under grant IIS 2338909.

## Ethics Statement

This research adheres to the ethical standards required for scientific inquiry. We have considered the potential societal impacts of our work and have found no clear negative implications. All experiments were conducted in compliance with relevant laws and ethical guidelines, ensuring the integrity of our findings. We are committed to transparency and reproducibility in our research processes.

## Reproducibility

We are committed to ensuring the reproducibility of our research. All experimental procedures, data sources, and algorithms used in this study are clearly documented in the paper. The code and datasets will be provided as the supplementary material and be made publicly available upon publication, allowing others to validate our findings and build upon our work.

## Appendix A The Proof of Theorem 2.1

###### Proof.

For any $n,\epsilon$, we derive that

$$ $\displaystyle\mathbb{E}_{x}\left[H\left(p_{\hat{\theta}}(\cdot\mid x,D_{n}), \pi(\cdot\mid x)\right)\right]$ $\displaystyle\leq\mathbb{E}_{x}\left[H\left(p_{\hat{\theta}}(\cdot\mid x,D_{n} ),\pi(\cdot\mid x,D_{n})\right)\right]$ (1) $\displaystyle\quad+\mathbb{E}_{x}\left[H\left(\pi(\cdot\mid x,D_{n}),\pi(\cdot \mid x)\right)\right]$ $\displaystyle\leq\sqrt{\frac{1}{2}\mathbb{E}_{x}\left[KL\left(p_{\hat{\theta}} (\cdot\mid x,D_{n}),\pi(\cdot\mid x,D_{n})\right)\right]}$ (2) $\displaystyle\quad+\mathbb{E}_{x}\left[H\left(\pi(\cdot\mid x,D_{n}),\pi(\cdot \mid x)\right)\right]$ $\displaystyle\leq\sqrt{\frac{\epsilon}{2}}+\mathbb{E}_{x}\left[1-\int_{ \mathcal{Y}}\sqrt{\int q(y\mid x)\pi(y\mid x)\mathrm{d}\Pi^{n}(q)}\mathrm{d}y \right]^{1/2}$ (3) $\displaystyle\leq\sqrt{\frac{\epsilon}{2}}+\left[1-\int_{\mathcal{X}}\pi(x) \int_{\mathcal{Y}}\frac{1}{\pi(x)}\sqrt{\int q(y,x)\pi(y,x)\mathrm{d}\Pi^{n}(q )}\mathrm{d}y\mathrm{d}x\right]^{1/2}$ $\displaystyle\leq\sqrt{\frac{\epsilon}{2}}+\left[1-\int_{\mathcal{X}}\int_{ \mathcal{Y}}\int\sqrt{q(y,x)\pi(y,x)}\mathrm{d}\Pi^{n}(q)\mathrm{d}y\mathrm{d} x\right]^{1/2}$ $\displaystyle=\sqrt{\frac{\epsilon}{2}}+\left[\int H\left(q,\pi\right)^{2} \mathrm{d}\Pi^{n}(q)\right]^{1/2}$ $\displaystyle\leq\sqrt{\frac{\epsilon}{2}}+\left[\int H\left(q,\pi\right)d\Pi^ {n}(q)\right]^{1/2}$ $\displaystyle=\sqrt{\frac{\epsilon}{2}}+\left[\int_{\{q:H(\pi,q)>\epsilon\}}H \left(q,\pi\right)\mathrm{d}\Pi^{n}(q)\right]^{1/2}$ $\displaystyle\quad+\left[\int_{\{q:H(\pi,q)\leq\epsilon\}}H\left(q,\pi\right) \mathrm{d}\Pi^{n}(q)\right]^{1/2}$ (4) $\displaystyle=\sqrt{\frac{\epsilon}{2}}+(\Pi^{n}(\{q:H(\pi,q)>\epsilon\})+ \epsilon)^{1/2}\rightarrow\sqrt{\frac{\epsilon}{2}}+\sqrt{\epsilon}\quad\text{ a.s.}$ (5) $$

The first inequality (1) is derived from the triangle inequality for the Hellinger distance, which states that for any intermediate distribution $q(\cdot\mid x,D_{n})$, we have

$$ $H\left(p_{\hat{\theta}}(\cdot\mid x,D_{n}),\pi(\cdot\mid x)\right)\leq H\left( p_{\hat{\theta}}(\cdot\mid x,D_{n}),q(\cdot\mid x,D_{n})\right)+H\left(q(\cdot \mid x,D_{n}),\pi(\cdot\mid x)\right).$ $$

The second inequality (2) uses the fact that the Hellinger distance $H(p,q)$ is bounded above by the square root of the KL divergence $KL(p\parallel q)$, such that

$$ $H(p,q)^{2}\leq\frac{1}{2}KL(p\parallel q).$ $$

Thus, we can bound the Hellinger distance by the KL divergence. In the third inequality (3), we make use of assumption

$$ $\mathbb{E}_{x}\left[KL\left(p_{\hat{\theta}}(\cdot\mid x,D_{n}),\pi(\cdot\mid x ,D_{n})\right)\right]<\epsilon,$ $$

and utilize the definition of the Hellinger distance.
In (4), we partition the domain into two regions–one where the Hellinger distance $H(\pi,q)$ exceeds $\epsilon$ and another where it is less than or equal to $\epsilon$–and use this partitioning to demonstrate the inequality.

Finally, in (5), by posterior consistency, the region where the Hellinger distance is greater than $\epsilon$ vanishes as $n\rightarrow\infty$ such that

$$ $\Pi^{n}\left\{q:H(\pi,q)>\epsilon\right\}\to 0\quad\text{almost surely}.$ $$

Since $\epsilon$ is arbitrary, we can conclude that

$$ $\mathbb{E}_{x}\left[H\left(p_{\hat{\theta}}(\cdot\mid x,D_{n}),\pi(\cdot\mid x )\right)\right]\xrightarrow{n\to\infty}0\quad\text{almost surely}.$ $$

∎

###### Proof.

## Appendix B Environments

We conducted the experiments using Python 3.8.19, PyTorch 2.4.0+cu121, scikit-learn 1.3.2, NumPy 1.24.4, and pandas 2.0.3, with CUDA 12.1, NVIDIA Driver 535.183.01, and an NVIDIA RTX A6000. Additionally, we used CUDA 12.4, NVIDIA Driver 550.67, and either an NVIDIA GeForce RTX 3090 or an NVIDIA TITAN RTX.

## Appendix C Additional Comparison Between Baselines

In addition to the comparison between baselines in 1, the additional comparisons between baselines is shown below. For Hyper-LR-PINN, the results are estimated solely for Phase 1 and the number of model parameters estimated refers specifically to the model used in sections 4.2.

Table: Table 5: Additional comparisons between baselines.

## Appendix D Datasets

The specific information about PDE types used in the study is following. $n$ represent the maximum values for each parameter $\beta,\nu,\rho_{1},\rho_{2}$ and $\rho_{3}$’s range, which are set to 5, 10, and 20 in our study.

Table: Table 6: PDEs used in our study and corresponding dataset information for each section. ”c-d-r” means ”convection-diffusion-reaction”.

For each specific PDE, we collect 256 initial points, 1,000 collocation points, 100 boundary points, and 1,000 test points. For each 1,000 collocation points, we sample 800 points for $D\cup T$ and 200 points for $\tilde{D}$ which are not overlapped.
During the training phase, 30% of the data points are designated as $D$, while the remaining 70% are allocated to $T$.

## Appendix E PINN Used in Prior Generation

In this study, we utilize the PINN introduced by Raissi et al. () to generate PINN-priors. The loss function employed during the training of the PINN is as follows:

$$ $\mathcal{L}=\mathcal{L}_{u}+\mathcal{L}_{f}+\mathcal{L}_{b},$ (14) $$

where $\mathcal{L}_{u},\mathcal{L}_{f}$ and $\mathcal{L}_{b}$ is defined as

$$ $\mathcal{L}_{u}=\frac{1}{N_{u}}\sum\left(\tilde{u}(x,0)-u(x,0)\right)^{2},\;\; \mathcal{L}_{f}=\frac{1}{N_{f}}\sum\left(\mathcal{N}(t,x,u,\alpha)\right)^{2}, \;\;\mathcal{L}_{b}=\frac{1}{N_{b}}\sum\left(\tilde{u}(0,t)-\tilde{u}(2\pi,t) \right)^{2},$ (15) $$

for $N_{u}$ points at initial condition, $N_{f}$ collocation points and $N_{b}$ boundary points.

## Appendix F Training Algorithm

We train the Transformer like following.

Figure: Algorithm 1 Training a Transformer

## Appendix G ICL of Transformers with Noisy Prior

In order to study the ICL capability of Transformers with noisy prior, we introduce four kinds of prior $\mathcal{D}$ like

P1 (noiseless) : $p(\mathcal{D})=p(\mathcal{U})$,

P2 (Gaussian noise) : $p(\mathcal{D})\sim\mathcal{N}(\mathcal{U},\sigma^{2}\mathbf{I})$,

P3 (salt-and-pepper noise) : $p(\mathcal{D})\sim p(s\cdot\mathcal{U})$ where $s=\begin{cases}\min(\mathcal{U})&\text{with probability }\frac{\gamma}{2},\\
\max(\mathcal{U})&\text{with probability }\frac{\gamma}{2},\\
1&\text{with probability }1-\gamma,\end{cases}$

P4 (uniform noise) : $p(\mathcal{D})\sim p(\mathcal{U}+U(-\epsilon,\epsilon))$ ($U$: uniform distribution).
We sample $D\cup T\sim p(\mathcal{D})$, where $\mathcal{D}$ is a noisy prior, and train the Transformer $\widetilde{u}_{\theta}$. We then test $\widetilde{u}_{\theta}$ with $\widetilde{D}\cup\widetilde{T}\sim p(\mathcal{U})$, demonstrating that the model can predict the true solution even when trained on noisy prior data. The experiment is conducted on reaction and convection-diffusion-reaction equations, which outperform other baselines, under three different noises: the Gaussian noise (P2), the salt-and-pepper noise (P3), and the uniform noise (P4). The standard deviation $\sigma$ of Gaussian noise is set to 1%, 5%, and 10% of the mean value of the ground truth solution. Additionally, for the experiment, the probe $\gamma$ for salt-and-pepper noise and the range $\epsilon$ for uniform noise are also set to 1%, 5%, and 10%.

Table: Table 7: The relative and absolute $L_{2}$ errors for the reaction and convection-diffusion-reaction systems using the P2 prior with varying levels of Gaussian noise $\sigma$, P3 prior with varying levels of noise probe $\gamma$, and P4 prior with varying levels of noise $\epsilon$ (1%, 5%, and 10%). For a comparison, the result of using P1 prior is notated.

Our model demonstrates robust performance across different types of noise injection as shown in Table 7. It shows our Transformer can perform ICL with zero-shot learning even if it is trained with inaccurate or noisy prior $D\cup T\sim p(\mathcal{D})$.

## Appendix H Experiments at PINN Failure Modes

Referring to Cho et al. () and Krishnapriyan et al. (), we test our method on PINN’s major failure modes: $\beta\in[30,40]$ with an initial condition $1+\sin(x)$ and $\rho\in[1,10]$ with an initial condition $\mathcal{N}\left(\pi,\left(\frac{\pi}{2}\right)^{2}\right)$. We have trained our model with this range with P1 prior and evaluate $L_{2}$ absolute and relative errors. The following are major results and solution profiles at failure modes.

Table: Table 8: The $L_{2}$ absolute and relative error at PINN failure modes.

Figure: (a) $\beta=30$
Refer to caption: extracted/5911933/Figure/Appendix/B_Failure_Modes/30.0_0.0_0.0.png

## Appendix I Hyperparameter List

The following hyperparameters were employed during the PINN-prior generation and the Transformer training process:

**Table 9: Hyperparameter configuration**
| Hyperparameter Name | Hyperparameter value |
| --- | --- |
| PINN training loss threshold | $1\times 10^{-3}$ |
| PINN maximum training epoch | 100 |
| PINN learning rate | $1\times 10^{-2}$ |
| Transformer layers | 3 |
| Transformer hidden size | 32 |
| Transformer learning rate | $1\times 10^{-5}$ |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - Can the authors investigate the method's performance on higher-dimensional PDEs?
- How does the computational cost scale with problem complexity?
- Will including physical constraints in the Transformer architecture improve performance?
- A discussion of potential real-world applications would strengthen the paper
soundness: 3
strengths: - The parallel drawn between LLMs learning from noisy internet data and SFMs learning from approximate PINN-generated data is insightful and well-motivated. Similarly the zero-shot inference approach without requiring knowledge of governing equations is equally important. 
- Clear mathematical formulation of the problem and the methodology. 
- Extensive experiments covering multiple scenarios. The baselines are sound, and the results are excellent. Especially notable is the "superconvergence" phenomenon observed (Remark 4.1)
summary: This paper introduced MaD-Scientist, a novel approach to solving partial differential equations (PDEs) using Transformer-based models trained on Physics-Informed Neural Network (PINN) generated prior data. The key innovation is demonstrating that scientific foundation models (SFMs) can learn effectively from approximate, low-cost PINN-generated data, similar to how large language models (LLMs) learn from noisy internet data. The authors focus on convection-diffusion-reaction (CDR) equations and show impressive zero-shot inference capabilities.
weaknesses: - The scope of the paper is very limited. Focus solely on 1D CDR equations may limit the broader applicability. Furthermore, no real-world applications are thoroughly explored (albeit a brief mention of semi-conductors). 
- Limited discussion of computational efficiency and scaling properties, with not much variations of the transformer architecture explored. 
- The appendices contain important information that should be in the main text like efficiency comparison (above point), training algorithm and some PINN failure modes. 

Based on the questions/suggestions below, I am willing to increase my score.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: - I feel like the idea behind Theorem 2.1 is rather intuitive. And it is a little out-of-place here. I don't see it referenced at all in the text. Could you motivate this theorem a bit more?
soundness: 3
strengths: - The paper is well organized and written.
- Encouraging results across datasets.
- To have a genearlizable zero-shot PDE solver trained from low-cost data would have high impact on the physical modeling community.
summary: This paper presents a scientific foundation model to solve PDEs with scarce data by training on low-cost data. This methodology is proven to be amenable to zero-shot learning with high performance, as evidenced by benchmark experiments across various kinds of PDEs.
weaknesses: - Figure 2. used a lot of page real estate to demonstrate concepts not unique to the paper. I would perhaps highly the innovation of this paper here.
- I feel like to brand this as a _scientific_ foundation model is slightly overselling it. To solve PDEs is hardly equal to _science_.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. Can you clarify specifically what you mean by 'zero-shot inference'? In Figure 2 (Inference Phase) it looks like you prompt your model with pairs $(x_i,t_i,u_i)$, and thus require data from the operator to be inferred in order to make predictions.

2. Can you explain your rationale for using the L^2 norm as an evaluation metric, as opposed to the H^1 norm or a more PDE-specific metric?
soundness: 2
strengths: 1. This work validates the hypothesis that foundation models are able to generalize even under noisy prior data, which has been thoroughly demonstrated in NLP but has been largely unexplored in scientific ML.

2. The model performs well even under substantial distribution shifts, which suggests that it is really learning to learn as opposed to memorizing the distribution of the training data.
summary: The paper presents a scientific foundation model to solve convection-diffusion-reaction equations in a zero-shot setting. Its main novelty is that the PDE solution data used to train the model is itself generated by a PINN solver, and is therefore subject to noise and measurement error. The authors demonstrate the the model can outperform baselines and make accurate predictions under parameter distribution shifts and time-domain extrapolation.
weaknesses: 1. The authors provide very few details on their architecture, other than that they use a transformer with self-attention and cross-attention. This makes it very difficult to meaningfully compare their model with other baselines. I would suggest adding a detailed description of the architecture to the appendix.

2. The class of PDEs considered is quite restrictive, since it is a family of 1D equations parameterized by relatively few variables. It doesn't seem too surprising that a sufficiently complex foundation model can learn this family of equations.

Typos: In Theorem 2.1, I think the first inequality should say that $\mathbb{E}[KL(p_{\widehat{\theta}}(\cdot | x, D_n) \parallel \pi(\cdot | x, D_n))] < \epsilon$ rather than $\widehat{\theta} < \epsilon$.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see below questions on the paper content:
1. line 078. The authors claim that their goal is "to predict solutions from observed quantities, such as velocity and pressure, without relying on governing equations". However, any data-driven ML model learns solutions from observed quantities without relying on governing physics. Therefore, why use a PINN with approximate PDE dictionaries while robust methods like neural operators are already there? 
2. line 269. In Eq. (12), define $\tilde{u}(x_T^{(j)},t_T^{(j)})$. Is it the same as $\tilde{u}(\alpha)$, or does it denote the observed solution field?
3. line 271. "...our model only needs observed quantities". The correct derivatives, as in the original CDR equation, are considered in Eq. (8). Thus, this sentence seems to be vague to me. Consider a large number of derivatives and their polynomials in Eq. (8) and then generate prior data to see the performance of the proposed method. 
4. line 302. it is written $D$ requires the solution u. Is $u$ the true observed quantity or the PINN-approximated data?
5. line 323. What value of $m$ is taken in the examples?
6. line 372. Is it supposed to be section 4.2.1?
7. Table 3. For 100% PINN-prior, the error in the convection equation is as high as 16%. This raises questions about the framework's ability to generalize effectively over a large number of unseen systems. 
8. Is there any reason similar to the above why the "TIME DOMAIN EXTRAPOLATION" study is not carried out on diffusion and reaction equations?
9. Fig. 3. This is a continuation of comment 7. See that the time extrapolation results for the convection equation is more than 50%, which again raises questions about the framework's generalization ability. 
10. Fig. 4. The solution field on the right is too smooth to draw a conclusion about the extrapolation ability.
11. line 480. Why is the $\beta$ = $\nu$ = 0 taken? I think a non-zero value will make the problem more challenging.
12. line 483. It is not fair to say "it can generalize to unseen PDEs" since the model is pre-trained on the CDR PDE, which is nothing but a mixture of convection, diffusion, and reaction.
13. line 487. How many training samples are taken? 
14. Table 4. A central goal of the paper is to achieve zero-shot inference for predicting PDE solutions. However, when the model is pre-trained on a mixture of reactions but tested on different reactions separately, the error increases significantly beyond 6% in most of the cases. This indicates that the proposed framework may also need fine-tuning or few-shot learning.
soundness: 3
strengths: 1. This paper integrates several non-trivial concepts of the scientific foundation model, approximated prior data, Bayesian inference, and pre-training that are already well-established in scientific machine learning. 
2. The concept of using PDE dictionaries in the PINN-prior for pre-training is original. 
3. This paper tries to address the challenges of the inherent noisiness of the data used for training.
summary: This paper introduces an in-context learning architecture for scientific foundation models. The proposed method incorporates Bayesian inference into the prediction by pre-training on PINN-based low-cost and noisy approximated data, which is referred to as the PINN-prior. The PINN is initialized as a general PDE with derivatives up to a certain degree. Initializing the PINN with a general PDE helps to overcome the fact that the underlying governing equations of physical systems are often unknown; instead, only the observations of the underlying systems are available. At the base, the method uses a transformer architecture, where a self-attention mechanism is used to capture relationships among known data points, and a cross-attention mechanism is used to extrapolate and infer solutions for unseen points without fine-tuning.
weaknesses: According to me, some parts of the paper need further clarification. Please see the comments below:
1. The underlying concept of using approximated prior data is a concept that has been introduced previously in scientific machine learning. This concept already exists in multi-fidelity learning, where a low-fidelity solver is first trained on a large number of available low-fidelity data, and the approximate predictions of low-fidelity solver are used in conjunction with the small high-fidelity data to train a fast high-fidelity emulator [1,2]. Except for the difference between the purely data-driven nature of the multi-fidelity learning methods and the partial physics-constrained nature of the proposed framework, I do not see any significant improvement.  Further clarification is required to understand the novelty of the proposed method.
2. The authors mention the "scientific foundation model" at several places in the paper. The purpose of the foundation models is to pre-train on a large number of datasets, which in this case is analogous to multiple PDEs, and then efficiently fine-tune or directly predict on unseen PDEs without sacrificing prediction accuracy. However, the authors consider only a simple CDR equation for pre-training and special cases of the CDR equation like convection, diffusion, and reaction equations for testing. This raises concerns about whether the methods can genuinely aid in foundation modeling, highlighting the need for more rigorous case studies to properly assess the effectiveness of the proposed PINN-prior framework. To be able to clearly appreciate the method's potential as a foundation model, in addition to the CDR equation, it is necessary to simultaneously pre-train on a few of more complex PDEs like the Kuramoto–Sivashinsky equation, Schrodinger equation, Korteweg–De Vries equation, Navier-Stokes equation, Burgers' equation, etc. and test on remaining PDEs.
3. In the PINN dictionary, the authors consider the exact derivative terms of the CDR equation. This is unlikely to be available in a practical scenario, which is also mentioned by the authors in the motivation. Thus, the performance of the PRIOR in the presence of a large number of dictionary functions needs to be inspected to understand further the potential of the proposed framework for scientific foundation modeling. The authors need to conduct additional experiments with a larger and more diverse set of dictionary functions to better simulate real-world scenarios where exact terms may not be known.
4. Many scientific foundation models have already been developed for pre-training and testing on multiple physical systems [3-6]. However, the authors do not mention them in this paper. Given the existing literature on foundation models, the authors should consider comparisons with a few of the existing scientific foundation models. This will further improve the contents of the paper.

[1] Meng, Xuhui, and George Em Karniadakis. "A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse PDE problems." Journal of Computational Physics 401 (2020): 109020.\
[2] Howard, Amanda A., et al. "Multifidelity deep operator networks for data-driven and physics-informed problems." Journal of Computational Physics 493 (2023): 112462.\
[3] McCabe, Michael, et al. "Multiple physics pretraining for physical surrogate models." arXiv preprint arXiv:2310.02994 (2023).\
[4] Tripura, Tapas, and Souvik Chakraborty. "A foundational neural operator that continuously learns without forgetting." arXiv preprint arXiv:2310.18885 (2023).\
[5] Hao, Zhongkai, et al. "Dpot: Auto-regressive denoising operator transformer for large-scale pde pre-training." arXiv preprint arXiv:2403.03542 (2024).\
[6] Herde, Maximilian, et al. "Poseidon: Efficient Foundation Models for PDEs." arXiv preprint arXiv:2405.19101 (2024).

Meta-review: This paper applies LLMs to scientific foundation models (SFMs) for solving PDEs. Technically, the algorithm collects low-cost physics-informed neural network (PINN)-based approximated prior data in the form of solutions to partial differential equations (PDEs) constructed through an arbitrary linear combination of mathematical dictionaries, and utilize Transformer architectures with self and cross-attention mechanisms to predict PDE solutions without knowledge of the governing equations in a zero-shot setting. Experimental results were provided for one-dimensional convection-diffusion-reaction equations, supporting the conjecture that SFMs can improve in a manner similar to LLMs.

There were adequate discussions during the rebuttal, and the reviewers acknowledged the notable efforts from the authors during the review and AC discussion period. However, there are still general concerns:

1) The class of PDEs considered is restricted, and the results of the paper seem to be only a marginal improvement compared to previous works on SciFMs. The claim to introducing a scientific foundation model that is pre-trained on PINN-simulated data is an overclaim: while the idea of foundation models is to learn multimodel features from multiple pre-training physical systems, each involving thousands of samples with varying PDE parameters, the proposed framework is trained on only a handful of PDEs of similar types.

2) The superiority in terms of prediction accuracy over the existing scientific foundation models is not provided. Besides, the results based on the PINN prior appear to be poorer than those based on numerical priors, often showing more than 10% relative error. This raises concerns about whether the use of low-cost PINN-prior data is justified given the low prediction accuracy and can, therefore, actually outperform existing scientific foundation models.

Therefore, given the competitiveness this year at ICLR 2025, the decision is to reject this paper. Hopefully this doesn't disappoint the authors - the paper has been improving during the rebuttal, but more efforts are needed to address the aforementioned concerns for future venues. There are also many appendices newly added to the rebuttal that the authors may consider reorganizing for future versions of the paper.

Decision: Reject