Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior
Authors: Shashank Subramanian
Peter Harrington
Kurt Keutzer
Wahid Bhimji
Dmitriy Morozov
Michael W. Mahoney
Amir Gholami
Venue: NeurIPS 2023 poster (2023)
Keywords: scientific machine learning
scaling
transfer learning
neural operators
foundation models
TL;DR: We study the scaling and transfer learning performance of neural operators on different PDE systems to build a pathway towards foundation models for scientific machine learning.

Abstract: Pre-trained machine learning (ML) models have shown great performance for a
wide range of applications, in particular in natural language processing (NLP)
and computer vision (CV). Here, we study how pre-training could be used for
scientific machine learning (SciML) applications, specifically in the context of
transfer learning. We study the transfer behavior of these models as (i) the pretrained
model size is scaled, (ii) the downstream training dataset size is scaled,
(iii) the physics parameters are systematically pushed out of distribution, and (iv)
how a single model pre-trained on a mixture of different physics problems can
be adapted to various downstream applications. We find that—when fine-tuned
appropriately—transfer learning can help reach desired accuracy levels with orders
of magnitude fewer downstream examples (across different tasks that can even be
out-of-distribution) than training from scratch, with consistent behaviour across a
wide range of downstream examples. We also find that fine-tuning these models
yields more performance gains as model size increases, compared to training from
scratch on new downstream tasks. These results hold for a broad range of PDE
learning tasks. All in all, our results demonstrate the potential of the “pre-train and
fine-tune” paradigm for SciML problems, demonstrating a path towards building
SciML foundation models. Our code is available as open-source.

Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior
Shashank Subramanian Lawrence Berkeley National Lab www.mnkswwuaaaaaannn11.....
Peter Harrington Lawrence Berkeley National Lab pharrington@lbl.ccm gov
Kurt Keutzer
Wahid Bhimji
UC Berkeley keutzerdeecs. berkeley.cecc edu
Lawrence Berkeley National Lab wbhimji@lbl.e gov
Dmitriy Morozov
Michael W. Mahoney
Lawrence Berkeley National Lab dmorozov@1bl. gov
ICSI, LBNL, and Department of Statistics mmahoney@stat. berkeley edu
Amir Gholami
ICSI, UC Berkeley
amirgh@berkeley.com edu
Abstract
Pre-trained machine learning (ML. models have shown great performance for a wide range of applications, in particular in natural language processing (NLP) and computer vision (CV). Here, we study how pre-training could be used for scientific machine learning (SciML) applications, specifically in the context of transfer learning. We study the transfer behavior of these models as (i) the pre- trained model size is scaled, (ii) the downstream training dataset size is scaled, (iii) the physics parameters are systematically pushed out distribution, and (iv) how single model pre-trained on mixture of different physics problems be adapted to various downstream applications. We find that that-wher when fine-tuned ampropriately-transtii learning can help reach desired accuracy levels with orders magnitude fewer downstream examples (across different tasks that can even be vittof--iitribbtion) than training from scratch, with consistent behaviour across wide range of downstream examples. We also find that fine-tuning these models yields more performance gains model size increases, compared to training from scratch on new downstream tasks. These results hold for a broad range of PDE learning tasks. All in all, our results demonstrate the potential of the "pre-train and fine-tune" paradigm for SeiML problems, demonstrating a path towards building SciML foundation models. Our code available as open-source at [1].
Introduction
Foundation models have received considerable interest recently [3]. This terminology refers to certain models that are trained on extremely large and diverse quantities of data and applied to a wide range of tasks. Rather than being designed for any single task, foundation model serves "prior"
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Create and pre-train on diverse PDE systems Vary/Sample inputs (PDE coefficients, source funstions, ... Include multiple differential operators, predict PDE solution
Adapt pre-trained model to different
downstream PDE tasks
Solve multiple systems using the same pre-trained model, outperforming training from scratch
V K Vu V Vu
POET
Neural Operator
Pre-trained Neural Operstor
PDE
Inputs F.K. V....
Outputs
Inputs
Outputs
1,K, v,
Figure 1: Our setup consists of creating diverse training datasets, sampling both PDE coefficients and source functions simultaneously with different PDE operators and input data (coefficients, sources) distributions for pre-training. given these inputs and the ground truth solutions (computed through PDE solvers). The pre-trained model is then adapted with minimal fine-tuning (zero-shot or few-shot), and it is used in various downstream lasks (PDE systems) that can be in-domain or out-of-domain from the pre-training datasets. The pre-training with multiple solution operators allows the same model to transfer to several very different systems. For instance, PDE 2 (Helmholtz) manifests highly oscillatory solutions compared to PDE 1 (Advection Diiffo-in)ff or PDE 3 (Poisson's s). We further characterize the scaling and transfer properties of this model as a function of downstream data scale and model size scale.
neural operator is then pre-trained to predict the PDE solutions
or "foundation" upon which other models can be built. It does so by using transfer learning (TL) methods to fine-tune or adapt the foundation model to a wide range of downstream tasks, using minimal additional data for each additional task. Perhaps the most well-known foundation models are tre-trained large-language models (LLMs) such as BERT [11] and the GPT models [38. 39, 5]. The scaling with respect to the amount of data, the size of the model, and the amount of compute [23, 20, 21] is key to the training of these models. An important aspect of trained foundation model the ability perform tasks scemingly different than those for which was trained by leveraging shared features across the training tasks. This approach to model development is quite different than the traditional approach of training one-off model from scratch for each specific problem and each specific dataset. Naturally, it is of interest how broadly this methodological approach can be applied.
Scientific machine learning (SciML) [45] is area that combines tools from ML and scientific computing to address domain-specific scientific and engineering challenges. It holds promise to drive the next wave of data-driven discovery the physical and engineering seiences. Recent work has highlighted the promise [42, 32, 30, 24] as well as some of the many challenges [28, 12] of developing SciML models-in general well as with the traditional one-off learning approach. Many SciML models emulate physical systems described by Partial Differential Equations (PDEs). For example, Physics-Informe
Neural Networks [42, 28] impose the PDE as soft penalty in the loss function. However, they are restricted to solving a single instance of the PDE. The Neural Network (NN) needs to be retrained for each new set of PDE physies coefficients, sources, and/or initial/boundary conditions (IC/BCs). Subsequent models have been developed to learn the full solution operator [30, 32] by training across different coefficients (and/or initial and boundary conditions). These neural operators learn mappings between two function spaces from a finite collection of input-output pairs (that represent the cocfficients/initial or boundary conditions as the input and the PDE solution function the output). This makes them more general and versatile in emulating any PDE system. However, with new coefficients/sources or new differential operators, they too need to be retrained from scratch.
In this paper, we adopt and evaluate the methodology that has been applied successfully in CV and NLP to develop foundation models, with the goal of determining whether such a model is even possible for SciML problems. particular, we provide an extensive analysis of the scaling and TL behavior of neural operators trained on diverse training datasets from multiple PDE systems. An important aspect this approach is to explore several dimensions that include the model (architecture and scale). data (diversity and scale), training recipes (pre-training and fine-tuning), and out-of-

---

distribution (OOD) transfer behaviour. For transfer behavior, a major difference from CV/NLP is the diversity of scales, features, and solution behaviors across PDEs. For example, while OOD shifts in CV/NLP typically involve things like different color distributions or semantic contexts, in SciMI,, it is possible to get significantly different behavior of a PDE solution as physics parameters and conditions change that manifest as novel features (for example, increased emphasis on very different frequency scales of the solution causing significant shifts in detail, sharpness, and other image attributes). For LLMs, given the maturity of the NLP community within ML, these dimensions are well-explored. For SciML problems, in contrast, all these dimensions are open questions. Here, we explore several of these questions. We do so in the context of specific model architecture, namely, the Fourier Neural Operator (FNO), as prototypical SciML model that has demonstrated promising results modeling PDEs across wide range of scientific applications [30, 37, 49, 53, 29]. We focus the scaling and TI. behavior of the FNO on common PDE systems that include Poisson's is Advection- and Helmholtz PDE systems. These systems underpin wide range of physical systems: fluid flow systems; biological simulations; wave propagation systems; and many others.
See Fig. 1 for schematic summary of our methodological approach. Our main results demonstrate the potential of the "pre-train and fine-tune" paradigm for SciML problems, demonstrating a path towards building SciML foundation models. In more detail, our main contributions are the following.
Pre-training dataset generation. We develop a large suite of datasets, and we train our models on data where all the variables (inputs) of any PDE operator are sampled. This is an important step towards developing NNs that can generalize across a variety of downstream tasks, and extends several previous works, including the original FNO [30], where certain inputs are kept fixed. Not sampling can trivially push the neural operator OOD if, e-g., the source function was changed. We study transfer learning to both in-domain and out-of-domain distributions, characterized by different samplings of PDE coefficients (diffusion, advection, wavenumbers, etc.) and inhomogeneous source functions. We emphasize the construction of pre-trained datasets with sufficient diversity as well as normalization strategies, without which we observe significant degradation of performance.
2. Downstream (target) data scaling. We study the effect of the scale of downstream (target) data in TL performance from the pre-trained model. Here, we assume that a large amount of data is available for pre-training, and data are limited for the downstream tasks (as in many scientific examples); and
interested in reaching desired accuracy levels with the least amount of additional downstream data. We consider both zero-shot and few-shot TL: zero-shot is the direct evaluation of the pre- traitra model on the downstream dataset; and few-shot involves using O(10) downstream data to fine-tune the pre-trained model. While few-shot typically refers to prompting with in-context examples during inference in NLP, here define it as fine-tuning the model with a few examples. We observe that TL from the pre-trained model can lead to significant performance gains over training the model from scratch on the downstream (target) data, with orders of magnitude less data needed to reach desired accuracy level (see Fig. 2). We observe this gain over wide range of data scales, until we enter the "large target data" regime (as much data as pre-training), where we observe similar accuracies for TL as training from scratch.
Model (parameter) size scaling. We study the parameter scaling of the model by scaling our model sizes from 64K to 256M parameters (a multiplicative factor of 4K). We observe an error saturation at small model sizes (due to insufficient model expressivity) that monotonically drops as we increase the model size. While both fine-tuned models and models trained from scratch exhibit gains with increased model size, we observe that fine-tuning achieves greater performance gains with parameter scaling (see Fig. 3).
Transfer learning behavior over underlying physics. We study the effect of varying the underlying physics in the target domain. In SciML (unlike traditional non-scientific ML), there are typically fundamental constraints such as conservation laws that govern the behaviour of the solution. In some cases, may even have access to parameters of the underlying physical constraints, and thus a physical understanding of the distribution of the data. It is natural, and necessary, to systematically quantify the effect of these parameters our downstream tasks go OOD, as this provides a good way to test the (OOD) generalization capability of pre-trained models for SciML applications. We find that for in-distribution TL, the pre-trained model can significantly outperform a model trained from scratch, irrespective of how many new data examples were used for fine-tuning, until the large target data regime (e.g., see Fig. 4a), showing orders of magnitude better accuracies than training from scratch. We also observe these gains for downstream tasks that are moderately OOD, with sho- fine-tuning providing again orders of magnitude better
3

---

accuracies (see Fig. 4b, Fig. 4c). As we systematically go further OOD (see the quantification in Tab. 1). we observe the performance gains expectedly reduce, with more significant drop in the low data regimes (e.g., see Fig. 4d).
Transfer learning behavior over multiple operators. We study the effect of simultaneous pre-training on multiple PDE systems that exhibit qualitatively different solution behaviors (e.g.. Poisson's and Helmholtz operator solutions show very dissimilar patterns, given a source function). We include the coefficient/source functions for all the operators as inputs to the model, with zero values if those terms do not exist in a given PDE instance. During inference, the zero inputs restrict the neural network to make predictions for the correct operator (see $4 for details). Among other things, we show that the same model pre-trained on different operators retains its gains across different downstream tasks (see Fig. 5). paving the way for it to be used in the foundational sense.
2 Related work
In recent years, there has been widespread interest in modeling PDE systems with neural operators or operator learning in broad variety of science and engineering applications [32. 30, 27, 41, 33, 37, 25, 2, 49, 4, 29, 26]. Following the success of TI. in CV and NLP tasks [54, 43, 48, 13, 36, 40, 22], there have been several investigations into how TL. can be leveraged for SciML problems involving differential equations. Most have focused on applications of Physics- Inforredd Neural Networks (PINNs) [35, 19, 15, 8, 18, 17, 50, 14, 10, 31], where models can be fine-tuned or adapted using a physics-based loss function determined by the specific target PDE/ODE system, Another common theme evaluating how well TL can account for the diversity of geometries [6, 47, 50, 16) and discretizations [34, 7, 44, 46] found in scientific computing. Recently, attention has also been devoted to TL for neural operators in SciML, where some works have explored certain components of our analysis in isolation. This includes studies on the TL performance of DeepONet [51, 16, 55] and FNO [31, 9J. where one of either the target domain, PDE coefficients, or PDE source functions shift, varied. In [9], the authors evaluate different model and training dataset sizes for different neural operator architectures, but their analysis is mostly limited to diabii-distribution experiments and single operators, and they do not analyze the fin-ttning performance of such models (particularly on OOD data). The works above have not considered the pre-training regimes over a diverse set of physical systems and most have not characterized their TL function of data and model scaling, nor included multiple PDE operators during training. To the best of our knowledge, the work of [52] (which has appeared concurrent this work) is the first (along with ours) consider the TL potential of neural networks across operators and data distributions. There, the authors adapt in- in-context learning (from LLMs) to solve differential equations with transformer-basee models. The main difference with [52] is that the authors focus on in-context learning which requires prompting (with up to five example demos) to solve OOD tasks including different differential equation coefficients and operators than those seen at train time (similar to our work). In contrast, we focus on TL with zero-shot and few-shot learning through fine-tuning. Thus the two approaches (TL through fine tuning vs in-context learning) are complementary but different. We also note that their investigation was performed at a much smaller scale (from both model and data perspective), and was limited to simpler dynamical systems, as compared to the scales tested in our work. Overall, the prior related work provide important initial results on the interpolation and extrapolation behavior of neural operators in the context of differential equations, and/or limited investigation into the behavior model and dataset sizes are increased. However, none of them consider these aspects simultancously with a more diverse pre-training corpus (by varying all input variables such as source functions and PDE coefficients and/or including different operators for pre-training), which is closer to methodology adopted by CV and NLP in the development of their foundation models, with emphasis the importance of characterizing scaling properties 21].
3 Methods
Pre-training foundation model requires that we first collect a large amount of diverse training data and then train a base model that could subsequently be used with TL for downstream application. There are several possible methods for TL of the foundation model. One approach is in-context learning, where the foundation model is prompted with few-shot input-output examples for the downstream problem followed by the target input. The model then sees these examples and learns how to compute the target output. This is the approach used by modern NLP models such as
4

---

GPT [38, 59 5] as well as the work of [52]. While this approach is very useful for cases with very few training datapoints available, often better to fine-tune a foundation model for downstream task, when one has access to more downstream training data. This is also supported by NLP models and it often results better performance- if enough training data available. We focus on the latter setup, and we study how the TL. performance behaves for different problem setups. Our goal is to understand the different moving parts associated with training a foundation model for SciML applications and, specifically, the impact of model scaling, dataset size, and different physics involved in the problem. Below, we discuss (i) the different physics operators considered and our pre-training setup that includes the training dataset generation, (ii) NN model architecture setup for training and inference, and (iii) performance metrics.
PDE/physics system setup. We consider three PDE systems that are common building blocks for many scientific application: 2D Poisson's s; PDE systems can be formulated as follows:
and Helmholtz equations. These
1. Poisson's 's (SYS-1): We consider prototypical elliptic system with periodic boundary condi- tions in domain Ω = [0, 1|2:
dii KVu f in ?
(1)
where u(x)) is the solution (state) function, f(x) is source (forcing) function, and K is the diffusion coefficient tensor. We use K to quantify the physics of this system.
2. Ditection-Diffusian (SYS-2): We also consider a steady-state advection advection-diffusion diffusion equation
that illustrates competing physical processes (advective and diffusive processes) through two differential operators. We use periodic boundary conditions in domain f [0, 1J2:
div KV + V Vu f in S2
(2)
where u(x) is the solution (state) function, f'x)) is a source (forcing) function, K is the diffusion coefficient tensor, and v is the velocity vector. To quantify the competing advective/diffusiv scales of this system, we define the ratio of advection to diffusion as V Vwl.V///lllll K Vwll--
3. Helmholtz (SYS-3): Finally, we also consider the inhomogencous Helmholtz equation with periodic boundary conditions in domain Q [0, 1J2. We take this as an example challenging system that can exhibit high-frequency oscillatory spatial patterns that can be difficult to generalize. This system is formulated as:
-Au + wu f in Ω
where u(x) is the solution (state) function, f(x) is a source (forcing) function, w > 0 is the wavenumber used to quantify the underlying physics of this system.
Data setup. For the above PDE systems, we are interested in (i) large and diverse training dataset for pre-training and (ii) several downstream datasets (tasks) to quantify the TL performance. Given that we can solve these PDEs numerically, we can generate diverse set of training and testing datasets in controllable fashion by varying the different parameters in these PDEs. In particular, we vary the following parameters for dataset generation:
(i) Source function sampling: We sample different source functions f ~ s). where S is a distribution that generates diverse and heterogencous functions. (see Fig. and Fig. A.I for examples). Here. S represents a parameterization of the source function as a linear combination of ny radial (Gaussian) basis functions (0; "" where
(x(x)) d(x - 244 is a Gaussian function centered at grid point K;. Specifically: f(x) T dz(2)p, with p = {p. 12 as the parameterization vector. The spatial proñile controlled by the standard deviation of the Gaussian function, is preset to a small value encourage high variability. Examples are sampled by uniformly randomly sampling p; 44, 1). We further introduce heterogeneity by controlling the sparsity of p (s defined as the number of non-zero components; see Appendix
$A.1 for full details).
(i) PDE coefficient sampling: SYS-1, we sample diffusion coefficient tensors K ~ IC(A). where IC is a distribution that generates varying scales of anisotropy and spread in the diffusion process: K- R DR with D = diag(1, e) and R rot(@), where is an eigenvalue of the tensor that controls the anisotropy and extent of diffusion and rot(O) is a rotation matrix with angle 0 (0, 2xx) that controls the general diffusion direction. In SYS-2, we additionally also sample the velocity vector direction from (0, 2T). The ratio of advection to diffusion y is changed by scaling the velocity. In SYS-3, we sample the wavenumber w uniform
5

---

integers. We visualize the sampling process and resulting solutions with different ranges of underlying physics in Fig. A.I.
For each of the problems that we consider in the results section, we generate 2 input-output samples (pairs) of data, where the inputs include the source as well as any PDE coefficients (K, W), along with 21 validation and testing samples each. The validation dataset is used for hyperparameter optimization, and the testing dataset is used for quantifying model performance. The pre-trained model is trained on the 2 training examples.
We then perform different experiments to evaluate how this model can adapt/TL to different down- stream tasks whose data could come from the following distributions: (i) same distribution as in the pre- -taininnn dataset (i.e.. different input/output pairs but drawn from the same distribution of PDE coefficient//sources)): and (ii) the harder task of adapting/TL. to a downstream problem that can have slight/large deviation from the dataset used pre-train the model. For the latter, we create the OOD) data by keeping the PDE operator the same as the pre-training task, but sample the coefficients from a different range as in the pre-training. Given this dataset, we then study the TL behaviour for each case (both within distribution and OOD) by scaling both the downstream dataset size, as well the model architecture size which is discussed next.
Pre-training method for training and inference. The inputs to our model are 2D spatial functions discretized at h x w and represent the sources and PDE coefficients.' These input discretized functions are batched together to form an input tensor in hxx The output of the model is the numerical solution of the PDE in gx For the model architecture, we consider the FNO (details in Appendix $A.2).. This model bears similarities to both vision transformer- like architectures (fixed image/feature resolution across the depth of the network) and convolutional architectures (successive global convolutions facilitated via FFTs). FNO is also a good choice for our setup as the problems consider all have periodic boundary conditions (the FNO can also be adapted for non- non-periodic boundaries). The main modification that we make to the FNO is to incorporate a per-instance normalization layer in the model. We found that this is a critical component as the norm of the input data spans a wide range of values (up 100x for our dataset). Please see Appendix $A.5 for details. Furthermore, we consider & loss function for pre-training. That is, the model is trained predict the output solution given the input by minimizing a mean-squared error loss between the prediction and the ground truth.
To test how this pre-trained model can adapt to different downstream applications we consider the following cases. First, we consider zero-shot adaptation where the pre-trained model is tested downstream application without any fine-tuning. Second, we consider few-shot fine-tuning, where the pre- tre-trained model can be fine-tuned on the downstream training dataset. We consider varying sizes for this downstream dataset size. Ideally, we prefer the pre-trained model to achieve good performance with few examples as needed from the downstream application. We perform an ablation study by training model from scratch on the downstream dataset alone to see how much gain can be achieved by using a pre-trained model. We also test how the pre-training adaptation performance changes as we scale the model size. This is motivated by observations in NLP [23] where we expect larger models to show better adaptation performance. To do the model scaling, we focus on two model hyperparameters: the embedding dimension d and the number of Fourier modes used in the FNO m. (See Appendix $A.2 for the details.) We first fix d = 16 and scale m E {4, 16), then fix m = 32 and scale d E {32, 128] to approximately increase the parameter count 16x in each scaling experiment from 64K to 256M parameters.
The main limitation of our work is that we focus on only the FNO model, whereas several other architectures (e.g. VIT models, DeepONets) exist. As discussed in $1, the FNO has been applied to a wide range of applications in SciMI. and is representative of performant SciML architecture- restricting our analysis one model architecture makes the scope of this study more feasible. However, this analysis needs to be done across other models architectures, as well.
4 Results
Our main results are the following. We demonstrate that pre-training a model on a diverse corpus of data and then fine-tuning it on downstream tasks leads to significantly better performance than training
Constant coefficients simply replicated across the h dimensions.

---

Pretrain
Downstream SYS-115, 10)
Pretrain
Downstream
SYS- 5)
DYS-2(0.25 1
SYS-2(1.
2)
02
0.4
0.6
08
10
22
14
1.6
1.8
2.0
Diffusion tensor eigenvalue
Advection-diffuse ratio
Testing errer (relative F3)
109
Training from Scratch
Testing enror (relative #:)
10
Training from Scratch
TL Pre-trained
Pre-trained
10-1
10-1
10
32 64 128
2K 4K
328
Dechot
32 M. 255 512 1K 2K 18K 32K
Number of downstream examples
Number of downstream examples
(a) SYS-1(5,10) pre-trained from SYS-1(1,5)
(b) SSS--2(11))) pre-trained from SYS 1
Figure 2: Addressing (Q1). Testing error as a function of downstream examples for SYS-1 and S-S-2. We visualize the distribution of pre-training and downstream dataset physics at the top to illustrate (and quantifiy) the extent of distributional shifts. We observe excellent zero-shot and few-shot Tl. "T.performance of the pre-trained model despite the modest OOD shifts and in medium-data regimes about 100x increase in data efficiency. We observe diminishing returns from pre-training at the large-data regime (O(2 examples), which has as many examples used in pre-training.
model from scratch. This holds even when the downstream data falls outside of the pre-training distribution, including when different physics models are combined. The advantage of pre-training is especially pronounced when the downstream data is limited, which is the most significant setting in practice, motivating the creation of foundation models for scientific machine learning.
To justify these conclusions, we focus four key questions. What is the effect of (Q1) downstream dataset size and (Q2) neural operator model parameter size on TL? What is the TL behavior of the neural operator (Q3) over the underlying physics and (Q4) over multiple solution operators?
(Q1): Downstream dataset scaling, For SYS-1, we consider the pre-training system SYS- ((1,5, with diffusion constructed by sampling eigenvalue
(11 of the diffusion tensor K. See Fig. A.1 for visualizations. This represents near isotropic to 5x anisotropic diffusion. We use ~ (5, 10) the downstream dataset SYS-1 This represents 5x-10x anisotropic diffusion and is moderately quantify the OOD effects in (Q2), we use this specific test-case to illustrate the effect of the size of the downstream dataset. We plot the behaviour of testing error as a function of downstream examples in Fig. 2a. We train FNO model for each number of downstream examples (x-axis on the plot) starting from "scratch" (random initialization) as well as from the pre-trained model parameters, with tuned hyperparameters for each experiment (see details in Appendix $A.44).
(OOD) from the pre-training dataset. While we systematically
We illustrate the extent of distributional shift between the pre- pre-training and downstream datasets through the range of diffusion tensor eigenvalue e-in this test case, a modest shift with no overlap (but relatively close). The testing error monotonically decreases as more downstream data is used for training. we expect. The zero-shot TL shows excellent performance despite the moderate OOD shift of downstream examples. When training from scratch, we define "zero-shot" predictions as the output of the model with random initialization. With "few-shot" learning (O(10) downstream examples), we observe a consistent performance increase over training from scratch. Given a desired error for downstream performance, TL from the pre-trained model can require orders of magnitude less data-for example, a desired error of 1F-2 needs only about 64 downstream data examples for fine-tuning, whereas training from scratch requires 8K (about 100x more) examples to reach the same accuracy level. We further explore this in Appendix $ and find the pre-trained model generally saves OLK 10K) data compared training from scratch in the few-shot learning setting, and outperforms training from scratch at all scales. Finally, with greater amounts of downstream data, the pre-training provides consistent performance gains until we enter the "large target data" regime, where the number of fine-tuning examples approaches the size of the entire pre- pre-training dataset, and we see diminishing returns from pre-training. We repeat this experiment for S-S-2, using the rates
following test-case: the pre-training system SYS-2(0.22, 1) consists of advection-to-diffun E (0.22 1), representing about x-5x diffusion (relative to advection). For the downstream test, we use modest TL with SYS-2(1,2) with y ((1 representing 1x-2x advection
7

---

64K, trairing from scratch
1M. training from scratch -
-
16M, trai ning from scratch
- 256M, training from scratch
64K, from pre-trained
from pre-trained
16M,
pre-trained
256M from pre-trinned
SYS-1
SYS-2
Testing error trelative fg)
10
Testing error error (relative f2)
109
10-1_
10-1
10-2
10-3
10-2
0-shot
16. 32 64 128 256512 1K 2K 4K BK 16K32K
0-shot
16. 32 128256 512 1K 2K aK 16K32K
Number of downstream examples
Number of downstream examples
Figure 3: Addressing (Q2). Model size scaling for SYS-1 and SYS-2 2 from 64K to 256M param- eters for medium OOD test-cases. While finetuning consistently improves the model performance and data efficiency, we observe higher errors for small parameter regimes at 64K due to insufficient model capucity. The performance guins are significantly boosted through finetuning with a larger model set sizes monotonically up to 256M parameters.
(relative to diffusion). We visualize this shift in Fig. 2b (top) and also show the testing error as function of downstream examples. Both experiments reveal the same trend: TL delivers much higher performance, which improves with fine-tuning up to a point of diminishing returns.
(Q2): Model (parameter count) scaling. As described in model setup, vary the embedding d and maximum Fourier modes m to approximately increase the parameter count 16x in each scaling experiment from 64K to 256M parameters. For models trained from scratch, we repeat the data scaling experiments for each parameter count. For the pre-trained model, we first identify the ideal hyperparameters (through grid-search hyperparameter tuning) for each model scale and repeat the above training experiments. We visualize the testing errors as function of downstream examples used for SYS-1 (pre-training dataset used: SYS-I(1,5) signifying moderate OOD) for the different model scales in Fig. (left). At the 64K parameter regime, the model capacity is insufficient, with large errors (greater than 1B-2) for either training recipe across the whole range of downstream example counts. As we move to larger models, both training from scratch and fine-tuning show higher performance that increase with more examples. Fine-tuning the pre-traue pre-trained model boosts its performance, compared to training from scratch, as we increase the model scale and particularly across a wide range of downstream example counts (with 256M parameter model showing the least errors). We repeat the model scaling process for Advection-Diffun SYS-2(0.2.. 2. 1) with moderately OOD SYS-2(1,2) for downstream) and observe similar trends in Fig. 3 (right).
SYS-2 (pre-training dataset
Table Different downstream datasets and extents of overlap with the pre-training dataset for SYS-1 and SYS-2, controlled by extent of anistropy (eigenvalue e) in diffusion tensor for SYS-I and amount of advection relative to diffusion (ratio v for SYS-2. Pre-training
Downstream
Shift
1 U(1,
None
SYS-I2257.5)
Mild
SYS-I(L.5): e e u(1
Med
SYS-1(10,20):==<1((1,
Large
SYS-2(0.2,0..4) V =0000.,,..4
None
SYS-2(0.2,0.4):: V c0(2.2
Mild
SYS-2(0.2.I): (0.2211
SYS-2(1,2) v E
Med
SYS-2(2,5) v E~(2,5)
Large
(Q3): TL behavior over underlying physics. We test both in-domain and out-of-domain physics effects by constructing downstream datasets that systematically deviate from the pre-training dataset. For SYS-1, we sample different ranges for with varying overlap with the pre-training dataset. Similarly, for SYS-2, we use different ranges of adveetion-to-diffusse ratio V showing different overlap. We highlight these systems (downstream and pre-training) in Tab, for the two PDE systems.
8

---

Downstream
Pretrain 5YS-1(1.
Pretrain
Downstream SYS-16 5. 7.5)
SYS-101. 51
5)
YS- 101, 5)
1.0
15
2.0
255
30
3.5
4.0
5.0
Diffusion tensor eigenvalue
Diffusion tensor eigenvalue
Testing errer (relative E3)
10a
Training from Scratch Pre-trained
Testing enror (relative (.)
109
Training from Scratch Pre-trained
10-1
20
10-2
10-3
10-3
Destat
32
512 1K
18K 32K
64
12n
512 downstream examples
256
1bc 32K
Number downstream examples
(a) SYS S (1,2.55
(b) 12,-1(2.5,7.55)
Protrain
Downstream SYS-115, 10)
Protrain
Downstream SYS
SYS 1(3,
SYS 1L.-11L.00
2(10, 20)
10
2.5
5.0
7.5
10.6
12.5
15.0
17.5
20.0
Diffusion tensor eigervalue
Diffusion tensor cigenvalue
Testina error (relative [.)
10"
--- Training from Scratch
Testing error (relative)) (.)
100
-
Training from Scratch
TL from Pre fre-traincd
from tre-trained
10-1
10-
10-3
0-shot 5
16
32 64 125 256 512 16 2K 4K 10K 32x
9.hett 16 32 66 125 256
512 1K 2K 4K B6 J6K 22X
Number of do sowsstream examples
Number of downstream examples
(c) SYS-1((510))
(d) SYS-1(10,20))
Figure& 4: Addressing (Q3). Testing error as a function of downstream examples for different downstream tasks used in SYS-1. We show the extent of overlap (signifying distributional shifts) between the pre-trained and downstream dataset at the top using the range of sampled diffusion tensor eigenvalue. For datasets within distribution, zero-shot TL is optimal. As the downstream dataset shifts moderately OOD, the zero-shot learning suffers gradually and is recovered through fine-tuning. This recovery slower as the distributional shifts increase.
We repeat downstream dataset scaling experiments the different downstream tasks and show the trends for SYS-1 in Fig. 4. In particular, in Fig. 4a, we consider the downstream dataset within distribution of the pre-training dataset (as visualized by the e distribution at the top). We observe excellent zero-shot performance that is unaffected by further fine-tuning. In Fig. 4b, the downstream dataset is shifted mildly OOD. Although the zero-shot performance drops, it still shows low errors, significantly smaller than training from scratch. Further, the performance is improved with few-shot TL up the point of diminishing returns with large numbers of downstream data examples. With further distributional shift (no overlap) in Fig. 4c, the zero-shot performance suffers more, but with a larger amount of fine-tuning recovers good performance. Finally, for large distributional shifts in Fig. 4d, the zero-shot and few-shot performance is poor, with TL showing relatively high errors, but even here TL improves over training from scratch. In this case, due to larger anisotropy. the system is also harder to emulate and might require more data in general. We repeat this analysis for SYS-2 and observe similar trends across different OOD downstream tasks (see Appendix IB.
(Q4): TL behavior over multiple operators. We further diversify the pre-training by including examples with different solution operators. We combine the datasets from three PDEs- Pois- son's SY)- Advection-Diffussonn SYS-2(0.22,)), and Helmholtz SYS-3( (1, 10) (where the wavenumber (1, 10)). Here, have additionally included the Helmholtz PDE, a challenging system due to the highly oscillatory behavior of the solutions (see PDE 2 in Fig. and Fig. A. for examples), very sensitive to the range of wavenumbers. When pre- pre-training single model on this "mixed" dataset, we simply use zero channels for those coefficients that do not exist when using examples from a specific operator. For example, the Helmholtz equation has a diffusion tensor input (identity matrix) with an additional input for the wavenumber but no advection (zero channel), while the Poisson's 's equation only has a diffusion tensor input and hence we append zero channels to signify wavenumbers and advection; similarly for Ddlvection-Diffusion This, effectively, serves

---

Downstream
Pretrain 5YS-1(1.
Down streatt
Fretrain
SYS-101.2 2.51
5)
SYS-311. 5)
SY 311. 10)
1.0
15
2.0
255
3.0
3.5
4.0
5.0
10
Diffusion tensor eigenvalue
Helmholtz frequency
Testing errer (relative E3)
10a
-
Trairing from Scratch
Testing error (relative t.)
10"
Training from Scretch
Pre-trained
from Pre-trained
10-1
TL from Mixed Pre-trained
10-1
TL from Mixed Pre-trained
10-2
10-2
10-3
10-1
512 1K 28
15K 32K
B-shot
16
256 512 Number downstream examples
15K 32X
Number downstream examples
(a) SYS-1(1,2.5) pre- pre-training using SY5-1(1,5) and mixed dataset
(b) SYS-3((111 pre-training from SYS-3(1,10) and mixed dataset
Figure 5: Addressing (Q4). Testing error as a function of downstream examples for SYS-1, SYS- 2. and S-S-3 with Fieettnnng from their respective PDE systems and from the mixed dataset (combination of SYS-1, SYS-2, and SYS-3). The model pre-trained on the mixed dataset performs better than training from scratch. More importantly, the same pre-trained model yields low errors on all the downstream PDEs with both zero-shot and task-specific, fine-tuning We show the results for S-S-2 and OOD performance $B.2.
as selection of the solution operator during the forward pass to predict the solution to the right operator. While more advance techniques such as in-context prompting (from LLMs) exist, here are interested in understanding if this simple and minimal selection/prompting is sufficient for the model to transfer effectively to downstream tasks. For the downstream tasks, we consider three within-distribution tasks of Poisson's SYS-1(1,2) (1.2,,,, Addeetion--iffussii Helmholtz SYS-3(1,5) and show our dataset scaling results in Fig. 5.
SY8-2(0.2,0.44) and
The results support our most compelling conclusion: substantial performance gains over training from scratch for all downstream tasks. The same model (pre-trained on three different tasks) is useful in all downstream tasks, in both the zero-shot and the fine-tuning settings. This indicates the input coefficient channels are sufficient to prompt the model to predict the correct downstream solution. We show the OOD downstream performance of this model in Appendix $B.2 and observe similar behavior.
fine-tuning from the mixed dataset retains the
5 Conclusions
We have provided extensive analysis of the scaling and transfer behavior of neural operator models on multiple PDE systems. This involved characterizing behavior as a function of model size, downstream dataset size, underlying physics of the downstream tasks in relation pre-training, the adaptation of these models to multiple downstream PDEs, and how all these behaviors scale with relevant problem and model parameters. Among other things, we have shown that it possible and beneficial to develop more general SciML models capable of solving multiple tasks with the same of weights, even when downstream tasks involve mmall-to-moderate distribution shifts relative to the pre-training data. All in all, this demonstrates the potential of the "pre-train and fine-tune" paradigm for SciML problems. paving a path towards building SciML foundation models. Moving forward, many questions remain. These include further exploration of model architectures (balancing expressivity and flexibility), pre-training protocols (including self-supervision components scale), fine-tuning strategies, and the integration of these within a specific compute (and memory) envelope. There are a number of future directions raised by our work. Our pre- train and fine-tune recipe may need more sophisticated prompting during inference, especially if only the operator form changed between two different PDE systems. Also, we do not look at self-supervision with the PDE loss penalty a means of large-scale pre-training, and we limit our analysis to 2D spatial systems. Moving to other architectures, larger scales, and more complex PDEs in space-time is a focus of future work.
10

---

Acknowledgements
This work was partially supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Scientific Discovery through Advanced Computing (SciDAC) program, under Contract Number DE-ACC2-05CHI122 1231 Lawrence Berkeley National I Laboratory. This research used the Perlmutter supercomputing resources of the National Energy Research Scientific Computing Center (NERSC), a U.S. Department of Energy Office of Science User Facility located at Lawrence Berkeley National I Laboratory, operated under Contract No. DE- AC02-05CH11231 We acknowledge gracious support from Google Cloud, Google TRC team, and specifically Jonathan Caton, and Prof. David Patterson. Prof. Keutzer's lab is sponsored by Intel corporation, Intel VLAB team, Intel One-API center of excellence, as well as funding through BDD and BAIR. Amir Gholami acknowledges support from Samsung SAIT and Intel corporations. Our conclusions do not necessarily reflect the position or the policy of our sponsors, and no official endorsement should be inferred.
References
[1] https://github.com http:///wcccccaaaaaoaaaeaaaaaata
ing, 2023.
[2] Kaushik Bhattacharya, Bamdad Hosseini. Nikola B Kovachki, and Andrew M Stuart. Model reduction and neural networks for parametric pdes. The SMAI journal of computational mathematicx, 7:121-157, 2021.
[3] Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bemsttein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07255. 2021.
[4] Nicolas Boullé and Alex Townsend. Learning elliptic partial diffcrential equations with randomized linear algebra. Foundations of Computational Mathematics, 23(2):709 703- 2023.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subhiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[6] Ayan Chakraborty, Cosmin Anitescu, Xiaoying Zhuang, and Timon Rabezuk. Domain adaptation based transfer learning approach for solving pdes complex geometries. Engineering with Computers, 38(5))4559-4588 2022.
[7] Souvik Chakraborty. Transfer learning based multi-fidelity physics informed deep neural network. Journal of Computational Physics, 126:20 2021.
[8] Xinhai Chen, Chunye Gong, Qian Wan, Liang Deng, Yunbo Wan, Yang Liu, Bo Chen, and Jie Liu. Transfer learning for deep neural network-hased partial differential equations solving. Advances in Aerodynamics, 3(1):1 14, 2021.
[9] Maarten De Hoop, Daniel Zhengyu Huang, Elizabeth Qian, and Andrew M Stuart. The cost-accuracy trade-off in operator learning with neural networks. arXiv preprint arXiv:2203. 13181, 2022.
[10] Shaan Desai, Marios Mattheakis, Hayden Joy, Pavlos Protopapas, and Stephen Roberts. One-shot transfer learning of physics-ifformed neural networks. arXiv preprint arXiv:2110.11286, 2021.
[1I] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[12] C. Edwards. Neural networks learn to speed up simulations. Communications of the ACM, 65(5):27-29. 2022.
[13] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domainnadversarial training neural networks. The journal of machine learning research, 17(1):2096200 2030, 2016.
[14] Yihang Gao, Ka Chun Cheung, and Michael K Ng. Svd-pinns: Transfer learning of physics-ifformed neural networks via singular value decomposition. arXiv preprint 1rXiv:2211.087660 2022.
[15] Somdatta Goswami, Cosmin Anitescu, Souvik Chakraborty, and Timon Rabezuk. Transfer learning enhanced physics informed neural network for phase phase-field modeling of fracture. Theoretical and Applied Fracture Mechanics, 106: 2020.

---

[16] Somdatta Goswami, Katiana Kontolati, Michael D Shields, and George Em Karniadakis, Deep transfer operator learning for partial differential equations under conditional shift. Nature Machine Intelligence. pages 1-10, 2022.
[17] Hongwei Guo, Xiaoying Zhuang, Pengwan Chen, Naif Alajlan, and Timon Rabezuk. Analysis of three- dimensional potential problems in non-homogenenus media with physics-informed deep collocation method using material transfer learning and sensitivity analysis. Engineering with Computers, pages 1 1-22, 2022.
[18] Ehsan Haghighat, Maziar Raissi, Adrian Moure, Hector Gomez, and Ruben Juanes. A physies-informed informed deep learning framework for inversion and surrogate modeling in solid mechanics. Compuder Methods in Applied Mechanics and Engineering. 379:1 13741, 2021.
[19] John Hanna, Jose V Aguado, Sebastien Comas-Care Cardona, Ramzi Askri, and Domenico Borzacchicllo. Residual-based adaptivity for two-phase flow simulation in porous media using physics-informed neural networks. arXiv preprint arXiv:2109. 14290, 2021.
[20] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal. Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint 2i0i /4701, 2020.
[21] Jordan Hoffmann, Sehastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. An empirical analysis of compute-optimal large language model training. Advances in Neural Information Processing Systems, 35:30016-30030, 2022.
[22] Neil Houlsby. Andrei Ciurgiu. Stanislaw Jastrzebski. Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Auariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799. PMIR, 2019.
[23] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, arXiv preprint arXiv:2001.0838 2020.
[24] George Em Karniadakis, Ioannis Kevrekidis, Lu Lu, Paris Perdikaris. Sifan Wang, and Liu Yang. Physics-informed machine learning. Nature Reviews Physics, 3(6):422-44) 440, 2021.
[25] Yuchaw Khoo, Jianfeng La, and Lexing Ying. Solving parametric pde prohlems with artificial neural networks. European Journal of Applied Mathematics, 32(3):221-335 2021.
[26] Georgios Kissas, Jacob H Seidman, Leonardo Ferreira Guilhoto, Victor M Preciado, George Pappas, and Paris Perdikaris. Learning operators with coupled attention. Journal of Machine Learning Research, 23(215):1-63 63, 2022.
[27] Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint 10Xiv:2108.08445 2021.
[28] A. S. Krishnapriyan, A. Gholami, S. Zhe, R. M. Kirby, and M. W. Mahoney. Characterizing possible failure modes in physics-infomeed neural networks. Technical Report Preprint: #xXiv:2109.01050 2021.
[29] Zongyi Li. Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for pdes on general geometries. arXiv preprint arXiv:2207.052008 2022.
[30] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXi::2010.0889, 2020.
[31] Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Нaoxuan Chen, Burigede Liu, Kamyar Aziz- zadeneshcli, and Anima Anandkumar, Physics-informat neural operator for lcarning partial diffcrential cquations. arXiv preprint X 2021.
[32] Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Kamiadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature machine intelligence, 3(3):218-222 2021.
[33] Lu Lu, Xuhui Meng. Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data. Computer Methods in Applied Mechanics and Engineering, 193:114778, apr 2022.
12

---

[34] Yanfang Lyu, Xiaoyu Zhao, Zhiqiang Gong. Xiao Kang. and Wen Yao. Multi-fidelity prediction of fluid flow and temperature field based transfer learning using fourier neural operator. arXiv preprint arXiv:2304.06972 2023.
[35] Marios Mattheakis, Hayden Joy, and Pavlos Protopapas. Unsupervised reservoir computing for solving ordinary differential equations. arXiv preprint arXiv:2 /1417, 2021.
[36] Maxime Oquab, Lcon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1717-1724, 2014.
[37] Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Arizzadenesheli, et al. Fourcastnet: global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202. /214, 2022.
[38] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[39] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)9, 2019.
[40]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555 2020.
[41] Md Ashiqur Rahman, Zachary Ross, and Kamyar Azizzadenesheli. U-no: U-shaped neural operators. arXiv preprint arXiv:2204. 11127, 2022.
[42] Maziar Raissi, Paris Perdikaris, and George F. Karniadakis. Physies-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics. 378:686-7011 707, 2019.
[43] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3723-3732, 2018.
[44] Dong H Song and Daniel M Tartakovsky. Transfer learning on multifidelity data. Journal of Machine Learning for Modeling and Computing, 3(1), 2022.
[45] Rick Stevens, Valerie Taylor, Jeff Nichols, Arthur Barney Maccabe, Katherine Yelick, and David Brown. Ai for science: Report on the department of energy (doe) town halls artificial intelligence (ai) for science. Technical report, Argonne National (AbL,, Argonne, IL. (United States), 2020.
[46] Adam Subel. Yifei Guan, Ashesh Chattopadhyay, and Pedram Hassanzadeh. Explaining the physics of transfer learning in data-driven turbulence modeling. PNAS Nexus, 2(3), 01 2023. pgad015.
[47] Hengjie Wang. Robert Planas, Aparna Chandramowlishwwrran, and Ramin Bostanabad. Mosaic flows: A transferable deep lcarning framework for solving pdes on unseon domains. Computer Methods in Applied Mechanics and Engineering, 389:1 14424, 2022.
[48] Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data, 3(1):1440, 2016.
[49] Gege Wen, Zongyi Li, Kamyar Azizzadenesheli, Anima Anandkumar, and Sally M Benson. U-fno-an enhanced fourier neural operator-based deep-learning model for multiphase flow. Advances in Water Resources, 163:104180, 2022.
[50] Chen Xu, Ba Trung Cao, Yong Yuan, and Günther Meschke. Transfer learning based physics-iffrmmd neural networks for solving inverse problems in engineering structures under different loading scenarios. Computer Methods in Applied Mechanics and Engineering. 405: 15552, 2023.
[51] Wuzhe Xu, Yulong Lu, and Li Wang. Transfer learning enhanced deeponet for long-time prediction of evolution equations. arXiv preprint arXiv:2212.0446 2022.
[52] Liu Yang, Siting Liu, Tingwei Meng, and Stanley J Osher. In-context operator learning for differential equation problems. arXiv preprint 23Xx:2304007993, 2023.
13

---

[53] Yan Yang, Angela F. Gao, Jorge C. Castellanos, Zachary E. Ross, Kamyar Azizzadenesheli, and Robert W. Clayton. Seismic propagation and inversion with neural operators, 2021.
[54] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
[55] Min Zhu, Handi Zhang, Anran Jiao, George Em Karniadakis, and Lu Lu. Reliable extrapolation of deep neural operators informed by physics sparse observations. arXiv preprint arXiv:2212.06347 2022.
14

---

Source Sampling
Increasing Diffusion Eigenvalue (Foxed Rotation)
Incressing Sparsity of Galasiens
Increasing Advection Diffusiin Ratio (Fixed Rotation, Diffusion)
SYS-2
Increasing Helmholtz Wavenumber
Figure A.1: Visualization of the source function sampling (left) and the effect of certain PDE coefficients (right) on the solutions for the different systems. On the left, as we go down, the sparsity of Gaussians is increasing, leading
more sparse and spread out source functions encouraging heterogeneity in the dataset. For one of these source functions. we apply the different PDE operators with varying ranges of certaino PDE coefficients illustrate their effect the solutions. On the top row, for SYS-1 (Poisson's 's), we show that by increasing the diffusion tensor eigenvalue e (but keeping the direction 0 fixed), we increasing anisotropy and diffusion as move towards the right. In the middle, we increase the velocity scales for SYS-2 (Advection- (ittion--iffuionn. but keep the diffusion tensor and velocity direction the same, demonstrate the increasing competing advection and diffusion processes as we go right. Finally, at the bottom, we show the highly oscillatory behavior in SYS-3 (Helmholtz) as we increase the wavenumber w. Note the significant differences between the solutions of the different systems.
A Appendix: Additional Details
A.1 Pre-train and downstream data creation
As described in $3, our sampling strategy involves (i) source functions and (ii) PDE coefficients. We use numerical discretization of 128 x 128 for all our experiments. Some additional details are as follows:
(i) Source function sampling: As described in $3, our source functions are a linear combination of ny radial (Gaussian) basis functions {x) where (x) o(x x4) is a Gaussian function centered at grid point 24. Specifically: f(x)
By dx (()),,, with p {p.) 1% as the parameterization vector. The spatial profile controlled by the standard deviation of the Gaussian function, preset small value 1/32 (hence 4 times the spatial resolution) to encourage high variability. The spacing between Gaussians is fixed at 2o. Examples are sampled by uniformly randomly sampling p; (4, 1). We further introduce heterogeneity by controlling the sparsity of p (we define s as the fraction of number of zero components of P:). Hence, s 0.6, implies only 40% of p is randomly sampled in (1, 1) and the rest (60%) are set to zero. Visualizations of this are in Fig. A. on the left. As we go down, the sparsity is increased (and hence Gaussians are more sparse and spread apart). In our pre-training and downstream datasets, we sample sparsity levels from 20% to 80% uniformly.
(i) PDE coefficient sampling: SYS-1, we sample diffusion coefficient tensors K ~ IC(A), where 1. is distribution that generates varying scales of anisotropy and spread in the diffusion process: K R DR with D diag(1. (1, e) and R rot( where e is an eigenvalue of the tensor that controls the anisotropy and extent of diffusion and rot(O) a rotation matrix with angle 0 ~ (0, 2T)) that controls the general diffusion direction. We visualize the effect of e in Fig. A. (top row). With a fixed (direction of diffusion), we see that with larger e, the solution is more anisotropic and diffuse. The sampling of and 0 in systematic fashion
15

---

SYS-2
T.K,o
Figure A.2: We illustrate the variability source function imputs (left) and solution outputs (right) for SYS-2, where the velocity direction, scales, and the diffusion tensor direction and anisotropy scales (eigenvalue) are all changed along with the source sampling to produce the input ousput pairs for the training dataset.
quantifies the underlying diffusion physics. In SYS-2, 2, we additionally also sample the velocity vector v direction from (0, 2T). We define = l/e Vull/|| K Vull, the ratio of advection to diffusion to quantify the different processes and V is changed by scaling the velocity. In Fig. A. (middle row), we visualize increasing advection (from left to right) when the diffusion tensor is kept the same Sobeeve that the solution changes significantly as V increases and SYS-3, we sample
the two processes (advection and diffusion) compete more strongly. the wavenumber as uniform integers. We visualize the effect of increasing w in Fig. A.1 (bottom row) and observe that increasing frequeecy leads to highly oscillatory behavior. We also underscore the significant differences in the output of the three systems here. For more details, see Appendix $A.1. We visualize the sampling process and resulting solutions with different ranges of underlying physies in Fig. A.1. In Fig. A.2, we show small sample of input sources and output solutions for SYS-2 to illustrate the variability in our datasets- note here all the variables (advection directions and scales, diffusion directions and scales, source functions) are changed to create the full dataset.
For each PDE system, use spectral methods compute the ground truth solution of the PDE. For example, for SYS-1:
div KVu = f,
(4)
the solution is:
(k22k k2 + k212) F(f())))
(5)
where is the Fourier transform. kg by are the frequencies in the Fourier domain, and h1 k2 Ki2 are the diagonal and off- -if-diagn coefficients of the diffusion tensor K. The solution is unique given zero-mean source functions. Similar solutions can be derived for the other systems.
A.2 Model architecture
We describe the architecture details of the FNO here. The central building block of FNO learns a kernel integral operator parameterized by a function operating in Fourier space. By composing several of these blocks together in sequence, the FNO can learn to approximate the solution operator for complex PDEs. For Fuclidean systems discretized at uniform resolution (e.g., in 2D with inputs and solutions having dimension Ig 1x) the implementation can be made efficient by using the
16

---

Fast Fourier Transform (FFT), denoted by F with inverse FFT F-1, We refer the reader to the original FNO paper [30] for details, but briefly summarize the basic mechanism, to highlight how the model complexity depends on key hyperparameters. The inputs to the 2D) FNO are spatial functions discretized at this resolution X w that represent sources, PDE coefficients, ICs/BCs of the PDE system-each of these is represented
separate channel, leading to input tensors in & with e input channels. As mentioned before, we use the resolution h X w 128 x 128. Given this input tensor A E IR bxx (c input channels representing the PDE inputs), the FNO first projects A into a tensor X E ggxwxx with embedding dimension d, which is passed through a series of FNO blocks. For given block I E {1, ... L} with input X E Rax=xd xxd the output at spatial index (i.j) is computed as
x!+1 = a (WiX (K.) (4.))
(i.) + FI (6.a))
(6)
where a is a pointwise nonlinear activation function, Wi E Raxd is learnable weight matrix, which performs pointwise linear transformation, and X = F(X') E Chxuxd are complex-valuee Fourier coefficients output by the FFT. The transformation K2 in Fourier space is parameterized by complex weights &' Cd x ma according to
(x (o)
1 (kL.)a) Aaa X!
A2),
for pairs of Fourier frequencie/wavenummbes k, E [I,... ma} and hn E (I....,m.h) The hyperpa- rameters mi and my control the "mode cutoff", beyond which Fourier modes are ignored by K, and they have a theoretical maximum of my = h/2 and my - w/2, i.e., the Nyquist limit along each spatial dimension. In practice, the mode cutoffs are key hyperparameter controlling model complex- ity along with the embedding dimension d, and they are often tuned to be less than the Nyquist limit to prevent overfitting and accelerate training. For square problems (h- w), symmetric truncation is adopted such that m = my = my. Thus, the per-layer parameter count is quadratic in both m and d, dominated by the complex weights क. The above (d and m) hyperparamters are the focus of our exploration in the model scaling experiments.
A.3 Training details and code open-source
For training, we use the Adam optimizer with a cosine learning rate decay schedule. All models are trained for 500 epochs with the best model saved at lowest validation loss. We tune batch size and initial learning rates using a grid hyperparameter search, and train every model using 4 NVIDIA A A10 GPUs (using standard )arraaaalalllsmmm on the Perlmutter supercomputer. A full training run (on all 32K examples used for pre- pre-training) completes in around 2.5 hours. To evaluate model performance, we use the mean relative error between predicted and target solution, defined as: [4g, 1/N ||u uoll2/ /I uo I|,, where is the predicted solution, up is the target solution, and N is total (fixed) number of testing (around 4K) examples. We open source our model training and data generation code at [1].
A.4 Hyperparameter tuning
For studying the relationship between model and dataset size, we perform a simple grid-search over different learning rates for each combination of model and dataset size. This is important because different dataset sizes may demand different hyperparameter values-for instance, we observe that for very small dataset sizes (small number of downstream examples), the tuned learning rates are significantly smaller (especially for zero- and few-shot transfer learning) to help mitigate over-fitting and optimization difficulties at that scale, whereas larger learning rates perform better model and dataset sizes increase. Hence, for any result (including the model scaling), these learning rates are tuned with the best values picked from the validation set metrics.
A.5 Input normalization
As mentioned in $3, key aspect of our training is the input normalization (for any PDE system). We first normalize every training example source and coefficient inputs with respect to a reference source value defined as the median source norm over the training dataset, i.e., fre == median, (IIf where i are the training examples. Hence for any example ft. we first compute ||/i|| and normalize f; and all coefficients with the relative norm []fille/ fres- First, this ensures the source norms are within reasonable scale range. Additionally, it implies that scaling both coefficients as well source
17

---

Downstream
Pretrain SYS-210.2 2,
Pretrain
Downsrream SYS-210 1.6)
SY5-2(0.2 2. 0.4
30
SVS-210
02
0.3
0.4
06
0.7
0.8
1.0
92
0.4
0.6
0.8
1.0
1.2
16
Advecti on-diffusion ratio
Advection- ratio
Testing eirer (relative E.)
10a
Training from Scratch
Testing errer reelative 4:)
10a
Training from Scratch
Pre-trained
Pre-trained
10-1
10-
10-2
Deltet
1K
15K
32K
512
18K 32X
downstream examples
Number downstream exampless
(a) SYS 2(0.2,0.4
(b) SYS-2(004,1.6))
Protrain
Downstream
Protrain S-2220 2.
Downstream SYS-212, 5)
SYS 200.2,
3)
SYS 2(1, 2)
02
0.4
266
D
10
12
14
1.8
14
2.5
Advectionn on dirfusion ratic
Advection diffusion ratio
Testing ertor (relative.))
100
-C-
Training from Scratch
(1
Training from Scratch Pro-trained
10
16 22 66 128
255 512
2K B6 19K 32x
9-2het 5
32 64 125
256 512
2K 4X 8K 165 22K
Number of downstream examples
Number of downstreamn exa mppees
(c) 2(1,)
(d) SYS--((2,5)))
Figure A.3: Addressing (Q3). Testing error as a fufuctiion of number of downstream dataset examples for different downstream tasks used in SYS-2. We show the extent of overlap between the pre- trained and downstream dataset at the top using the range of auvection-diffus datasets. Similar SYS-1, we observe good zero-shot performance that gradually decreases with distributional shifts but can be recovered through few-shot learning. With large shifts, the number of examples needed reach desired error levels also increases.
ratios of the two
functions of the inputs by constant yields the same inputs the neural operator. For example: the inputs f and K to the Poisson's 's equation (SYS-1) are equivalent to the inputs 10f and 10Kh-both have the same solution function. The above normalization makes equivalent, since 10f and 10K simply get normalized by 10 before passing to the network. Then, ensure the coefficient inputs are comparable scales and within reasonable range, the coefficient values (that typically have very different scales depending on the physical process, up to 100x differences) are normalized by constant reference values pre-computed across the training dataset as the median values of the coefficients. We observe that without normalization the performance of FNO can be quite poor, especially cases with multiple channels representing different physical scales (such as in advection diffecioondiifusion SYS-2).
that these two input pairs
B Additional Results
B.1
TL behavior over underlying physies and model scaling
We show the TL performance under physics shift (Q3) for SYS-2 in Fig. A.3. Similar to SYS-1, observe excellent TL performance for in in-distribution tasks (see Fig. A.3a) that independent of the downstream dataset size. As systematically go OOD, we continue to observe good performance gains with TL (both zero-shot and few-shot) until we are significantly OOD as in Fig. A.3d.
In both SYS-1 and SYS-2, we find that TL from pre-trained models outperforms training from scratch with the number of downstream examples required to reach given accuracy being orders of magnitude smaller with TL. To quantify this in detail, we estimate the amount of "data savings" for TI. by interpolating the equivalent number of examples needed to reach given model error when training from scratch. We plot these "from-scratch" data requirements function of number of
18

---

32K
32K-
Equivalert date, from scratth
16k-
Equlvaaaa deta from S
16k-
4K
4K
2K:
512-
512
236-1
Downstream dataset distribution shift, SYS-1
256
Downstream dataset
128-
128
distributi shift, SYS-2
64
None
64
None
16
Med
Med
Large
Large
4K
32K
16K
downstream examples
downstream examples
Figure B.1: Addressing (Q3). Equivalent data needed to reach the accuracy of TL when training from scratch, for SYS-1 and SYS-2 distribution shifts defined in Table 1. Data from experiments where TL outperforms even best (i.e., largest dataset) from-scratch models is not plotted.
Pretrain SYS-101
Downstream SYS-1/10. 20)
10
BAK fnetured
GAT acatch
scnaton
Diffusion eigenvalue
Masttstmon ma retatie crer
Instured
Testing errer (relative E3)
109
Training Scratch
10-4
16M scratt)
TL from Pre trained
16M tratuned
10-1
10-2
10:3
3GK 35K
Number
Figure B.2: Some additional results: (left) for extreme shifts and challenging systems, the number of examples to reach a desired accuracy increases. For example, for TL SYS-1(10,20), the training from scratch and TL curves intersect 128K examples. (right) model parameter scaling for TL to SY,-3(1,5) showing similar trends as other systems.
TL ane-tuning examples in Fig. B. In these plots, any points above the dashed black line indicate an improvement over training from scratch; points for TL experiments which outperform even the best from-scratch models (which use maximum of 32K training examples) are not plotted. We observe that even the downstream dataset distribution shift moves from "None" to "Large" (as defined in Table 1), the models consistently reach the performance from-scratch models which would otherwise require thousands to tens of thousands of training examples. One exception is the largest OOD shift for SYS-1, where we observe smaller (but still substantial) advantages for the TL approach in zero- and few-shot learning. We note that characterizing distribution shifts consistently for different systems challenging: this result suggests perhaps the "Large" distribution shift for SYS-1 is more significant shift than that of SYS-2.
Finally, we include two additional results: For extreme shifts such SYS-1(10,20), we note that the errors are higher as its is a more challenging system (high anisotropy in diffusion, in this case) and hence needs more data to achieve lower errors. Fig. B.2 (left) we show that the training from scratch and TL from pre-trained model curves converge at a much larger dataset of 128K examples. We also include the model scaling experiments for Helmholtz system SYS-3(1,5) in Fig. B.2 (right).
B.2 TL behavior underlying multiple operators
We show additional results for TL behavior over multiple operators in Fig. B.3. Each row represents PDE system with in-distribution (left) and OOD (right) downstream tasks. We observe that the mixed pre-trained model is able retain TL gains as system-specific pre-trained model over all the systems. Hence, single model is now adapted to three different PDE systems in downstream fine-tuning. We also note SYS-3 OOD results-we observe that for the Helmholtz, going even moderately OOD, affects both TL and training from scratch performance. This PDE system is
19

---

Downstream
Pretrain SYS
Pretrain
Downstream SYS 1 5. 7.5)
SYS 1(1, 2.5)
1(1, 5)
YS 111, 5)
3.5
Diffusion tensor eigenvalue
Diffusion tensor eigenvalue
Testing errer (relative fs)
10a
Training
Testing error (relative [z)
100
Training Scratch
TL from Pre-traired
TL from Pre-trained
10:
TL from Mixed Pre trained
TL from Mixed Pre trained
10-2
10
10-1
32K
120 Number
32K
Number examples
examples
(a) SYS- 111225; pre-training using SYS-1(1,5) and mixed dataset
(b) 1 pre-training using SYS- 1(1,5 and mixed dataset
Downstream
Pretrain SYS-210
Pretra
Down Downskrean Kream SYS-210 4, 1.6)
5---21022 2.
13
SY 5/0-2 2.
02
0.3
34
0.
26
07
0.5
0.9
02
0.4
6
0.5
1.0
1.2
Advecti on diffusion ratio
advection-diffuiion ratio
Testing eiror (reletive 53)
10a
-O-
Training from Scratch
Testing anne Cepeoae 4.3
10a
-O-
Training from Scratch
from Pre-traired
from Pre-trained
TL from Mixed Pre-trained
TL from Mixed Pre-trained
10-4
10-1
10
10-2
Dnhhot
16
32
M
128
256
512
1K
28
45
DK
13K
32K
Dechet
36
32
M
128
256
512
2K
DC
15K
32K
Number of downstream examples
Number of downstream examples
(c) SSYS-2(0.200.. pre-training using (02-2(0.2,1) and mixed dataset
(d) SYS422004.1.66); pre-training using SYS-2(0.2,1) and mixed dataset
Downstream SYS 3(1, 5)
Protra
Downstream SYS-312 312,
30.
SYS: 3(1, 10)
Helmholiz frequency
Helmholtz frequency
Testing ertor (relative E3)
109
Trairing
Testing error (relative f.)
100
Training
TL from Pre-traired
TL from Pre-trained
10-1
TL from Mixed Pre-trained
TL from Mixed Pre-trained
10-1
10-2
32x
Number of donsttream examples
Number of downstream examples
(e) SYS-3(1,5); pre-training using SYS-3(1,10) and mixed dataset
( SYS-3(2,12); pre-training using SYS-3(1,10) and mixed dataset
Figure B.3: Addressing (Q4). Testing error as a function of downstream examples for SYS-1 (top). SYS-2 (middle), and SYS-3 (bottom) with fine-tuning from their respective PDE systems and from the mixed dataset (combination of SYS-1, SYS-2, and SYS-3). each row: on the left, the downstream task is in-distribution and, on the right, it is OOD. The model pre-trained the mixed dataset performs better than training from scratch. More importantly, the same pre-traines model yields low errors all the downstream PDEs with both shot and task-specift fne-tuning. We also note that SYS-3 is a particularly challenging system that shows larger performance drops as we go OOD the mixed pre-trade pre-training still retains the same gains the task-specific pre- rre-trained model.
20

---

particularly challenging and is very sensitive to the input wavenumber. Slight changes introduce more oscillatory features the solution, pushing the downstream task OOD easily. However, we note that both the mixed pre-training and pre-training from Helmholtz dataset show similar performance even on the OOD task.
B.3 Sensitivity random seeds
To quantify the variability of both training from scratch and performing TL from the pre-trained model, we repeat the "Med" OOD shift experiment for SYS-1 and SYS-2 (see Table 1) times with different random seeds and record the testing eπror for each trial. The resulting distributions indicate how sensitive each approach is the random data shuffle used when training on the downstream dataset. We plot the mean testing error along with 1* and 3rd quartiles for each downstream dataset size in Fig. B.4. We observe small sensitivity for both TL and training from scratch. Further, not surprisingly, we observe that the variability across random seeds is generally larger when training from scratch. We note that the small variability in TL is difficult to see due to the log-spaced y-axis.
Protrain
Protrain
SY5-1(3,
SY5-2(0.2.
SY5-2(1,
0.4
0.6
08
1.0
12
14
1.6
18
Diffusion tensor eigenvalue
Advection advection-diffusion ratio
Testing eiror (reletive 65)
10a
Training from Scratch TL from Pre-trained
Testing enron (relative 6,1
100
Training from Scratch
TL from Pre-trained
10-1
10-1
Dahot
32 64
512
2K
15K 32K
Dehhot
22 64
120
255
512 1K 2K
18K 32x
Number of downstream examples
Number of downstream examples
(a) SYS-1(5,10) pre-trained from SYS-1(1,5)
(b) SYS-2(1.2)) pre-trained from SYS-2(0.2,1)
Figure B.4: Testing error function of downstream examples for SYS-1 and SYS-2, aggregated over 5 trials with different random seeds for each experiment. The dots indicate the mean testing error each downstream dataset size, and the shaded region represents the spread between the 1t and 3rd quartiles. We observe small variance that is slightly larger when training from scratch compared to from the pre-trained model.
21


Reviews:
[{"review_id": "10PqO5TpPG", "reviewer": "Reviewer_Hq79", "summary": "This paper explores the behavior and performance of neural operator models on multiple partial differential equation (PDE) systems in the transfer learning setting. The authors investigate the impact of various factors on the performance of neural operator models, including model size, downstream dataset size, underlying physics of the downstream tasks in relation to pre-training, and distributional shifts.\n\nThe study employs the \"pre-train and fine-tune\" paradigm for SciML (Scientific Machine Learning) problems. The authors show that it is possible and beneficial to develop more general SciML models capable of solving multiple tasks with the same set of weights, even when downstream tasks involve small-to-moderate distribution shifts relative to the pre-training data. The research demonstrates the potential of neural operator models for transfer learning, paving the way towards building foundation models for SciML.", "questions": "While exploring transferability across multiple downstream PDE systems is an intriguing and significant area of research, is there any differences in transfer learning between PDE problems and well-established domains like Language Model Learning (LLM) and Computer Vision (CV). What does the nature of PDEs and their associated physical systems introduce unique challenges and characteristic that may affect the transferability of learned representations and the generalization of neural operator models?\n\nIn my opinion, understanding and addressing these differences are crucial for insights from the transfer learning perspective in PDEs.", "limitations": "Listed in the Weaknesses and Limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality: \n\nThe paper demonstrates a high level of originality in several aspects. Firstly, it explores the use of neural operator models for solving multiple partial differential equation (PDE) systems, a novel approach that combines the power of neural networks with physics-informed learning. The authors investigate the behavior of these models under various conditions, including model size and downstream dataset size, providing valuable insights into their scalability and transferability. Secondly, the study applies the \"pre-train and fine-tune\" paradigm to SciML problems, showing the potential of developing more general SciML foundation models. This approach is innovative and opens up possibilities for efficiently addressing diverse downstream tasks with a single pre-trained model. Furthermore, the paper explores the impact of distributional shifts in downstream tasks relative to the pre-training data, contributing to the understanding of transfer learning behavior in the context of PDEs. The inclusion of multiple downstream PDE systems and the analysis of various performance metrics demonstrate a comprehensive and original investigation.\n\nClarity: \n\nThe paper is well-written and presents its findings in a clear and organized manner. The subsection is organized logically following the order of the posted questions.\n\nSignificance:\n\nThe significance of the paper lies in its contribution to the field of SciML and transfer learning for PDEs. By characterizing the behavior of neural operator models under different conditions and exploring transferability across multiple downstream PDE systems, the paper provides valuable insights for researchers working in the area of computational physics and scientific modeling. The findings have practical implications, as they suggest the potential for developing more versatile and efficient foundation models for SciML tasks. The \"pre-train and fine-tune\" approach demonstrated in the paper can be leveraged to achieve better generalization and performance in solving PDEs across a wide range of scenarios.", "weaknesses": "Motivation: \n\nWhile the paper extensively analyzes the performance of neural operator models under different conditions, it lacks direct motivation of understanding the transfer learning for PDEs problem in SciML, as these different conditions have been well-explored in the LLM and CV community.\n\nPotential Overfitting in Transfer Learning Settings: \n\nThe paper briefly mentions that few-shot transfer learning outperforms training from scratch, but it does not thoroughly address the possibility of overfitting to the specific downstream tasks during fine-tuning. An investigation into potential overfitting issues and strategies to mitigate them would be valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough assessment and kind words regarding our work. The suggestions they provide are very helpful, and we address specific comments below:\n\n### R1-1: Motivation for understanding the transfer learning (TL) for PDEs in SciML:\nWe agree that the introduction can state our motivation more clearly. Given the success of the foundation model paradigm in the CV/NLP domains, a major motivation of ours is to provide an early systematic evaluation of TL and model/data scaling in the PDE domain. Solving PDE systems is a core component for many practical and applied problems such as climate modeling, molecular dynamics simulations, computational fluid dynamics, cancer growth modeling, etc. Fast and efficient solvers that can provide approximate solutions or serve as numerical preconditioners can be extremely valuable for these applications. Deep learning models have proven to be capable of serving this objective. However, the majority of current works train separate models for separate PDE systems. Here, TL and, specifically foundation models, offer an opportunity to reuse models and tailor them to specific downstream applications. This can be particularly useful in the PDE domain because the numerical solvers required to generate training data for many applications are very computationally expensive, leaving most efforts bottlenecked by limited data. Given the vast diversity of PDE systems and the range of scales/features in the solutions as well as coefficients of the PDEs, it is unrealistic to expect a single training set to span sufficiently diverse scenarios; in other words, we expect adapting models via fine-tuning will continue to be relevant for data-driven modeling of PDEs.  In the following section we expand on this point, as well as other important differences between TL in the PDE domain versus CV/NLP which we feel make our work especially important and timely for the SciML community.\n\n### R1-2: Differences between transferability in PDEs vs. more traditional CV/NLP tasks:\nWhile we do mention some TL characteristics unique to PDEs in the text, we agree they could be emphasized more as a primary focus of the work. Other reviews also have similar suggestions. Broadly, a major difference in the PDE domain is the diversity of scales, features, and solution behaviors across PDE coefficients as well as systems of interest in science. For example, while OOD shifts in CV/NLP typically involve things like different color distributions or semantic contexts, in a PDE system, it is possible to get significantly different behavior of the PDE solution as physics parameters (and/or initial and boundary conditions) change. Capturing these is critical to any scientific objective. These can manifest as new features (for example, increased emphasis on very different frequency scales of the solution causing significant shifts in detail, sharpness, and other image attributes to be visible in the PDE solution) and depend on the physics coefficients in a highly non-linear fashion. One of our goals is to systematically quantify the TL performance as these OOD patterns emerge in our target solutions through modifying the physics parameters that control this data distribution. We visualize this diversity and sensitivity for our PDE systems in Appendix Fig A.1 to highlight the above point. In general, this diversity can be far greater in larger scientific systems (such as bifurcation in fluid flows where the dynamics can change dramatically with relatively small changes to the PDE inputs) and further amplified by ill-conditioned or numerically stiff systems. From a deep learning perspective, the dense prediction (and TL) of solutions exhibiting these characteristics with input numerical scales that can span orders of magnitude becomes a challenging task. One specific example of this that we mention in section 3 and appendix A.5 is our normalization strategy, which our results are quite sensitive to. We find that the normalization of physics coefficients, inputs, solutions to similar numerical ranges is very important and especially so while TL–choosing the pre-training dataset normalization for the fine-tuned model yields significantly better performance than the normalization from the downstream dataset/task. \n\n### R1-3: Concerns about overfitting during few-shot fine-tuning:\nWe agree it is critical to control for overfitting in the few-shot fine-tuning experiments, and that it should be called out more directly in the paper. Not surprisingly, for the larger model sizes we observe some overfitting in few-shot fine-tuning, especially on extreme OOD shifts, and we partially mitigate this by employing early stopping and tuning the learning rate for each specific model and fine-tuning dataset size. We broadly find that smaller learning rates work best for smaller downstream datasets. More generally, it is important to note that the fine-tuning strategy itself and how it may change with dataset size is an open question in this domain, and our work is intended as just a first step by systematically quantifying the TL behavior for a variety of representative PDE systems with the same fine-tuning procedure. It is very likely in the PDE domain there are alternate fine-tuning strategies which would work well for TL on OOD data, potentially involving supervision using physical laws directly from the PDE systems. As we mention briefly in Sec. 3, a notable alternative for the few-shot TL scenario is in-context learning, which is adopted in an interesting concurrent work (Yang et al. 2023, https://arxiv.org/pdf/2304.07993.pdf). While these directions are very interesting and warrant further research, we consider them out of scope for our investigation, and highlight them as limitations/topics for future work in Sec. 5.\n\n\n\n"}, {"review_id": "j5Fk4tbGpw", "reviewer": "Reviewer_9JLh", "summary": "This paper analyzes the transfer performance of the pre-trained foundation model for scientific machine learning applications described by PDEs. A wide range of settings is considered: various sizes of downstream datasets, pretrained model, out-of-distribution data, as well as multi-tasks and applications. Conclusions, though not necessarily surprising, are drawn under these settings. ", "questions": "We can observe from the figures that the performance gap is reduced when more fine-tuning examples are observed. This is not surprising, and the advantage of the pre-trained foundation model is indeed when fine-tuning examples is less. In this case, the performance is worse. I am wondering how to interpret the error, and at what magnitude, the error becomes acceptable for these problems. ", "limitations": "The author adequately addressed the limitations. ", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- To the best of the reviewer's knowledge, this is a relatively comprehensive analysis of pre-training and transfer learning regimes over a diverse set of physical systems. \n\n- The presentation is clear and easy to follow, and the experiments are comprehensive and convincing. ", "weaknesses": "- For the experiments, more network architectures can be considered.\n\n- The findings provided by the paper are more or less consistent with NLP and cv tasks. This is not necessarily a drawback, but the paper would be stronger if it provides some findings that make Scientific Machine Learning different from other well-explored AI tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the positive feedback and constructive criticisms from the reviewer. While we are happy there are no major concerns, we address some of the specific suggestions below. \n\n### R2-1: Exploring only one architecture:\nWe agree this is a major limitation of our work, and we call this out in Sec. 1, 2 and 5. We choose the FNO partly because we consider PDE systems with periodic boundary conditions in this work, which FNO is well adapted for, and primarily due to its status as one of the more widely-used architectures in SciML. Since its introduction (in which all experiments focused on PDE systems), FNO and its variants have been applied to a variety of challenging SciML applications including large physical systems like global weather forecasting (https://arxiv.org/abs/2202.11214), multiphase flow (https://arxiv.org/abs/2109.03697), and seismology (https://arxiv.org/abs/2108.05421) to name a few. However, we would like to note our intentional choice to (1) only evaluate one architecture and (2) use the FNO, in particular. The primary reason for experimenting with a single architecture was to make the scope of the study more feasible from a compute and analysis perspective – as mentioned in Sec. 1 l45, the reality is that the domain of SciML for PDEs is far from converged on a single dominant architecture and the different architectures in circulation can have very different properties. This is partly due to the great diversity of physical scales, behavior, and geometries found in SciML applications; that is why we wanted to focus on some critical open questions (dataset diversity/scale, model scaling, and generalization across physics) and fix some of the other variables (model architecture, pretrain/fine-tune strategy) in order to enable a systematic study within a realistic compute budget. However, we do agree with the reviewer that our study should be repeated with different architectures, and comparative analysis between both existing popular SciML architectures (FNO, DeepONet, etc) as well as more conventional ones successful in general ML applications (e.g. ViTs and related variants) would be useful for the community.\n\n### R2-2: Results being more or less consistent with findings in CV/NLP domains:\nWe agree the differences in the PDE domain compared to CV/NLP should be called out more clearly in the paper, both in the introduction and in the results. A similar concern was expressed in R1-2 and we copy our response here: A major difference in the PDE domain is the diversity of scales, features, and solution behaviors across PDE coefficients as well as systems of interest in science. For example, while OOD shifts in CV/NLP typically involve things like different color distributions or semantic contexts, in a PDE system, it is possible to get significantly different behavior of the PDE solution as physics parameters (and/or initial and boundary conditions) change. These can manifest as new features (for example, increased emphasis on very different frequency scales of the solution causing significant shifts in detail, sharpness, and other image attributes to be visible in the PDE solution) and depend on the physics coefficients in a highly non-linear fashion. One of our goals is to systematically quantify the TL performance as these OOD patterns emerge in our target solutions through modifying the physics parameters that control this data distribution. We visualize this diversity and sensitivity for our PDE systems in Appendix Fig A.1 to highlight the above point. In general, this diversity can be far greater in larger scientific systems (such as bifurcation in fluid flows where the dynamics can change dramatically with relatively small changes to the PDE inputs) and further amplified by ill-conditioned or numerically stiff systems. From a deep learning perspective, the dense prediction (and TL) of solutions exhibiting these characteristics with input numerical scales that can span orders of magnitude becomes a challenging task. One specific example of this that we mention in section 3 and appendix A.5 is our normalization strategy, which our results are quite sensitive to. \n\n### R2-3: Interpreting error values and determining what level of accuracy is sufficient:\nThis is an excellent question, and slightly difficult to answer as the level of acceptable error in most SciML applications is a very domain-specific consideration. For instance in some inverse problem applications, an error of O(1%) may be very good, especially for cases where the noise in data collection can be higher. But there are also problems in scientific computing such as fluid dynamics problems where many more digits of accuracy are needed to correctly model a phenomena and respect the problem boundaries/scales. In general the domain of PDEs is a data-limited scenario as running PDE solvers to generate data is expensive, so the cost-accuracy tradeoff (see e.g. de Hoop et al. 2022 https://arxiv.org/abs/2203.13181) is evaluated on a case-by-case basis (some PDEs are easier to solve than others). In other applications, data may come directly from observations and it is very expensive to collect these. The approach we take in the paper is to focus less on the absolute error values and more on benchmarking against the baseline of training from scratch. Interpreting results from this perspective is especially useful when considering the different training data requirements to reach a desired level of accuracy, as we show in Appendix Figure B.2. This figure helps demonstrate the dramatic advantages of pretraining, even when the downstream dataset is OOD, but we felt the simple l2 error used in most of our plots would be easier for most people to quickly interpret.\n\n\n\n"}, {"review_id": "ZVg9QjDyws", "reviewer": "Reviewer_o2Fy", "summary": "This paper investigates the finetunability, transfer, and generalization properties of a foundational model applied to the field of scientific machine learning, as functions of finetuning dataset size, pretrained model size, and value of physics parameters governing the equations of the system. ", "questions": "Small suggestion: please provide common examples of physical systems/phenomena described by the PDEs explored in this work.", "limitations": "While disclosing the limitation of this analysis being limited to FNO, the author could be more forthcoming about the consequences of this limitations and the applicability, at a broader level, of these findings. Overall, however, the authors do an excellent job at not overstating their claims and clearly explaining the experimental setup and application domain this paper applies to.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written and well-organized, easy to follow and understand. The authors present their method clearly, and they provide a detailed evaluation of their method, with findings clearly highlighted and supported by strong empirical evidence. The experimental hypothesis are clearly stated up front and verified throughout the paper, with sound methodology and experiment design.", "weaknesses": "A weakness, also acknowledged by the authors at the end of Section 3 -- but without much contextualization --, is that they only explore the questions raised in the paper in the Fourier Neural Operator (FNO) architecture, and present it as a prototypical SciML architecture. Section 3 explains that this is a good choice for problems with periodic boundary conditions like the ones explored in this paper. However, it's unclear whether any of these results would hold for other types of OOD problems or other types of architectures. Therefore, without more contextualization, it's unclear whether these findings are generalizable and would allow the community to build on top of these findings, or whether they are anecdotal, and would only hold for the architecture and type of problems explored in this work.\n\nAnother weakness is that some of the findings, as currently present, merely seem to provide additional experimental evidence for notions that are already widely accepted in the community, such as the fact that, in the low downstream data regime, pretraining a model and then finetuning it on this data is preferable (and yields higher performance) than training a model on this data from scratch, or that finetuning small capacity models plateaus to a certain irreducible error even if data continues to increase. \n\nFinally, while the authors acknowledge that a concurrent paper covers the alternative approach of in-context learning (as opposed to finetuning), I would have expected the authors of this paper to compare against that alternative, as opposed to simply benchmarking against training from scratch, which is a much less tight comparison. I think this paper would benefit from the inclusion of that baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and constructive criticisms and address specific questions and suggestions below:\n\n### R3-1: Exploring only one architecture:\nWe agree with this limitation of using just the FNO and make this point in the paper. We are happy to expand on this more. This concern was also raised in R2-1 and we copy our responses below here:\nWe call out this limitation in Sec. 1, 2 and 5. We choose the FNO partly because we consider PDE systems with periodic boundary conditions in this work, which FNO is well adapted for, and primarily due to its status as one of the more widely-used architectures in SciML. Since its introduction (in which all experiments focused on PDE systems), FNO and its variants have been applied to a variety of challenging SciML applications including large physical systems like global weather forecasting (https://arxiv.org/abs/2202.11214), multiphase flow (https://arxiv.org/abs/2109.03697), and seismology (https://arxiv.org/abs/2108.05421) to name a few. However, we would like to note our intentional choice to (1) only evaluate one architecture and (2) use the FNO, in particular. The primary reason for experimenting with a single architecture was to make the scope of the study more feasible from a compute and analysis perspective – as mentioned in Sec. 1 l45, the reality is that the domain of SciML for PDEs is far from converged on a single dominant architecture and the different architectures in circulation can have very different properties. This is partly due to the great diversity of physical scales, behavior, and geometries found in SciML applications; that is why we wanted to focus on some critical open questions (dataset diversity/scale, model scaling, and generalization across physics) and fix some of the other variables (model architecture, pretrain/fine-tune strategy) in order to enable a systematic study within a realistic compute budget. However, we do agree with the reviewer that our study should be repeated with different architectures, and comparative analysis between both existing popular SciML architectures (FNO, DeepONet, etc) as well as more conventional ones successful in general ML applications (e.g. ViTs and related variants) would be useful for the community.\n\n### R2-2: Results being more or less consistent with findings in CV/NLP domains:\nWe agree the differences in the PDE domain compared to CV/NLP should be called out more clearly in the paper, both in the introduction and in the results. A similar concern was expressed in R1-2 and we copy our response here: \nA major difference in the PDE domain is the diversity of scales, features, and solution behaviors across PDEs. For example, while OOD shifts in CV/NLP typically involve things like different color distributions or semantic contexts, in a PDE system, it is possible to get significantly different behavior of the PDE solution as physics parameters (and/or initial and boundary conditions) change. These can manifest as new features (for example, increased emphasis on very different frequency scales of the solution causing significant shifts in detail, sharpness, and other image attributes to be visible in the PDE solution) and depend on the physics coefficients in a highly non-linear fashion. We visualize this diversity and sensitivity for our PDE systems in Appendix Fig A.1 to highlight the above point. In general, this diversity can be far greater in larger scientific systems (such as bifurcation in fluid flows where the dynamics can change dramatically with relatively small changes to the PDE inputs) and further amplified by ill-conditioned or numerically stiff systems. From a deep learning perspective, the dense prediction (and TL) of solutions exhibiting these characteristics with input numerical scales that can span orders of magnitude becomes a challenging task. One specific example of this that we mention in section 3 and appendix A.5 is our normalization strategy, which our results are quite sensitive to. \n\n### R3-3: Examples of scientific systems:\nThis is a very good point and we will add examples for the three systems we considered that are fundamental in many science applications. The Poisson’s equation and its variants are used in electrostatics (potential fields caused by an electric charge distributions), solid mechanics (deformation fields through external forces), fluid mechanics (pressure distributions in incompressible flows), amongst other applications in astrophysics, chemistry, mechanics, image processing. The advection-diffusion and its variants are used in fluid mechanics (transport of particles, energy, and other physical quantities), atmospheric physics (wind forecasting), semiconductor physics, solid mechanics (for example, eulerian formulations of cancer growth models). The Helmholtz system, is a particularly challenging system, and has applications in electromagnetics and acoustics (propagation of waves) and seismology (earthquake simulations). \n\n### R3-4: Comparisons to in-context learning:\nWe agree with the reviewer that the concurrent work that appeared with our paper in introducing in-context learning is an interesting methodology to compare against and repeat our systematic analysis. However, that work (Yang et al. 2023, https://arxiv.org/pdf/2304.07993.pdf) focuses on much simpler systems (ODEs and 1D) and at much smaller scales and is also more suitable for cases where only limited amount of data is available for a target problem (3-5 in the case of that paper). Generally, fine-tuning approaches should perform better when more data is available for a target domain. As for the small data regime, however, we agree that in-context learning could be a very suitable method to apply foundation models. This is a direction that we are actively pursuing and we will clarify this point in the revised version of the paper.\n\n\n"}, {"review_id": "UOnnCteTE6", "reviewer": "Reviewer_QnfD", "summary": "This work explores and demonstrates how a \"pretraining+finetuning\" paradigm may be leveraged for neural networks applied to PDEs. Specifically, for three different systems governed by distinct PDEs, the authors study how different factors influence performance: (1) downstream dataset scale; (2) model scale; (3) OOD-ness of the downstream data; (4) pretraining on systems governed by different physics. The authors find that pretraining systematically improves performance, delivering significant performance gains in limited downstream data settings -- which are particularly relevant to SciML, which is often more data-constrained than NLP. ", "questions": "Overall, this is an interesting paper addressing important topics for the community, with the potential to be highly valuable. It is however slightly held back by some concerns around the showcased experiments. Addressing these would further validate the wide applicability of the approach, confirming its significance and increasing the quality of the work. Accordingly, I am rating this work as a **Weak Accept (6)** but would be willing to increase my rating should my concerns/questions be addressed. \n\n**EDIT: following rebuttal, I have updated my score to an Accept (7)**.\n\n* **Q1.** (W1.1.) Could the authors provide results on SYS-3 for downstream dataset scaling and model scaling?\n\n* **Q2.** (W1.2.) Could the authors provide additional results for Figure 5./Q4, demonstrating the approach on SYS-2, and across all systems but OOD? (like in Figure 2).\n\n* **Q3.** (W1.3.) Could the authors comment on why the extreme OOD in Figure 4d) behaves differently?  \n\n* **Q4.** (W2.3.) Could the authors clarify their statement on l227/provide a reference? \n\n* (W2.2.) As a suggestion, to avoid confusion for readers, I would recommend systematically saying \"few-shot finetuning\" and not using \"few-shot learning\": l73 \"few-shot TL\" could be unclear, l132 \"few-shot learning\" --> \"few-shot finetuning\", l221 \"few-shot learning\" --> \"few-shot finetuning\"", "limitations": "The authors discuss some of the limitations of their work in the conclusion, but I would encourage a more critical look regarding when the proposed method could fail. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* **S1.** This work could be very significant for the community: it explores important topics for SciML, presents extensive strong results, and is easy-to-follow, well-structured, and pleasant to read. The paper is also extensively detailed (regarding data setup, architectures, etc.).\n\n* **S2.** The authors study a variety of scenarios (OOD-ness, downstream dataset size, etc.) and perform appropriate comparisons with a \"from scratch\" baseline. The identified strength of the approach (in the low downstream data regime) answer particularly well challenges in the field.\n\n* **S3.** The authors open-source their code, which will help with propagating their work, increasing its impact, and enabling better reproducibility. ", "weaknesses": "* **W1. The experiments could be more systematic across the three systems proposed.** \n    * **W1.1.** The SYS-3 setup is not featured for downstream dataset scaling & model scaling; it would be valuable to see it in these scenarios to further validate the method, especially since it is described as the most challenging setting.\n    * **W1.2.** The behaviour over multiple operator could be better explored: it is lacking a comparison with SYS-2, and is done only on a very \"in-domain\" case (while Figure 1 for instance is done with disjoints domains) thus causing the n-shot behaviour to be flat. \n   * **W1.3.** (minor) In Figure 4d) (with extreme OOD on SYS-1), the from scratch behaviour is different, never catching-up with the TL case. This diverges from the trend and warrants some explanations. \n\n* **W2. Some of the framing around large language models is inaccurate.**\n   * **W2.1.** l25 \"An important aspect of the trained foundation model is the notion of emergence—the model is able to perform tasks seemingly different than those for which it was trained by leveraging shared features across the training tasks.\" It is misleading to characterise emergence with this sentence. As discussed in Wei et al., 2022 emergence is more accurately described by \"abilities that are not present in smaller-scale models but are present in large-scale models\".\n   * **W2.2.** The few-shot terminology can be confusing in some cases. For large language models, zero/few-shot is synonymous with in-context learning. Mentioning few-shot alone simply means prompting the model. Here, in this work, it systematically means finetuning the model on these few shots. \n   * **W2.3.** l227 \"This is motivated by observations in NLP that after a critical model size, we can expect significantly better adaptation performance\" it's unclear what the authors refer to here, as finetuning efficiency in NLP is closely related to sample efficiency and is known to increase smoothly with model size (Kaplan et al., 2020). \n   * **W2.4.** (minor) l145 the focus on GPT models is a bit odd: finetuning applies to all classes of modern language model, and in fact the GPT (i.e., causal decoder-only) models are somewhat unique in their ability to not require finetuning. Conversely, they do not perform as well for fine-tuned tasks as other models: see What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? Wang et al., 2022. \n   * **W2.5.** (minor) l25 the references to scaling laws are missing a citation for Hoffmann et al., 2022, which is the reference work at this time and which highlighted the need for joint scaling of data and model size instead of model size alone predominantly. \n\n* **W3.** Smaller nits: l44 \"TL\" to refer to transfer learning as not been introduced as an acronym. \n ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback, thorough assessment and valuable comments/suggestions regarding our paper. \n\n### R4-1: W1: Additional results for SYS-3/SYS-2 to be more systematic\nThat is a good point. We should mention that SYS-3 (helmholtz) was chosen specifically as a system to stress-test our findings rather than just a third PDE system to boost our results. It is a well-known system in scientific computing that demonstrates very challenging numerical issues while solving. This system shows dramatic frequency spectrum changes as coefficients change and can also exhibit ill-conditioned behavior where the solution norm increases by a large amount when the coefficient changes a little, making it particularly hard for a neural network model to continuously model (some examples in Appendix Fig A.1), and even harder to generalize. Our goal here is to investigate if such difficulties preclude us from building a model trained on multiple PDEs. We agree that this motivation can be made more explicit and expanded with more detail in l166 as well as in the results. Specifically, we address the weaknesses mentioned in the review below:\n\n**W1.1/Q1**: The downstream dataset scaling can be seen for SYS-3 in Fig. 5.b for in-distribution and Appendix Fig. B.3.f for OOD (the 'circle' labels are the downstream dataset scaling). As we can see, the in-distribution SYS-3 shows similar properties, whereas the OOD performance is poorer than the other systems (the model shows high errors for coefficients that are OOD, reinforcing the challenging nature of this system). For model scaling, we agree with the reviewer to include the SYS-3 scaling result as well. We repeat our experiments for SYS-3 in-distribution and will include the result in Fig 1 of the attached PDF in the global response in the paper.\n\n**W1.2/Q2**: The SYS-2 results as well as OOD results are in Appendix Fig. B.3. Due to space constraints, we were unable to place this figure in the main text. We observe that mixing datasets (multiple operators) and fine-tuning the pre-trained model still demonstrates gains across all the systems: see Appendix Fig. B.3 for all three systems, evaluated both in- and out-of-distribution. Even for the OOD SYS-3 which shows poor performance, there is no additional degradation from using a pre-trained model trained on the mixed (multiple operators) datasets. \n\n**W1.3/Q3**: The extreme OOD case is a more challenging system with large anisotropy in diffusion, so we expect the network to have more difficulty modeling this case. We note that the errors on the y-axis are increasing (for the same number of downstream examples) as we move more OOD and to more challenging systems. In other words, larger amounts of data might be needed to obtain lower errors. We add two more data points with 64K and 128K dataset sizes for the extreme OOD case and observe that at 128K, the curves from training from scratch and fine-tuning converge as shown in Fig 2 of the attached PDF in the global response. \n\nOur interpretation of this result is that for more extreme datasets as this, more samples are required to reach a given accuracy; we also note that our method of generating OOD data and measuring the extent of OOD the samples are, is based on the diffusion tensor eigenvalues (and other coefficients for the other two systems), and the categories of “Mild/Medium/Large” distribution shifts in Table 1 are qualitative categories we use to help the reader understand each dataset easier; the exact point at which from-scratch training intersects the error of a pre-trained model will not necessarily always occur in the same place for all systems and distribution shifts. \n\n### R4-2: W2: Framing of language around NLP models\nWe mainly include examples of LLMs as a regime of DL where foundation models have been shown to be successful and use them to frame our high-level discussion points. However, we agree the language used could be improved and clarified and deeply appreciate the reviewer’s suggestions and feedback here.\n\n**W2.1**: That is right, we will fix this in the revised version of the paper and clarify based on the reviewer's suggestion.\n\n**W2.2**: We will change the terminology everywhere to “few-shot finetuning” as-per the reviewer’s suggest to remove this confusion. We will also clarify that this is what we mean by “few-shot TL”.\n\n**W2.3/Q4**: With this sentence, we simply meant that larger models show better testing performance given a dataset size and refer to Fig 2 of Kaplan el al. (2020). We agree that the wording is quite confusing and will clarify to the above sentence.\n\n**W2.4**: We will clarify this in the text. We mostly mentioned GPT as a popular example of a foundation model but we see how this can cause confusion.\n\n**W2.5**: We will add this reference to l25.\n\n**W3**: We will fix this reference."}]

Author rebuttal:
We thank all reviewers for their insightful comments and suggestions. We attach additional figures for the model scaling experiments for Helmholtz and the additional data points for the extreme OOD test-case addressing R4-1 comments in Fig 1 and Fig 2, respectively. We refer the reviewers to the individual rebuttals for specific comments.

Meta-review: This paper studies the transfer performance of the pre-trained foundation model for scientific machine learning applications described by PDEs, and it provides a comprehensive analysis of pre-training and transfer learning regimes over a diverse set of physical systems. 

Most of the reviewers acknowledge the significance and novelty of the work, and find the presentation of the work clear and easy to follow.

In the final version, we suggest the authors carefully incorporate all the feedback provided by the reviewers.
Decision: Accept (poster)