Pretraining a Neural Operator in Lower Dimensions
Authors: AmirPouya Hemmasian
Amir Barati Farimani
Venue: Accepted by TMLR ()

Abstract: There has recently been increasing attention towards developing foundational neural Partial Differential Equation (PDE) solvers and neural operators through large-scale pertaining. However, unlike vision and language models that make use of abundant and inexpensive (unlabeled) data for pretraining, these neural solvers usually rely on simulated PDE data, which can be costly to obtain, especially for high dimensional PDEs. In this work, we aim to Pretrain neural PDE solvers on Lower Dimensional PDEs (PreLowD) where data collection is the least expensive. We evaluated the effectiveness of this pretraining strategy in similar PDEs in higher dimensions. We use the Factorized Fourier Neural Operator (FFNO) due to having the necessary flexibility to be applied to PDE data of arbitrary spatial dimensions and reuse trained parameters in lower dimensions. In addition, our work sheds light on the effect of the fine-tuning configuration to make the most of this pretraining strategy. Code is available at https://github.com/BaratiLab/PreLowD.

Published in Transactions on Machine Learning Research (11/2024)
Pretraining Neural Operator in Lower Dimensions
AmirPouya Hemmasian
Semmmas@emna edu
Department of Mechamical Engineering Carnegie Mellon University
Amir Barati Farimani
barati@emu..cm edu
Department Mechanical Engineering
Carnegie Mellon University
Reviewed on OpenReview: Atips: // openreview. net/
met/ forum? id- ZewaRoZeh.
Abstract
There has recently been increasing attention towards developing foundational neural Partial Differential Equation (PDE) solvers and neural operators through large-scale pretraining- However, unlike vision and language models that make use of abundant and inexpensive (unlabeled) data for pretraining, these neural solvers usually rely on simulated PDE data, which can be costly to obtain, especially for nigh-dimensional PDEs. In this work, we aim to Pretrain neural PDE solvers on Lower Dimensional PDEs (PreLowD) where data collection is the least expensive. We evaluated the effectiveness of this pretraining strategy in similar PDEs in higher dimensions. We use the Factorized Fourier Neural Operator (FFNO) due to having the necessary flexibility to be applied to PDE data of arbitrary spatial dimensions and reuse trained paramcters in lower dimensions. In addition, our work sheds light on the effect of the fine-tuning configuration to make the most of this pretraining strategy. Code is available at ltttps///ithubb.eo saairtuub..oom/Baaaaaaaaaa PreLowD
1 Introduction
Many of the recent brcakthroughs in artificial intelligence and deep learning are the fruits of largo-scale pretrained models in all kinds of applications, ranging from computer vision (Chen el. al.] 2023). language processing (Devlin et al.) 2018), prediction of molecular properties (Wang et al. 2022 Yu et al., 2021). and so on. Protraining makes use of abundant data to Iearn useful and gencralizable features and patterns that can be utilized in downstream task. The effectiveness of using pretraining models is generally more significant. when the downstream problem is complicated and/or the training data scarce and expensive collect (Chakraborty et al. (022)). In such cases, a simple model is prone to underfitting and a complex one to overfitting. Using a pretrained model can spare the learning of basic and fundamental features from scratch and reduce the mentioned risks in the downstream task.
There are several core strategies for pretraining neural networks in different applications depending on the task and data at hand. For example, a common strategy in many computer vision tasks is to use a pretrained model on the ImageNet dataset with simple task such as classification (He et al.] 2019). Due to the large amount of data, this stratogy has been beneficial in most computer vision tasks (Ridnik et al., 2021). This case of pretraining is done in a supervised learning framework, which requires large amount of data that expensive to collect but now is available. However, this is not the situation in many cases and that is where self-supervised learning comes into play.
Self-supervised learning (Liu et al. 2021 is similar to supervised learning with the difference that labels are automatically or trivially obtained from the data by itself without human input and are therefore inexpensive and fast to obtain. One of the most common and effective self-supervised pretraining strategics masked autoencoding and prediction (Devlin et al., 20188 He al., 20222 Wei al., 2022; Song et al., (2020). in which the model has to predict missing parts certain features or targets from masked, noisy, modified

---

Published in Transactions on Machine Learning Research (11/2024)
version of the imput. This strategy allows the model to extract and learn general patterns and features based on the context in which different parts of the data frequently appear. This framework is the key to the success behind many famous language models like BERT (Devlin et al., 2018) and CPatGPT (Wu et al. 2023). Image and video applications have also been shown to benefit from such strategies (He et al. 2022 Tong el al. ).
In addition to masked autoencoding and prediction tasks, other strategies have been introduced and proven effective. In general, any modification that maintains the semantics and central features of the data can be used as labels for model to learn and thus extract generalizable features (Hu et al.
2019 These secondary tasks for self-supervised learning are also known as proxy tasks. Some examples of proxy tasks include sorting a shuffled input (scrambled words or image patches) (Kim et al. 2019 Panda et al. 2021). regressing a similarity or relevance score between pairs of data samples generated by augmenting or modifying identical or different raw samples (known as contrastive learning) (Hénaff et al.) 2021 Rethmeier & Augenstein 2023
Jaiswal et al., 2020), or similarly aligning the representation of encoding the same object from different data modalities (audilory, visual, and textual) (Lin et. al., 20200 Wang et al.] 2023). Due to the great novelty and success of all these strategies, neural PDE solvers and neural operators have also started to adapt the framework of pretrained models.
2 Related works
With the increasing availability of large PDE datasels and the success of pretraining strategies in classical applications, many researchers have been adapting such techniques for neural PDE solvers and neural op- crators. simple question that might arisc is whether it effective to use a pretrained model trained on one PDE to learn the same PDE with different coefficients or different PDEs. To that point, SSbramanian et al. (024)) investigated the transfer of the Fourier neural operator (FNO) to different coefficients, different PDEs, and how the result may vary with model scale and dataset size. They also tried pretraining the model on a mixed dataset consisting of several PDEs, which is the main focus of many following works. Some examples of such works include Multiple Physics Pretraining (MPP) by McCabe el al. (2023), Unified PDE Solvers (UPS) by Shen et al. (2024), Universal Physics Transformer (UPT) by Allin et al. (2024), PDEFormer by Ye et al. (2024), and Denoising Pretrained Operator Transformer (DPOT) by Hao et al. (2024b. These frameworks focus on designing a network architecture that is versatile enough learn all PDEs in the training dataset, well other necessary techniques such unifying the representation space and balancing the data feed between PDEs during training (Hao ct al., 2024). The common characteristic of all these works is that the pretraining task and the downstream task are the same, that is, predicting the next time-step of the system (time-dependent PDE) or output approximation (boundary value problem).
Although not as popular as in classical applications, pretraining via proxy tasks has also been explored for neural PDE solvers, including elaborations on how to transfer and tune the pretrained models. Yang et al. (2023) defined a context-based prompt answering architecture, which they showed to be a few-shot PDE learner. Zhang et al. (2024) deciphered and integrated invariant fcatures of physical systems in time and space in contrastive learning framework and used them to fine-tune the first layer of U-Net or FNO. Lorsung & Farimani (2024) used PDF. coefficients to develop a more flexible pretraining approach based on contrastive learning. 'Tripura & Chakraborty (2023) selectively fine-tuned certain components of their designed architecture while preserving the pretrained weights of the pavelet-based components. Zhou & Farimani (2024) examined the capability of masked autoencoders as PDE learners, and later investigated a wide range of pretraining strategics for several neural operators and PDEs (Zhou et al. 2024). Mialon et al. (023) proposed contrastive learning strategy based on Lie symmetries in PDEs. Chen et al. (2024) applied classical pretraining strategies such reconstructing blureed or masked inputs for neural PDE solvers. They used unlabeled data during pretraining, by which they meant the initial conditions in their datasets. Therefore, the pretraining stage actually relies on cheap data rather than expensive PDE datasets.
Up to this point, virtually all the cited works relied on a large pretraining dataset with a collection cost similar to that of the downstream task. Some focused on what one might rather call a multitask lcarning framework, while others looked into different ways to use the same kind of data with strategics to design proxy tasks or use and tune pretrained models. In contrast, we propose novel pretraining approach to
2

---

Published in Transactions on Machine Learning Research (11/2024)
increase the accuracy and pretraining efficiency of neural operator for high-dimensional PDEs, that is, to pretrain the model in lower dimensions (1D). Since the cost of data collection and model training in 1D can be significantly lower than in 2D or 3D, there is a strong incentive to make use of this pretraining strategy. For example, to have a certain resolution N per axis, a 1D and 2D system would be discretized to N and N2 points respectively. In traditional numerical solvers, each point represents an explicit update equation or an implicit equation to be solved. If the implicit solver relies on matrix inversion, can increase the cost further to power of 3, leading to a cost of O(N²) in ID and O(N5) in 2D. In deep learning models, point-wise transformations are applied to each point, and field transformations (like those in the Fourier domain) have a cost that increases with the number of dimensions and the size of each dimension.
The pretraining strategy proposed in this work is not applicable to any arbitrary architecture or PDE, but when possible is much less costly than other pretraining approaches. In this work, we experiment with two fundamental PDEs that have a valid 1D and 2D version, and neural operator that passes the requirement to be utilized in this framework. Since the fundamental blocks of PDEs are 1D derivatives and operators, this work is polentially a step towards building foundational models for solving PDEs as well. In our experiments, we observe that the average relative error of a prelowded factorized Fourier Neural Operator (FFNO) on the 2D diffusion equation can be 50% smaller than the same model trained from randomly initialized parameters over 5 rollout steps, which may not be achieved by simply increasing the amount of training data for the main training task. The gain of prelowding the model seems to amplify over more prediction time-steps and more rapidly changing systems (higher diffusion coefficient). Although the same results are not observed for the advection equation, this sheds light on the potential success of this approach in certain situations.
3 Background
3.1 Neural operators
Unlike the typical task of neural networks which is to approximate mappings between Euclidean spaces, neural operators approximate mappings between function spaces. An operator G : A - U maps an input function E A to an output function u U. In the application of solving PDEs, a can include PDE coefficients and initial and boundary conditions, and is the solution of the PDE. In this work, are considering time-dependent PDEs, where the input is the state of a physical system, and the output is the state at a later time. The operator then learns the mapping 2, + Ut
Although variables are defined and processed as functions, a discretized representation is eventually needed for numerical calculations. A higher resolution and a finer grid typically lead to more accurate results, but with the downside of higher computational cost and memory consumption, both for traditional numerical solvers and neural solvers. Typically, the cost increases exponentially with respect to the number of spatial axes. However, this is not the case for certain neural solvers that factorize the calculations across spatial axes. Some examples of such models are the Factorized Fourier Neural Operator (FFNO) (Tran et al.
2021
Axial Vision Transformer (AViT) (Ho et al. 2019 Bertasius et al. 2021 used in MPP (MeCabe et al. and FactFormer (Li et al. 2024b c). The central parameters and calculations of such models are defined per
2023
spatial axis, which makes the parameter count and computational cost the sum (rather than the product) of the single-axis amount, leading to linear cost with respect to the number of spatial axes instead of an exponential cost. Moreover, it is possible to reuse the parameters of pretrained 1D model when learning to predict relevant system in higher dimensions, Our quest in this work is to investigate the potential of model Pretrained in Lower Dimensions (PreLowDed) to be finc-tuned in higher dinensions. We choose to explore this stralegy with the Factorized Fourier Neural Operator (FFNO) (Tran el al., (202)) because of its efficiency due to the fast Fourier transform (FFT) and factorization. Following works can look into composition and factorization of other attention-based architoctures like Galerkin Transformer (Cao) (021) and OFormer (Li et al. 2022), as well as using this strategy in a learned encoding space (Hemmasian & Hemmasian & Farimann
Barati Farimani 2023
2024 Li et al. 2024a).
3

---

Published in Transactions on Machine Learning Research (11/2024)
a)
P
(()
A
Q
Uc+1
b)
FNO Layer
FFNO Layer
XW; E R#xcH + E CR
x E RcH +b, E IR."
K
H
XW RHXH
+b F
K. in 2D
R HXXXY
E !XXM XMy
CHMMy
E RRIXXXY
FFT
LP LPF My
xR @HIXHMM,XM,
IFFT
d)
K F in 1D
E R
CHXNx
C
HHXM
[B
FFT
LPFM
xR, @HXHXMg
IFFT,
JCp in 2D
FFT,
E X
LPFM
E
Cli xMXMyX
xR, QHKHXM,
IFFT.
E Mxxxxx
E IR HXXXX
E cxxxxy
XixxxMy
E CHXXMM
FFT,
LPFMy
XRy CHXHXMy
IFFT,
Figure 1: a) General schematic neural operator. b) The nonlinear operator layer in FNO and FFNO. c) The kernel integral operator FNO. LPF stands for a low pass filter that keeps the first few Fourier modes in each axis and discards the higher frequency modes. d) ID and 2D factorized kernel integral operator in FFNO. The red arrows show the possible transfer of pretrained parameters from ID model to 2D model if M,- My.

---

Published in Transactions on Machine Learning Research (11/2024)
3.2 Factorized Fourier Neural Operator
The original Fourier Neural Operator (FNO) was introduced by Li et al. (2020a) in a line of work based on an architecture analogous to typical neural networks. By replacing linear layers in a neural network with linear operators, Li ct al. (2020b) proposed an architocture template shown in figure Ta to construct a ncural operator through iterative linear and nonlinear transformations of the imput:
o... == G(a)= (2 o ( ( oP) (o),
where is function composition, L is the number of layers, P is the input projector from the physical space to the latent space, (() is the Ith non-lincar operator layer, and Q is the output projector from the latent space back to the physical space. A nonlinear operator layer of FNO executes the following transformations on the latent representation:
K0 (2(1-1)) - -IFFT R -R. -FF()
(2)
200 = c( (2(-1)) =o (w())((1)) b + (=(11)))))
where XC() is kernel integral operator that executes a convolution integral via matrix multiplication in the Fourier space, (2()) is the Fourier weight matrix, a R + R is a point-wise nonlinear activation function, and (())((-1) + b(9) is an affine point-wist wise map in physical space. You can see a simple illustration of the FNO layer on the left. figure D, and a 2D kernel integral operator block in figure D. Assuming the same number of Fourier modes in each axis (M₂ = My = M), each layer has distinct or shared complex weight matrix R()) containing H²M M complex parameters, where H represents the hidden or latent dimension (also known width) and D is the number of spatial axes.
Later, Tran et al. (201)) proposed factorized version of FNO (FFNO) with a factorized kernel integral operator KF and modified layer (();
kC(l (2(111)) = Σ IFFT ( FFFo(2
(4)
200) = ((() (2 = (1 + w/ o W (1) x KP (2 12)- + +660 + 6000
(5)
Here, Kr is a factorized kernel integral operator, R.O the Fourier weight matrix of axis d, and o, W.b rep- resent the activation, weights, and biases of the feedforward layers respectively. simple visualization of the FFNO layer and its factorized kernel integral operator are provided in figure Dp and [ [1 respectively. FFNO fully preserves the residual connection throughout the layers to keep the original input and its information as much as possible, and applies the nonlinearity to the output of the kernel integral instead. It also uses a two-layer feedforward layer instead of the single-layer one in FNO, which is inspired by the transformer architecture (Vaswani et al. 2017). The central fcature of FFNO is the factorized kernel intogral operator whose cost and parameters count in each dimension is O(H²M), summinng up to O(H2MD).
Not only does the factorized architecture save a lot of computational cost and memory usage when storing, training, and testing the model, but it also allows weights to be of the same shape across different axes if the number of modes is the same. This opens up the possibility of transferring weights across different problems with an arbitrary number of axes. In this work, we evaluate the effectiveness of this strategy on the advection and diffusion equations, two simple but fundamental PDEs.
4 Methodology
Before the formal definition of how to prelowd a neural PDE solver or operator, a couple of requirements must be met. First, the architecture of the neural operator has to be compartmentalized per spatial axis, which is usually indicated by being called an axial or factorized neural solver or operator. This enables the generalization and recyeling of pretrained parameters of an axial module other axes in the high- dimensional PDE of the downstream task. The next requirement is the relevance of the pretraining PDF,
5

---

Published in Transactions on Machine Learning Research (11/2024)
which is defined in a space with fewer dimensions. Some terms in PDEs such as the gradient or the Laplacian, or their nonlinear combination with other terms, can be defined and appear in spaces with different numbers of dimensions. The existence of such similar terms in PDE is a reasonable justification to use it to construct the pretraining dataset. The step-by-step pipeline of prelowding a factorized or axial noural PDE solver can be found Algorithm
Algorithm PreLowDing a neural PDE solver
Input: a neural PDE solver and a target (downstream) PDE dataset to learn Step 1: Verify the neural solver has a factorized/asae /xxial architecture.
Step 2: Choose a relevant low-dimensional (1D) PDE for pretraining. A relevant PDE would consist of
similar terms and similar physical phenomena (advection, diffusion, etc) in the lower dimensions. Construct a training dataset from the chosen PDE(s).
Step 3: Train the neural operator on the pretraining dataset. This step is where the pretraining is executed. Step 4: After training, replicate the axial module(s) of the model to the number of dimensions in the downstream (target) PDE.
Step 4.5: Apply any appropriate modification before the neural solver is trained (fine-tuned) on the downstream dataset. An example of such modification is freezing certain components not to be changed in the fine-tuning stage.
Output: The prelowded model ready to be trained (fine-tuned) on the downstream PDE
Moving on, we will continue the framework focusing on FFNO as our choice of neural PDE solver. As ex- plained previously and illustrated in figure a 1D FFNO and a 2D FFNO can share all their corresponding parameters the number of preserved Fourier modes is the same in x and y. The projection layers and the feedforward layers are point-wise transformations in the physical space, and all Fourier weights are defined for single axis. After transferring the pretrained parameters, we explore different configurations to freeze or fine-tune them for the downstream task. In classical applications like computer vision, the typical strategy is to freeze the weights of the initial layers (also known as the stem) and only tune the parameters of the final layer(s). This is justified due to the hicrarchical nature of the architecture and the learned features. Moreover, tuning the model is cheaper since the backpropagation is not executed all the way back to the input, and the sheer number of trainable parameters is also decreased by freezing the stem of the model. In addition, the scarcity of training data in the downstream task introduces the risk of overfitting, which can be mitigated by this strategy due to the low number of trainable parameters. Keeping this in mind, we have to decide how to choose the best way to perform the fine-tuning considering the architecture of neural operators and PDE data.
The first difference that we pay attention to is the absence of a hicrarchical architecture and feature extraction in models like FNO and FFNO. Unlike convolutional neural networks that have a shrinking architecture towards the output, these neural operators maintain the same resolution and stay in roughly the same latent space throughout their hidden layers. Therefore, the classic fine-tuning strategies of classic applications may not be the best choice here. However, these models consist of several types of components, each of which may learn certain properties and features of the system. For example, the parameters of FFNO belong to either projectors, kernel integral operator layers, or feedforward layers. We can choose to fine-tune only a subset of the model's components and see how the tuned model performs in the downstream task. This can also pave the path toward the interpretability of such models by breaking down the information learned by each component.
In addition to choosing the tunable subset of the model based component type, we also consider fine-tuning based on the chronological order of the layers. This includes not only the classical approach of fine-tuning the last layer(s), but also fine-tuning the first layer(s) or combination of both. If we want to try all the possible configurations, the number of experiments exponentially increases, since we may choose to tune or not tune each component of each layer. Therefore, we narrow our options down to eight configurations as shown in table [ We will represent the model without pretraining as CO. The projector layers P, Q are left to be tuned in all cases. The configurations C2 to C8 were selected so the tunable modules are either in proximity to the model's input or output, or are of the same type (Fourier feedforward). While CI
6

---

Published in Transactions on Machine Learning Research (11/2024)
represents fully tunable model, C2 and C3 have certain tunable layer type across all layers and the remaining configurations have one both layer types from the first and/or last layer.
Table 1: The trainable components of the fine-tuning configurations.
parameters
C1
C2
C
C4
C5
C6
C7
C8
R
v
X
v
W B(O)
4
x
x
x
( < <I<L)
/
x
x
x
x
x
x
W ((0) (0 <<< L)
V
x
x
x
x
x
x
R!
V
x
x
x
x
W B(T)
V
v
<
v
v
We will evaluate the performance of the models by comparing the next-step prediction error and the average error over 5-step autoregressive rollout. We also conduct experiments with different numbers of training samples in the downstream task. As we shall see, the advantage of pretraining depends on the availability of training data in the downstream task.
5
Experiments
5.1 Datasets and objectives
The general format and domains of the PDEs in this work are shown in equation 6 where N is a linear or nonlinear function, e is the vector of the PDE coefficients, u the quantity of interest and f, are the time and space variables, respectively. Each sample of a dataset consists of the evolution the system the specified time domain sampled every At 0.05, resulting which the model will be trained. The spatial resolutions for the 1D and 2D datasets are 642, respectively. Different samples of each dataset governed by the same equation and coefficients, and differ only in the initial condition defined by equalion where A;, ni di are random variables drawn from uniform distributions, similar to the multi-dimensional datasets in PDEBench assume periodic boundary conditions in all cases.
snapshots and 20 input-output pairs on
1024 and
(Takamoto et al. 2022). We
U = N(c, u, Uny Uzz: May:....
(6)
u = u(x E R, tE [0, 1}, X (0,1), (1
uo(x) = u(xx 1= 0) Σ A,sin(2TT,x + +
(7)
For each dimensionality and each value of c, 10000 samples were generated, 2000 of which are held for validation. Since the charactcristics of typical scenario for pretraining a model the high collection cost and scarcity of data, set up the experiments to resemble such scenarios. For each case (a particular PDE with particular value for the coefficient)), we first pretrain the model on the relevant 1D pretraining dataset. Then, we fine-tune the pretrained model with different amounts of available downstream training data. That is to find out the relationship between downstream dataset size and the gain of pretraining compared to training from scratch. To clarify, the experiment for each PDE and each value for the coefficient(s) of the PDE are separate and independent of cach other. For example, the downstream task is to learn 2D diffusion with a diffusion coefficient of 0.002, the pretraining dataset consists only of samples from 1D diffusion equation with the same diffusion coefficient of 0.002.
For the downstream task, we compare model trained from random initialization and 8 pretrained models with fine-tuning configurations shown in I The objective of both training and validation is the relative Le norm, also known as nRMSE (Takamoto et al. 2022), of the prediction enror shown in equation X where j.y are the model output and target value for the quantity of interest, respectively. In our application, y is the state of the system at the next time-step. We train the model on the next-step prediction loss and evaluate the model using the same metric the validation dataset, well the average loss over 5-step
7

---

Published in Transactions on Machine Learning Research (11/2024)
autoregressive rollout similar to what was done in PDEBench Takamoto et al. 2022) Finally, we each experiment with 3 different random seeds and report the average result to take into account the randomness of the network initialization, as well as the randomly selected subset of the 2D training dataset. Now, we will look at the specifics of cach equation, the result for each dataset, and discuss the effect of the fino-tuning strategy, the number of training samples, and the PDE coefficient, based on the observed results. We will continue the discussion here using concise plots. Please refer to Appendix for detailed numerical tables on all the curated results.
LL2 v) 110 - v|12
|
Each training stage consists of 5000 iterations of the AdamW optimizer with an initial learning rate of 0.001. The learning rate is multiplied by 0.2 when the loss reaches plateau and does not improve for more than 100 iterations. Our choice of architecture for FFNO has 4 hidden layers with a latent dimension of 128 and 16 Fourier modes in each axis.On our GeForce RTX 2080 Ti Nvidia GPUs, with a batch size of 64 and 5000 optimization iterations for each stage, the pretraining takes about 3.35 minutes while the fine-tuning or training for the 2D task takes about 21 minutes. This is an example of how the training cost differs for a 2D PDE compared to 1D. This is despite the resolution in 2D being 64 while the 1D data has a resolution of 1024.
5.2 Diffusion equation
We generated six different ditfusion datasets by solving equation 9 in ID and 2D for each coefficient. An implicit Euler scheme with At 0.001 was utilized.
u = vv V E
(9)
The results for the diffusion datasets are shown in figure 2 training samples, and the y-axis represents the average loss on the 2000 validation samples. We exclude the instances trained with only 1 or 2 training samples due to their unacceptable amount of error. It appears that the fine-tuning configurations C1, C2, and C8 perform as well or better than the randomly initialized model CO, and the remaining configurations fail to achieve the same error as CO. Focusing on C1 (all parameters trainable) and CO, the first trivial indicator of success that C1 outperforms CO and reduces the error up to 80% in the low-data regime. The second interesting observation is the different trend of the error with respect to the number of training parameters. The error of CO starts from very high value in the left side with very few samples, and improves significantly with the increase of the training dataset size. However, the performance of C1 even with very few samples on the left is already comparable to the performance on the right end of the plot. An interesting observation is that C1 trained on 8 samples outperforms CO trained on 1024 samples for v = 0.004. There we can also see more clearly that the advantage of pretraining is amplified over autoregressive rollout. For exact values, refer to Appendix A
The x-axis represents the number of downstream
The second trend that discuss is how the advantage varies for different cocfficient values of the PDE. In the diffusion equation, the larger coefficient means more rapidly changing system. It can be seen in figure 2 that for a larger dataset size, the differences between C1 and CO are relatively small for v 0.001. However, the gap becomes more significant as we increase the diffussin coefficient. For the 5-step rollout for u = 0.004, C1 still achieves a 40-50% error reduction compared to CO. Statistically speaking, the distribution of the system state remains relatively more familiar over time the model changes less rapidly. Therefore, more unfamiliar samples and a different state distribution are presented to the model during validation if the system has a rapid rate of change, further increasing the risk of overfitting and poor performance on the unseen validation data. However, the pretrained models have already seen large amount of data, hence mitigating this risk to some extent.
Focusing on the fine-tuning configurations, C1, C2, and C8 are the models that perform best. The explanation is trivial for C1, since it is totally free to be tuned on the downstream data. However, it is interesting that C8 with only the last layer left trainable outperforms C2 (all Fourier layers trainable) despite having fewer trainable parameters. Each Fourier and feodforward layer have about 524K and 60K parameters, respectively. This means that C8 with less than 600K trainable parameters performs as well or better than C2 with
8

---

Published in Transactions on Machine Learning Research (11/2024)
Diffusion v=0.001
r=1
r=5
1.0
0.8
0.8
% fl. error
0.6
% rt2 error
0.6
0.4
0.4
0.2
0.2
C6
0.0
0.0
8
16
32
64
128
256
512
1024
8
16
32
64
128
256
512
1024
#samples
#samples
Diffusion,
v=0.002
r=1
r=5 5
1.0
1.0
0.8
0.8
% rL2 error
0.6
% rL, error
0.6
0.4
0.4
0.2
0.2
C8
0.0
0.0
32
128
256
1024
#samples
#samples
Diffusion,
v=0.004
r=1
r=5 5
1.0
1.0
0.8
0.8
% rL2 error
0.6
rL, error
0.6
0.2
0.2
C8
8
32 #samples
128
256
512
1024
8
16
32
64
128
256
512
1024
#samples
Figure 2: Average 1Zg loss in percentage for the diffusion equation. CO is the randomly initialized model and the rest are PreLowDed models fine-tuned according to table T The left column shows the next-step error (rollout=1) and the right column shows the average error over the next five autoregressive steps (rollout=5).

---

Published in Transactions on Machine Learning Research (11/2024)
more than 2 million trainable parameters. This can be indicative of hierarchical order of the learning mechanism of the model for the diffusion equation. In the extreme low-data regime (less than 10), tuning configurations with less trainable parameters appear to have the most advantage according to the tables in Appendix A This makes sense since having fewer trainable paramcters reduces the risk of overfitting. However, they fall behind when more training samples are available.
5.3 Advection equation
Similarly to the diffusion dataset, six ad datasets were provided for our experiments. The 1D datasets are provided by PDEBench (Takamoto et al. 2022), and we generated the 2D datasets using the exact solution function of u(z.y,t) un(x St. y 3t).
U = VVn, (0.1,04,110))
(10)
For the advection equations, the strategy does not seem to be as successful as the dilfusion equations. Based on the results provided in figure 3 the difference between the best pretrained models and CO is significant only to the far left of the plot with very few samples. The two fine-tuning configurations that seem to be on par with CO are C1 and C2, which are the ones with the most mumber of trainable parameters. Unlike the case for diffusion equations, C8 performs poorly, falling into the second last model for 3 = 1.0, which can indicate a fundamental difference in how the model extracts features and learns solve the advection equation compared to the diffusion equation. With an increase of 3, the rate of change increases, and the prediction task becomes more difficult. Unlike the diffusion equations where pretraining showed even more benefit over autoregressive rollout, there does not seem to be any significant difference for the advection equations.
6 Conclusion
This work proposes a novel strategy to reduce the cost of data collection, training, and error of a neural operator in multidimensional PDEs. For a neural operator with a factorized architecture like FFNO, we can pretrain a model in lower dimensions (PreLowD) and transfer the weights to a multidimensionall model. We showed that this strategy is successful for the 2D diffusion equation, but not for the advection equation. When successful, the PreLowDed model outperformed the randomly initialized model by up to 80% in low- data regimes and slow-changing system. When the system has faster rate of change or the model is used autoregressively, the gain of the PreLow Ded model is also very significant, achieving an error reduction of 50% over 5-step rollout even with 1024 training samples.
We explored several fine-tuning strategies to find the best way to freeze or tune the transferred weights from a PreLowDed model. It seems that with sulficient training, more trainable parameters will result in a lower error. However, tuning a smaller subset of the parameters may achieve a lower error when very few training samples are available. For the advection equation, the only important factor seemed to be the sheer number of tunable parameters. However, tuning the final full layer for the diffusion datasets was among the best choices, indicating a possibly hierarchical feature extraction of the model. Our general recommendation is to fine-tune the final layer in case of extreme scarcity of training data in the downstream task, or to fine-tune the whole model with all paramcters being trainable.
In future work, this pretraining strategy may be utilized with other neural operators and PDE solvers that have the necessary properties as well. However, it may not be straightforward find or define similar physical system with fewer number of dimensions for pretraining. Moreover, the selective fine-tuning strategy can be applied to any neural operator and can provide insight into the generalizability and interpretability of such models. For example, tuning different parts of the model while varying a certain cocfficient or term for the PDE of the downstream task can help discover the correspondence of the neural operator's components and different modalities and terms of the PDE. Finally, an important potential venue for research is the study of different possible compositions of low-dimensional models and factorization of high-dimensional models.
The ultimate goal of this direction is to improve the prediction accuracy and efficiency in more complex or 3D systems, and this work was just humble starting step towards that end.
10

---

Published in Transactions on Machine Learning Research (11/2024)
Advection, B=0.1
r=1
r=5
5
4
% error
3
% ft, error
3
2
2
C4
C6
C8
0
64 #samples
128
256
1024
8
32
128
256
512
1024
#samples
Advection,
8=0.4
r=1
r=5
10
10
CD
% fL, error
% error
C5
C7
16
32
64
128
256
512
1024
16
32
64
128
256
512
1024
#samples
#samples
Advection
B=1.0
r=5
10
CO
% fL, error
% fL2
C5
C6
C8
16
64 #samples
1024
64 #samples
256
1024
Figure 3: Average rLg loss in percentage for the advection equation. CO is the randomly initialized model and the rest are PreLowDed models fine-tuned according to table 1 The left column shows the next-step (rollout=1) and the right column shows the average error over the next five autoregressive steps (rollout=5).
11

---

Published in Transactions on Machine Learning Research (11/2024)
References
Benedikt Alkin, Andreas Fürst, Simon Schmid, Lukas Gruber, Markus Holzleitner, and Johannes Brand- stetter. Universal physics transformers. arXiv preprint arXiv:2/02.1 2024.
Godas Bertasius, Heng Wang, and Lorenzo Torresani. Is spaco-time attention all you need for video under- standing? In ICML, volume 2, pp. 4, 2021.
Shuhao Cao. Choose transformer: Fourier galerkin. Advances in neural information processing systems, 34:24924 24940, 2021.
Shuvam Chakraborty, Burak Uzkent, Kumar Ayush, Kumar Tanmay, Evan Sheehan, and Stefano Ermon. Efficient conditional pre-training for transfer learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 4241-4250, 2022.
Fei-Long Chen, Du-Zhen Zhang, Ming-Lun Han, Xiu- Yi Chen, Jing Shi, Shuang Xu, and Bo Xu. Vip: A survey on vision-language pre-training. Machine Intelligence Research, 20(1):38-56 56, 2023.
Wuyang Chen, Jialin Song, Pu Ren, Shashank Subramanian, Dmitriy Morozov, and Michael W Mahoney. Data-efficient operator learning via unsupervised pretraining and in-context learning. arXiv preprint 1iXi::242..15944 2024.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 181.004000 2018.
Zhongkai Hao, Chang Su, Songming Liu, Julius Berner, Chengyang Ying, Hang Su, Anima Anandkumar, Jian Song, and Jun Zhu. Dpot: Auto-regressive denoising operator transformer for large-scale pde pre- training. Xin preprint 2rX3:22003.0554, 2024.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 4918-4927, 2019.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16000 16009, 2022.
AmirPouya Hemmasian and Amir Barati Farimani. Reduced-order modeling of fluid flows with transformers. Physics of Fluids, 35)),, 2023.
AmirPouya Hemmasian and Amir Barati Farimani. Multi-scale time-stepping of partial differential equations with transformers. Computer Methods in Applied Mechanics and Engineering, 16993, 2024.
Olivier Hénaff, Skanda Koppula, Cann-Bapistt Alayrac, Aaron Van den Oord, Oriol Vinyals, and Joao Carreira. Efficient visual pretraining with rastive detection. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pp. 10086-10096, 2021.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimensional transformers. arXiv preprint drXim:1912.12188 2019.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. arXiv preprint arXic:1905. 12265, 2019.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised lcarning. Technologies, 9(1):2, 2020.
Dahun Kim, Donghyeon Cho, and In So Kweon. Self-supervised video representation learning with space-time cubic puzzles. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 8545-8552 2019.
Zijie Li, Kazem Meidani, and Amir Barati Farimani. Transformer for partial differential equations' operator learning. Xixio preprint ar ar::::20 13671, 2022.
12

---

Published in Transactions on Machine Learning Research (11/2024)
Zijie Li, Saurabh Patil, Francis Ogoke, Dule Shu, Wilson Zhen, Michael Schneier, John R. Buchanan Jr, and Amir Barati Farimani. Latent neural pde solver: reduced-order modelling framework for partial differential equations. arXiv preprint ar 17833, 2024a.
Zijie Li, Dule Shu, and Amir Barati Farimani. Scalable transformer for pde surrogate modeling. Advances in Neural Information Processing Systems, 36, 2024b.
Zijie Li, Anthony Zhou, Saurabh Patil, and Amir Barati Farimani. Cafa: Global weather forecasting with factorized atlention on sphere. Xic preprint 2rXxv:2405.07395, 2024c.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint ar Xiv:2000 08895, 2020a.
Zongyi Li, Nikola Kovachki, Kamyar Azizadeneshhli,, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXiv preprint Xix:::0003. 03485, 2020b.
Junyang Lin, An Yang, Yichang Zhang, Jie Liu, Jingren Zhou, and Hongxia Yang. Interbert: Vision-and- language interaction for modti-modal pretraining, arXiv preprint wxXi:2003.13198 2020.
Xiao Liu, Fanjin Zhang, Zhenyn Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self-supervised learning: Generative or contrastive. IEEE transactions on knowledge and data engineering, 35(1):857 876, 2021.
Cooper Lorsung and Amir Barati Farimani. Picl: Physics informed contrastive learning for partial differential equations. arXie preprint arXiv:2401.16222 2024.
Michael McCabe, Bruno Régaldo-Saint Blancard, Liam Holden Parker, Ruben Ohana, Miles Cranmer, Al- berto Bietti, Michael Eickenberg, Siavash Golkar, Gerand Krawezik, Francois Lanusse, et al. Multiple physics pretraining for physical surrogate models. ar preprinl ar urx::22310. Xic::330000294, 2023.
Grégoire Mialon, Quentin Garrido, Hannah Lawrence, Danyal Rehman, Yann LeCun, and Bobak Kiani. Self- supervised learning with lie symmetries for partial differential equations. Advances Neural Information Processing Systems, 36:28973 29004, 2023.
Subhadarshi Panda, Anjali Agrawal, Jeewon Ha, and Benjamin Bloch. Shuffled-token detection for refin- ing pre-trained roberta. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pp. 88 08-93, 2021.
Nils Rethmeier and Isabelle Augenstein. A primer on contrastive pretraining in language processing: Meth- ods, lessons learned, and perspectives. ACM Computing Surveys, 55(10):1-17, 2023.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. arXiv preprint ar 10972, 2021.
Junhong Shen, Tanya Marwah, and Ameet Talwalkar. Ups: Towards foundation models for pde solving via
crosl-modal adaptation. arXiv preprint ar arXiv:240.001187 2024.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tic Liu. Mpnet: Masked and permuted pre-training for language understanding. Advances in neural information processing systems, 33:16857 33:16857-16967, 2020.
Shashank Subramanian, Peter Harrington, Kurt Keutzer, Wahid Bhimji, Dmitriy Morozov, Michael W Mahoney, and Amir Gholami. Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior. Advances in Neural Information Processing Systems, 36, 2024.
Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems, 35:1596-14 1611, 2022.
13

---

Published in Transactions on Machine Learning Research (11/2024)
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Vidcomae: Masked autocncoders data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35: 10078 10093, 2022.
Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators. arXiv preprint arXtc:211113300, 2021.
Tapas Tripura and Souvik Chakraborty. A foundational neural operator that continuously learns without forgetting. arXiv preprint arXiv:2310.18885, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Atlention is all you need. Advances in neural informalion processing systems, 30, 2017.
Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao Xiao-Yong Wei, Yaowei Wang, Yonghong Tian, and Wen Gao. Large-scale multi-modal pre-trained models: A comprehensive survey. Machine Intelligence Research, 20(4):447 482, 2023.
Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 4(3):279 287, 2022.
Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 14668-14678, 2022.
Tianyu Wu, Shizhu He, Jingping Liu, Siqi Sun, Kang Liu, Qing-Long Han, and Yang Tang. brief overview of chatgpt: The history, status quo and potential future development. IEEE/CAA Journal of Automatica Sinica, 10(5):1122-1133) 2023.
Liu Yang, Siting Liu, Tingwei Meng, and Stanley J Osher. In-context operator learning with data prompts for differential equation problems. Proceedings of the National Academy of Sciences, 120(39) (20(33):e233))) 22310142120 2023.
Zhanhong Ye, Xiang Huang, Leheng Chen, Hongsheng Liu, Zidong Wang, and Bin Dong. Pde- former: Towards a foundation model for one-dimensional partial differential equations. orXiv preprint arXiv:2402.0 12652, 2024.
Linhui Yu, Yansen Su, Yuansheng Liu, and Xiangxiang Zeng. Review of unsupervised pretraining strategics for molecules representation. Briesings in funclional genomics, 20(5):323 332, 2021.
Rui Zhang, Qi Meng, and Zhi-Ming Ma. Deciphoring and intograting invariants for neural operator learning with various physical mechanisms. National Science Review, 11 (4):3w33336, 2024.
Anthony Zhou and Amir Barali Farimani. Masked autoencoders are pde learners. arXiv preprinl S.XXv:2243.117222 2024.
Anthony Zhou, Cooper Lorsung, AmirPouya Hemmasian, and Amir Barati Farimani. Strategies for pre- training neural operators. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL
https://openreview https///ppppeeeeeeeeeee///mmrrss
14

---

Published in Transactions on Machine Learning Research (11/2024)
A
Appendix
All the results are presented in detail in this section. The reported metrics are the next-step prediction error and the average error over the next five autoregressive rollout steps of the model. CO is the model trained from a random initialization and the rest are pretrained models with different fine-tuning configurations as specified in table !
Table 2: Results for Diffusson v - 0.001. Average rL2 loss in percentage, and the relative change compared to CO (no pretraining) in parenthesis. Negative changes indicate improvement.
CD
C1
C6
C7
#samples
rollout
15.42
6.08(-680.66)
3.79(-76.430)
0.93(-0339%(
1.000-83.1 156)
1-680--99.2588
1.50 9%%)
0.90K 656)
1.84(-88, 1%))
5
30.55
(4.4
13.33(-56.4%)
2.55(-916%%)
3.06/-90.0%%
5.05(-89.3%)
4.73(-84.5%)
2.81(-90.855
5.08(-80.4%)
0.82
3.2611-06 896)
1.377-76.896)
0.621-903799)
0.78(-92.005
1-32(-96-63)
1.05(-803%%
0.621-933 756)
1.67(-83119)
5
17.43
7.42(-57.47% (7%))
9.231-52.8%)
(.8(-90045)
2.231-87 25%)
4.091-70.5%)
3.13(-82.0/%)
1. 1.70(-90.254
4.24(-75. 7%)
1.67
0.311-81.3%
0.5611-06443
0-411-76.6%)
0.69K-58 050)
0.301-64.430
0.40(-72 458)
0.341-77
5
4.08
0.761-81.5%%
1.501-63.3%)
1.00--5.59%)
1.991-52.9%8
1 1.600-60.9%)
1.254-00.550
1 1.19(-70.9%) 70.99%
0.84[-79.5%%
0.71
0.191-72.-27 770)
0.371-47.336
0.331-53.19)
04.15-14355)
0-431399000
0.30 (9.6%)
0.341-51. 755)
0.241160.65
1.91
(42(777.9%)
0.951-49.9%)
0.77(-50.6%%
1 1.677-12.455 455)
1.077-43.773
0.87(-54.11
0.82(-50. 756)
0.556-70.5%)
10
0.44
0.211--22 430)
03300-33.000)
0.351-116889
10000 +-34.38))
0-400-104330
0.39[ 6%6)
0 0.320-27 450)
0.221
1.13
0.47(-57.998 57.9%)
0. 0.741-34.7%)
0.831-26.658 6%
19911 (44 096)
0.971-13.6%)
0,74(-34.22
0.75(-33 596)
0.52( 53.8%)
322
0.27
0. 0.161-90.390
02211-28.000
0.321+17 17.088
0.627
+ 030
00.00 49.3%)
0.30 (.%))
0.34K 636)
0.201 490)
0.64
0.331
0%)
0.47(-27.0%)
0.7311 (9.00%
1.701 160.399
0.38
52.65%
0.07( (..2%)
0.81( (26 136)
0.44(-31008
64
0:20
0.1 1.151-25
0.181-11.820)
+200 188)
0.311 87 436)
0.201 4-45886)
0.31 54 226)
0.19 8.80
0.45
0.301
0.38(-14.7%)
0.74( 63.59%)
1.700 (280.09))
0.89(
100.5%)
0.651 1 44.9%%
0,70X 58 236)
0.43(-3.9%%
128
0.19
0.1 41-25.111300 190)
0.161-13.660
0.38 99 6%)
0:258 0:28(+50.09
0,299( 52 896)
D. 0.
0.41
0.281-12.0%)
0.34[-16.5%)
0,681 66.4%%
1.55, 1 1277 798)
0.900 120.4%)
0.02( 1 52.09%)
0.65( (5.00%)
0.42( 2.8%)
256
0.18
0.1 31131-27.60 690)
0.17(-5.8%)
0.33] 0.33[+588.00 +588
6%)
4206 (15)
0.361 396.3%
0.28((+00115)
0.32( 79 336)
D.79(.44.6%
0.38
0.24(-36.3%)
0.35(-9.8%)
0.70(1 1 100.2%)
1.48| | (280.49)
0.84( 121.9%)
0.62( 1 63.8%)
0,73K (11 (36)
0.42) 1 9.9%)
612
0.22
0.1 3113-000934)
0.171-20.066
0.600 0.600 +176.855)
0.351 -02 4%))
0.201+33133 (091433.130 11.1
0.299(4332 436)
1.1
r-
0.49
0.241-50.0%)
0.37(.24.6%)
0.09[ (2.00%)
1.65; | (338.99)
0.84( 72.7%)
0.63( | (28 69))
0.65( (33
0.41(-15.5%%
1024
0.17
0.1 61131-38775
1.17(-1.1%
0.34 ( 09.6
0.600 0.600 436)
0.37(+1122 4-121 896)
0.201+74.35%
0.32( 836)
8.68
0.35
0.249-30.9%)
0.34(-1.2%)
0.77( | 120.2%)
1.65; (373.49)
0.88( | 151.5%)
0.64[ I 84.5%)
0.72( (107.7%)
0.41() 16.9%)
Table 3: Results for Diffusion v = 0.002. Average rL2 loss in percentage, and the relative change compared to CO (no pretraining) in paranthesis. Negative changes indicate improvement.
CD
C1
C3
C4
C5
C6
C7
C8
#sumplee
rollout
17.29
7.45(-56.9%)
3.21(-81.4%)
0.97(-94.4%%
1.8(-93.8%)
188((-994%%
L.1(-00.7%)
0.87(-95.0%
1.37(-92.08%
5
32,67
16.72(-18.8%6
10.59 67)
2.68(-91.8 8%%
3.09(-90.5%)
5.81(-62.2%)
(883)85.296)
2-12(-92.6% 6%)
3.87(-88. 198)
1-1
10.21
3.66( 196)
2.49(-75.7%)
0.71
1814(-8.4%)
0.896
0.88; 93.6%)
1.17[-90.3%
17,60
7.84(-58.8%)
8.78(-50.4%)
1.91(-89.20 256)
2.26(-87.299
3.61(-79.6%)
2.47
1.79/-80.996)
2.87(-80.88
1=
1.84
(-34.22))
0.37(-70.8%)
0-13:-78.005 0%)
0.78(-50.9%%
0.500-74.0%)
0.47(-76.8%)
0-43 77.9%)
0.29]-86 (88)
1.63
0.75(-833.98 956)
1.50(-67 6%))
0.98
2.12 (.136))1.36))
1.27( 72.59))
1.19-77.396)
(08(-76.8%))
0.71(-81.0 6.6)
0.73
(120((72.89
0.61[-17.095
0.38(-48.890 48.8%%
(.32()55.7%)
0.32
0.22; (997))
1.83
0.44[-77.28%
0.83(-56.9%)
0.68/-64.99 (56)
1.50
0.87(-54.8%)
0.74(-61.9%)
0.73/-62. 156)
(
16
7m
0.62
0.21 (-6000000000
49.9%)
0.37 0.37 336)
0.060 - 26. 7%0
0.391 1%)
0.31(-4033%)
0-20(-62 4%)
5=5
1.29
0. 0.15(-64. 896)
0.61(-52.77 (2))
0.82 30---36.000 4%6)
1.75 35. 7%6
0.86(-32.990 32.9%
0.68/-16.99 16.5%)
0.67
0.441-06.1 196)
32
0.34
(95)))
-11.5%)
0.35(+4.056
0.62
-8 6..6%
0.36(
0.31()7.09%)
0,28 15.3
0.17 29.0999
5-
0.50
0.35(-56.99%
0.42(-48.0%)
0.76(-5.6%)
1.64( 104.9%)
0.82( 12.2%)
0.68(-14.7%)
0.62/-29 005)
0.37(-53 (3.45)
64
0.14(-34.458
(22.6%
0.333 54.2%%
0.660 810
0.34( (%0.%%
0.28 431.1%)
0.207 4:30.49)
0.17(-19
5
0.46
0. 0.26(-18.480 496)
0.33 20- 3%()
0.69 (+89 89))
1.77(+2252.0) 5%)
0. 0,77(+ 0%))
0. 0.58(+26. 3%)
0.63 [+ 7%6)
0.37 19 0%))
0.23
0.14[-37. 756)
0.322 + (00))
0.65( + 180.0%)
0.33 43. 1%)
0.29K 1 (24)
0.288 | (70)
0.17[228 2%%)
0.50
0. 0.28/-44.6%% 696
0.33(-34 3-- 2%%
0.60 +-32 8%)
1.69(+239.799 7%)
0. 0.73( + 9%%)
0.50K + 18 (7%)
0.60 20%
0.36 0%)
256
1=1
0.18
1.11(-21.3 31%
1..96-13.9% )
0.35 +-88. 1%)
0.58( (188+-211.90 99%%%
0.33( +50 9%%)
0.30) +61. 6%))
0.29(+33. 1%%)
0.16| 12 9%%)
5
0.38
0. 0.26/-32.22 2%%)
0.51 20. 1%)
0.73( +91.5%
+.48(+2866.))
0.74( + 91.6%)
0.64[ 44%)
0.64) 67. 2%)
0.351-9.3%%
512
0.19
0.12[ 256)
1.177(13.6%
0.360 188 (8.39%
0.531[ | 204.8%%
0.34[ 1 76.8%)
0.29K 1 (1100))
0.307 1 (66.4%)
0.17 13. 13.7%
0.23/-44.35%
0.33(-19.4%6
0.78( 4-91.3%)
1.64(4-777.0 (6%)
(.766 +66.8))
0.61(4000220
0.68(465.889
0.361-11 49%)
1021
1=1
0.20
0.11(-31, 9%)
0,19(-7 1%)
0.35( (63. 1%)
0.66)(2250.22
0.33(+71.99% 99%)
0,29( 0%)
0.30 33300+18..3) 3%%)
0.16 18 9%)
0.42
0.25(-40.00%)
0.37(-12.7%)
0.68 (686++62455
0. 0.76(+66.77% 1.177+
+-6.5%))
0.35 0.65(+
15

---

Published in Transactions on Machine Learning Research (11/2024)
Table 4: Results for Diffusion v = 0.004.
Average r loss in percentage, and the relative change
compared to CO (no pretraining) in paranthesis, Negative changes indicate improvement.
CD
C1
C2
C3
C4
C5
C6
C7
Cs
#sumples
vollout
Y=1
19.36
8.15(-57.9%)
3.37(-82.6%)
1.56;-422008
1.92((90011%)
1.98(-89.8%)
2.25/-88.4%%
(.11(-94.3%)
1.70(-91. 298)
5=5
34,00
16:93/-51 1%6)
11 11.68(-0622))
4.19 9%)
3.01(-85.4%6)
6.29(-81.8%)
6.53(-81.1%)
2.89 6%)
4.54[[86.00
12.01
5.97(-50.35)
1.82(-81..%)
0.62(-91.%)
0.79(-93.4%%
1.07(-91.1%)
1.35(-87.15)
0.59
1.64[-80.39
5-5
19.50
11.20|-42.60 6%)
5.42(
1.47(-92.5%)
1.97(-99.9%)
5.05(-84.3%)
4.76(-75.6%)
1.39(-92.9%)
4.10
1.69
0.33)-80.7%)
0.200-70.2%)
0.46(-72.%%
0.76(-55.0%)
0.300-70.6%)
0.42(-75:0%)
0.29(-80.19)
5
3.92
0.71(-81.9%)
1.21(-69.1%)
0.93(-76.498)
1.81(-33.7%)
(1.(-71.2%))
1.11--7..8))
0.90(-77.0%)
0.60 84.8%%
0.04
0.22|
08.29,
0.40) 136)
0.62[[2.6%5
-.321-33.77)
0.381 44.5%)
0,23,,63.00
7m0
1.62
0.42(-74. 196)
0.72(-65.4%)
0.76 636)
1.42(-12.119) 12.)
0.891-45.7%
0.70K-06.886
0.71 976)
0.45(-72 (155(72.00 0%)
1=1
0.59
0. 0.25(-57. 358)
-6.29))6.19)
0.11 23.3))
0.67 1777-13,.000 9%)
0.46 2%%)
0.37(-37.11
0.36 3366.-3..55 896)
0
1.37
0,49( (.99-64.694
0.68(-50.6%)
0.85%-41.55%
1.58i 1000--55. 1100--5...00 1963
0.971-29 29.5%)
4.78(-47.16)
0.71 47.9%)
0. 0966669
32
0.34
0.21 (-38.00%)
0.35( 3.0%)
0.641 85. 795)
0.400 1 (17.5%
0:36 6.2%%
0.333 4.55%
0.21
0.366-49.000 6%6
0.42(-41.0%%
0-633-12.058
1.47(+ 104.3%)
0.800 +11-+1110000 .0%)
(.69(-3.4%)
0.62(-14.10)
0.386-46
61
0.31
0.17 636)
0:21(-31 (3)))))))))
(711(+33..0 8%)
0.63) + 106 106.0%)
0. 1.390 (500+22..25 1%)
+-.88)
0.331 +-7 (2%)
0.191-37 69%
0.26(-56.65 496)
0.37( 99.)
0.78(+22) +183++2..00 196)
1.45(+ 361++188.99
0.74( (000+21.550 5%%)
(.57
0.62[++1.000
0.34
128
0.27
0.17( 296
0.19) 5%.5%)
0.37 37.
0.631 1 13133%
0.400 45.8%)
0.333 1 12233%%)
0.333 1 (21.2%
0.191
899)
0.52
0.27(-49. 556)
0.33(-35.9%
0.87[ 0.67[4-30.00) (330.3)))
1.44( 1-801118..220 178. 2%6)
0.77( 4-4.3%))
0.61 18.5%)
0.35
256
0.27
0.17(-366 7981
0. 0.199-228. 196)
0.40 (0044447000 9%)
0.65(+-141 (9%)
0.41, +-5.-52233 4%)
0.341 4%)
0.33, 8%%)
0,19[-30. 99%
5=5
0.50
0.26[-18. 18.00%)
0.32( -3666)))))
0.73( 1.73(++55.8%)
1.50(+1997.2%
(.81(+61.490
0. 0.63(+24.7%)
0.62(+23300g 0%))
0.33(-31.779
0.17(-44.8%)
0.21(-34.2%%
0.38( 4-0)4
0.69
0.36(+11.6%)
0.33) +-3.5)))
0.19[-39
5=
0.61
0,27(-56. (10--50.333
0.35(-12 196)
1.63[+167 .9()
0. 76((+2133%)
0.64[
0.60(-0.886)
0.34[ 0%))
0,19( 31.25)
0.22(-19.4%)
0.69[+ 1555.4499
0.41( (5.88555.555
0.35(
0.32( 19.7%)
0,201
Y-5
0.50
0.29(-41.68% 41.6%)
(33(4224.%)
0.71 40.0%)
1,63( | (22.1%)
0.80( (58.5%)
0.64[ 220.3%)
0.600 18.7%)
0.35(-30. 19))
Table 5: Results for Advection 3 = 0.1. Average r rL2 loss in percentage, and the relative change compared to CO (no pretraining) in paranthesis. Negative changes indicate improvement.
CD
C1
C2
C3
C1
C5
C6
C7
C8
#samples
rollout
1=1
57.53
35.40(3355%)
32. 18(-44.111)
12.92(-77.9%6
7.93(-66.3))
14.71(-7444%%
>100
18:86(-67 (2))
25.87(-55.09)
>100
21.72(-10008) 08))
17.97(-100.9)
37.25/-100.08
2100
33.92(-100.0%)
>-10
2
34.24
59,00(
(-84.1)))
7-59(-78-49)
4.72(-86.2%)
16.02(-56. 196)
>100
13.83(-39.6%)
14.31(-58 290)
>100
2000
100
14.21(-100.0%)
10.16(-100.9%))
47.16(-100.0%
>400
23.62(1100.0%)
1-00
4
1=
3.67
2. 2.39
9%)
2.67(-30158)
2.500-1.880
2888-21.75)
2.55(-80.7%)
2.74(226445 450)
2.43/333 955)
2.301-37.4%%
7.84
4.37
1.08(-35.1%)
4.966-44.4%)
5.92(22.558)
4.49(-42.7%)
5.45(-90.59)
4.15/-47.155
4.06 -48 2%0
8
2.77
2.51 16)
2.32 16.2%)
237(-14.%%)
2.71(2.4%5
2.11(-19:11)
2.55 276)
2.161 22.0%
5
5.38
4.51
2211(22.000 6%)
1.46/-19.899
3.97( 26 6%))
5.15(-2.19)
2119(21.7%)
4 18. 7%6)
3.97 28 0%))
3.79( 31.99)
16
1
1.95
1.90 2.7%
L.3((1.3%)
2.32( 10.0%%
2.73 2.73(+39
2.37 21.2%%
213
2.18( (1.)
2.79)
5
3.31
2.92 11 8%)
3.19(-3.5%%
3.67(-10.8%) 10.8%)
5.12 (66.7)))
3.80 -1.79)
3.78(+13.66
3.51[ (331(+6.00)
3.000
322
1.75
1.71
(12 356)
1.76(+0335)
2:207 22. 28.7%)
2.39 4-47 856)
2.39 32.5%)
Y.66 12 256)
2.230 4226.50) 756)
1.840=5.19
5
2661
2,39( 1ook--8.00 390)
2.60[-0.2%
3.17 33.0%)
5.00 (8%)
3.00 10.2%)
3.21(+21.29)
3.44 1311 99%)
2.79
64
1.67
1,70/+1.94 1996
1.731+3.0%)
2:281 (P%)
2.67 (500
2.33 10.1%)
2:200
4:32 036)
2:251 4-3 (956
8.88
5
2.27
2,20 1 0.65%)
2.45%)
3.46 52.0%)
5 128.395)
3.64( 60.0%)
3.47( 1 152.5555
3.44( 151.11111 156)
2 2.65( 16.6%)
128
L65
164400035
LoBI (2.7%)
2,25 36.3%)
2.08( (68[(1022) (16335)
2.56 43.2%)
2.15(+30.00 +30.7%
2.25 ( 1366676 676)
1.81 11.8%)
7
2.19
2.134-2.958 956)
2.32 15.8%)
3.39( 54.9%)
5,22(1 1377
3.71( 69.2%)
3.38( 1 54 196)
3.41( 65 65.66)
2.79( 24.6%)
256
L65
10.36)
2.200 38.7%)
2.70 ( (33.2%)
2.5M 42.2%)
2.18. 132 176)
2:22
196)
1.800 8.99)
2.17
2.17( 0
2.25|+3.5%% +3.5%)
3.47( 59.59)
6,26(+141.75)
3.09(-69.00 669.6%)
3.41
58) 796)
5.34| +53 +53.7%) 7%)
2 2.65(+22 (2%)
312
2.27 12.0%)
2.67
(+06 (%5)
2.310 11.6%]
2.11(432
2:33 +39
1.70 11.9%)
5
2.03
2.15(+5.5%)
2.301 12.9%)
3.20 67.19%
5,20
0%%)
3.59(-70.5%%
0%) 3.28
3.38| +66.0%))
2 2.65(+29.199
1024
1.62
1.65(+1.75))
1.88 (3.33)
2.20 11.4%)
2.63 8553
2.31
2,214436
2.77 10.15)
1.78(=9.7%5
5
2.00
2.1(422.06)
2.26 (8.25)
3.45( 64.9%)
6.14(+145.8)
3.57( 70.7%)
3.42( 463 636)
2 2.50(+23.700 29.7
16

---

Published in Transactions on Machine Learning Research (11/2024)
Table 6: Results for Advection 3 = 0.4. Average r rL loss in percentage, and the relative change compared to CO (no pretraining) in paranthesis. Negative changes indicate improvement.
CO
C1
C2
C5
C7
CB
#samples
vollout
33.36
34 54(+..3))
20.75(-10.8%)
13,201-604430)
26.86(-19.3%)
18.37(-44.9%)
78.35(+13689))
18-30(-44.9%)
26 (6.81(-19)6%)
>100
2100
>-10
>000
>1100
41.04(-100.0))
2100
46.80(-100.992)
>100
2
20.71
27.96
27.06C9089)
9.05(-67.83)
20.77-300199 196
13:25(-35449)
S100
12.99-56639)
20 20:3900)2))
2-00
2-00
5-00
1000
2100
23.40(-100.9%)
2100
48 48-16(-100.09))
>100
4
[=1
5.27
0.37 1377117.000 17.0%)
4.831 2X)
(32-14119)
1.300-10.9)
8.91
11 (1.34(-0-3%)
8.901-21 .053
11-70(+2999)
8.966-11 395)
9.92(-12.96)
P=1
3.01
3.16; 11.0%)
3.65 0.890
3.00 0.2%)
12.71(225.10) 198)
3.3)
3.81(2.39)
O. 6.00 (50)
0.07
6.488228.65)
10 88/ (9%)
8.681-1.99
41 73 360 096)
7.85013.690
8.591-6.195
7.44--88.0%
16 781 84.9503
2.72, +3000
5.7(+30035))
+321+00.88
5
4.55
4 (.665)
4.95/ +8 796)
6.58( +458(+44500 596
38, 7 098)
6.51 +-38 656
6.44+414499
11 11.911+1011 1616 656)
32
2.11
0.99%)
2.21 +6.3%)
3.15( 39.3%
11.14 122 199)
+36.090
3.90 +87,99))
3.42
3.100 6.75%
3 3,60% 696)
5-51(+66 19%)
+7.011+1015,88
6000000 090)
5.50 996)
5.57
5.57 050
9 5,56++ 15%)
64
1.4
L.500
2007)
2 +5.5%)
3.00 H01309)
11. + 191009)
4 111
2.70 022000
3.981
2.73
2,58/ 5.39%
1.98++9 250)
5.13(+87
18.716+12200 1245+1225000 0%6
6.60
5.101+86,
5.79
112.1111.8
+91--202.25
128
1.81
1332( 4.85%)
1.59,45.7%
3.131
+00 00:5599 579)
11. +520079)
4.23
2.50 +61993)
3:24
3.62( +99.0055
2.43
2.54
2.66/+9.22 250)
5,90(+117
36,00K 1379. 255)
6.39/+102
5,13 110.7%
5,281 110.+108.88)
+
256
1,88
1.505 F2.655
1 1.86)
37)
2.90(166.00
3.84 104.4%)
5
2,51
+0.56)
2.55; +1
(877++125.00
36. rar 1230+1361.18
6. 6.49/+158 298
5,421+115.85
+941+130,998
9.50 277.8%
612
1.76
1.82( 14.3%
1.99( 10.176)
3.23( (4.579)
11.57
315)
2.33( (66.3%)
3.15 80.2%)
119.8%)
pw5
2,19
2.31(+55.88)
2.581 15.536)
6,42(+114700 (09)
95.941+ 999)
6.621 +0000+202.000 005
+.980+127.2%
5,27(+110..p) 495)
9.57( +336,4%
1024
1.80
1.84(1 ( 22,255
1,87( (4. (4.2%)
3.04( 00 226)
11.42| I (36.67%)
4.311 140.075)
2.72| (51225)
3,47(1 (1332225)
3.67( (14.255)
Dw0
2,31
2.52(++
2.42(+44.85)
480(+1100225
35,971 (457.0%%
6.60( (5600+188) 1185.0%)
(11.5%)
5.64 (.64(+106.258)
8.85[+253.255
Table 7: Results for Advection 3 = 1.0. Average LL2 loss in percentage, and the relative change compared to CO (no pretraining) in paranthesis. Negative changes indicate improvement.
Du
C1
C2
C3
C4
C5
C7
Cs
#sampks
rollout
r=1
33.41
34. 34.00(+3.2%)
24,32(-27 2%)
25.801 22.6%)
91.66(171..1%)
(6.89(-19.395)
27:02/-19. (9.1-1911%)
29.06(-19.3%) 19.3%%
65. (5..0()85.0%
7-5
2100
>-00
5-00
>-00
>100
86.98(-66.3%)
>100
2-100
-00
24.63
34.99
1-2..1%)
14.79((40.0%)
16.16i-84.4%)
78.68(-219.7)
16.83(-31.7)
38.39(
16.801-31 6%)
41.10X 66688%
5
>100
5100
5-00
88.25(-100005)
>100
66. 06.00(-100.08 0%))
>100
31.88(-100.08
>100
4
1=1
4.76
4.04(-16. 1%6)
4.381-8 195)
6 6.80(-44.9)
57 57.3% 1103.755)
6.646 4554
4.03( (.033(+3.500
16.99))
22.61( 8%%)
10.78
9.011-16
4%)
19/-671 896)
14.37(+33.355
15.12(+21.05 (%%)
12.74(+16
2%6)
7L. +562
8
2=01
3.49
3.255-7.4%
4.5K 42.8%1
4V.70: 1265.99)
4.080 4.08 134.1%)
4.06) +15.495
4.56(-30.110
-30.1%
14.17 -006, 8%))
5
7,80
6.99%-11.9%)
7.74(-088)
10 24 146.09)
100
11,24( 44.116)
10.10( + 30.35)
9.91 (2 27.083
46.800 - 150 0%5)
16
3.99)
2.00
3.85
6%)
100000
1315.2%)
3.501 +33.895
11.00
6.130-66-88)
5
4.79
4 4.536-6.8%)
5.469-14.9)
13.85
190 195)
> 100
10.57 + 120 696)
8.52( + 698
8.680-5 -81. -8.44%
31.74/
$1.74/-660
-602 6%%)
32
2.10
2.10 0.13
+3.0
3 252 61.8%
36.020
161309%
1.30 708.7%)
3.15: 199)
5350 319.399
5.7%)
400(+11.00 9%%
9.23
118.00000 (9)
10,000+ 477..4%)
7,23 +.200-999.70 1166
30009 47.9%%
04
1.30
2007 1 4.25)
3. 3.32 07%0
4.37 120
5:300
1 1000996
71000 1 2822 1%5
3.44/
11%
9.37( 202
100
10.01
220.15
1.50 142.15)
25. 7299272 2%5)
128
1.89
1.980 5.255
0.00), 7.505
3.07( 6%0
32.71( 16366995)
4.34 190.955)
3.19 69.438
8.766-99..00 999449)
7.85( 7.005
2.81
49.
3.20 196
7.82
796)
> 100
2.96,.255.199
7.27 05.0
7.44 196
27 (7.611/-83,. 9%%
256
1.90 4.0%%
1.92(+3.49%
3.00 4%
30. 18110.09)))
1111++21.05
3.10(
1.81/+106.20
5
2.69
2.85%+
2 7..29)
6.657-147.888
100
2.1.52220.555
6.906+1588 8%)
30 10.00/11000, 4%))
512
1.85
1.50 1 4365
1.M0 4 4.3%)
3.000 66 63.%%
86.44) 1814226)
4.14. 11 128655)
ALIBI ( (6685)
A.62) 160.6%)
7.722 317. 175)
2.54( 99.55)
6.98 140
100
2.311+244.0055
6.86 455.
7.01C+ 161.44)
902,
1021
1=]
1.85
1.88,+6.00
3. 3.100 2%
36, 1833889)
0.300 (3.7%)
3.30 (5300+55.11
3.651 1.65).855.00 8%6
V.7 7711 0%%
zw5
2,07
2,756+3.0%6
3.02(+13.22
7.37(
1810%)
100
9.756 2665.666
7.88( + 190.0%)
7.11(+ 100.3%)
28. +170,
17


Reviews:
[{"review_id": "jYmZYaxxSZ", "reviewer": "Reviewer_ksvx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The approach of pretraining then fine-tuning has led to the huge success of LLMs. It is interesting to explore how to adapt this strategy to solving PDEs. This paper particularly studies one of the key bottlenecks: the lack of data.\n* The PreLowD approach demonstrates significant error reduction compared to training from scratch.\n\nWeaknesses:\n\n* This paper's writing quality could be significantly improved. Some sections are hard to follow.\n  1. Introduction: Much of the content of the introduction could be moved to related works. The main contribution of the paper is mentioned in the last paragraph, but it is better to also describe the approach with slightly more details and highlight some key experiment results.\n  2. Methodology: This is the most confusing part. The proposed approach should have been described in detail here, but it is very hard to see what the PreLowD approach really is through its current presentation. It is better to describe the approach step by step, and mention what type of models, data and training procedures should be used in each step. In fact, even after reading the entire paper, I still do not know what the pretraining PDEs are.\n* Experiments are not thorough enough.\n  1. The pretrained neural operators were only tested on two 2D PDEs. It would be better to test them on more PDEs.\n  2. This paper proposes 8 fine-tuning approaches in a row. It would be better to design some metrics (such as the average loss over downstream tasks) to compare them thoroughly and make a recommendation for one particular approach.\n  3. The pretraining dataset seems to be constructed a bit arbitrarily. Can the authors provide some ablation studies on what data should be included in pretraining?\n* The paper mentioned that 1D PDE data is easier to generate. Why is this the case? It would be better to write a paragraph to compare the costs of generating 1D and 2D PDE data.\n\nTypos:\n* Section 1.2: Subramanian et al. Subramanian et al. (2024)\n* Section 2.2: Li et al. Li et al. (2020b)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Pretraining via proxy tasks has been explored for neural PDE solvers, but most of the existing works relied on a large pretraining dataset with a high collection cost. In this paper, the authors presents the approach of pretraining neural PDE solvers on lower dimensional PDEs (PreLowD), and argues that 1D PDE data is significantly lower than high-dimensional PDE data. In the experiments, the authors pretrain fine-tuning factorized Fourier neural operators on 10K 1D PDE samples, and then fine-tune them on two 2D PDEs (diffusion and advection) to show that they outperform neural operators trained from scratch.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weaknesses above. It would be good if the authors could (1) improve the writing quality and (2) include more experiments to justify their approach.", "broader_impact_concerns": "There are no significant ethical concerns related to the broader impact of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "B6FmLhkm4r", "reviewer": "Reviewer_X8Wp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The method is straightforward, and the paper is well-written in terms of discussing existing work on operators and finetuning operators for downstream tasks.\n2. The empirical results show that the proposed method can be useful for some PDEs, like the diffusion equation. As would be expected the method works well than simple pre-training in the low data regime and converges to that of the performance of directly pre-training on the downstream tasks for more sample budget.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary:  The authors propose a pretraining approach for neural operator-based models to increase the accuracy and training efficiency of a neural operator for high-dimensional PDEs. The strategy is simple, i.e., to use a pre-trained model in lower dimensions and fine-tune it in higher dimensions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Weaknesses / Requested Changes\nI do not have any major criticisms of the paper since the goal seems pretty straightforward. However, I have some minor comments:\n\n1.The pretraining strategy proposed seems to be compatible only with FFNO-like operators and, hence, does not seem like a generic approach for solver fine-tuning. It would be great if the authors could elaborate more on this limitation of their approach.\n\n2. How were the combinations in Table 1 chosen, and what is the motivation behind such a selection? I would request the authors to clarify the experimental setup with motivation here.\n\n3. Throughout the paper, citations are in line with the text without a delimiter like () which makes reading at times very tedious. I would request the authors to fix this problem.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jedQCTbKfF", "reviewer": "Reviewer_YiZw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths__\nThere is much value in establishing the best practices on pretraining-vs-finetuning for neural PDE solvers. This work undertakes preliminary peek at this core issue. \n\n__Weakness__ \nThe best recipe for balancing pretraining-vs-finetuning depends on the cost of data acquisition, training compute resources, model scaling, and sample complexity of downstream tasks. There is insufficient rigorous discussion on each of these facets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper evaluates the effectiveness of pretraining for learning neural PDE solvers. Due to the cost of acquiring quality simulations of high dimensional PDEs, the authors suggest pretraining neural PDE solvers with 1D PDEs and fine-tuning them on 2D downstream tasks. Amongst the selected list of fine-tuning configurations, fine-tuning all parameters (C1) appear to work best.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The authors mention the need for exponential samples for fine-discretizations of higher dimensional PDEs but they are no hard numbers. For eg, the downstream relative error of the best-fine-tuned model is about 20%. How should one perceive this value? Does the potentially costly 2D data acquisition drive the relative error of the full-trained models to 0%? If so for how much sample data? What is the trade-off between pretaining-vs-finetuning?\n\nIf one cares about achieving relative error at most $\\epsilon$, is this achievable with only pertaining? or is data acquisition necessary? Even if proving sample complexity theorems on the data-ratios needed for pertaining is non-trivial, There should be a more comprehensive evaluation of observed empirical sample complexity since the data distributions here are synthetic mixtures derived from well founded PDE equations. \n\n\n2. There are some text style issues with citation (check for repeated or mishaped citations).", "broader_impact_concerns": "I have no concerns on the ethical implications at this stage.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers unanimously voted "Leaning Accept". The core issues presented by reviewers were:
  * Clarity of writing and readability - this has been resolved.
  * Limited novelty - currently the method is only applicable to FFNO / a specific set of neural operators. 
  * Limited experimental ablations - this has not been fully resolved, i.e. the paper performs experiments only on two specific types of PDEs and there are many questions raised around model scaling / sample complexity, etc.
Decision: Accept (poster)