Strategies for Pretraining Neural Operators
Authors: Anthony Zhou
Cooper Lorsung
AmirPouya Hemmasian
Amir Barati Farimani
Venue: Accepted by TMLR ()

Abstract: Pretraining for partial differential equation (PDE) modeling has recently shown promise in scaling neural operators across datasets to improve generalizability and performance. Despite these advances, our understanding of how pretraining affects neural operators is still limited; studies generally propose tailored architectures and datasets that make it challenging to compare or examine different pretraining frameworks. To address this, we compare various pretraining methods without optimizing architecture choices to characterize pretraining dynamics on different models and datasets as well as to understand its scaling and generalization behavior. We find that pretraining is highly dependent on model and dataset choices, but in general transfer learning or physics-based pretraining strategies work best. In addition, pretraining performance can be further improved by using data augmentations. Lastly, pretraining can be additionally beneficial when fine-tuning in scarce data regimes or when generalizing to downstream data similar to the pretraining distribution. Through providing insights into pretraining neural operators for physics prediction, we hope to motivate future work in developing and evaluating pretraining methods for PDEs.

Published in Transactions on Machine Learning Research (09/2024)
Strategies for Pretraining Neural Operators
Anthony Zhou'
ayz29andrew crna. eduu
Department Mechanical Engineering
Carnegie Mellon University
Cooper Lorsung'
clorsung@@andreu.cmu.com
Depariment of Mechanical Engineering Carnegie Mellon University
AmirPouya Hemmasian
abemmasi@@endrew.com.
Department of Mechanical Engineering Carnegie Mellon University
Amir Barati Farimani"
baruti@erma edu
Department of Mechanical Engineering
Department of Machine Learning Carnegie Mellon University
Reviewed on OpenReview: https: // openreview. net/ forum? id- 9vEVeX9oIv
Abstract
Pretraining for partial differential equation (PDE) modeling has recently shown promise in scaling neural operators across datasets improve generalizability and performance. De- spite these advances, our understanding of how pretraining affects neural operators is still limited; studies generally propose tailored architectures and datasets that make it challeng- ing to compare examine different pretraining frameworks. To address this, compare varions pretraining methods without optimizing architecture choices to characterize pre- training dynamics on different models and datasets as well as to understand its scaling and generalization behavior. We find that pretraining is highly dependent on model and dataset choices, but general transfer learning or physics-based pretraining strategies work best. In addition, pretraining performance can be further improved by using data augmentations. Lastly, pretraining be additionally beneficial when finc-tuning in scarce data regimes or when generalizing to downstream data similar to the pretraining distribution. Through pro- viding insights into pretraining neural operators for physics prediction, we hope to motivate future work in developing and evaluating pretraining methods for PDEs.
1 Introduction
Pretraining is an immensely popular technique in deep learning in which models learn meaningful context from a large dataset and apply this knowledge to downstream tasks (Devlin et al.) 2019 Chen et al. 2023
Schiappa ot al. 2022). In particular, recent work has highlighted the importance of self-supervised learning.
which can leverage the inherent structure of unlabeled data and learn meaningful latent representalin
[Bardes et al. 2022 Chen al.)
2020 Leyva-Vallina Vallina al.) 2023 He al. (021). The success of these self-
supervisod pretraining strategics has motivated their application broad scientific and engincering problems
WWan et al.
2022a Cao et al.
2023 Zhou & Farimani] 2024a Meidani et al.) 2024 Li et. al. 2024 Nguyen
et al. 2023). In particular, pretraining has been used in partial differential equation (PDE) modeling to
improve neural operators and evaluate their scalability and generalizability (McCabe et al. 2023
Hao et al.
2024h.
Equal Contribution. 'Corresponding Author.

---

Published in Transactions on Machine Learning Research (09/2024)
Neural operators for PDEs have gained substantial interest in recent years due to their ability to quickly predict physics through inference
[Li et al. 2021 Lu et al. 2021a Brandstetter et al. 2023). Despite
potential speed gains, neural operators currently struggle to generalize to unseen physies, and initial training can be slow (Lu et al.
2022 Gupta & Brandstetter 2022). To address this issue, many works have explored
different strategies to im impooe generalization by incorporating additional system information (Lorsung et al.
2024 Liu et al. 2023b
Takamoto et al. 2023) and pretraining neural operators across large, diverse physics
to quickly line-ture to solve PDEs (HH et al. 2024 McCabe al.) 2023: Shen et al. 2024 Hang et al. 2024
Goswami et al. 2022). Despite showing good performance, these works usually require the use of tailored
neural operation and datasets to learn different physics. This contrasts with broader deep learning trends in which pretraining methods can universally benefit models; for example, pretraining losses that are applied
across CNN models (Noroozi & Favaro 2016 Lee et. al. 2017) or GNN models (Hu et al., 2020). As a result, in this work, we consider existing pretraining frameworks, as well as propose novel methods for pretraining PDE models that are ficxible and can be applicd across architectures or datasots.
By considering pretraining methods that are model agnostic, we can provide a detailed and level comparison of pretraining methods on shared experimental setup. To our knowledge, this is the first work that makes an effort to compare pretraining strategics without tailored architecture choices, which allows an understanding of how pretraining affects learning in different regimes. Specifically, we compare different pretraining strategies and consider the effect of PDE data augmentations, a popular technique to improve pretrained model performance (He et al.) 2021 Xie et al. 2022 Zhou & Farimani 2024b Brandstetter et al. 2022). Additionally, we study the performance of pretrained models with scarce fine-tuning data as well as their generalization behavior to unseen coefficients PDFs.
Through this work, we hope to broaden the understanding of how neural operators can be pretrained for physics prediction. We organize existing pretraining strategies, propose novel vision-inspired strategies, and include common pretraining baselines to assemble broad set of methods for learning PDE representations. We focus on vision-inspired strategies because the point-wise represontation of system values on a grid for a PDE is analogous to pixel-based representations of images, and the temporal structure of PDE system evolution analogous to videos. We find that PDE pretraining varies depending on model and dataset choice, but in general using transfer or physics-based protraining strategics work well. In addition, transformer CNN-based architectures tend to benefit more from pretraining than vanilla neural operators. Furthermore, the use of data augmentations consistently improves pretraining performance in different models, datasets, and pretraining strategies. Lastly, we find that pretraining is more beneficial when fine-tuning in low-data regimes, or when downstream data is more similar to pretraining data. We hope that these insights can be used to guide future work in the development and evaluation of pretraining methods for PDEs. We make code available here: https://github com/
and use 2D PDE datasets from Zhou & Farimani (2024b) which can be found here: https:://eenooo.rr//eeee www.o//eeeoeo//33eee//03.
2
Related Works
The field of neural operators has grown rapidly in recent years, with many architectures developed
accurately approximate solutions to PDEs (Li et al. 2021 2023a Gupta & Brandstetter] 2022 Brandstetter
et al. quickly with less compute, or on irregular grids (Li et al. 2023b Hemmasian & Barati Parimani 2023 Li
2023 Lu et al. 2021a). Many works expanded on this to propose architectures to solve PDEs more
et al. 2023c). and as a result, within a range of test problems, neural operators can solve PDEs quickly and accurately. However, neural operators still struggle to generalize across diverse physics, and as a result many approaches have been developed to pretrain neural operators. We summarize these past works in Table [ and briefly describe the main approaches here. Furthermore, we review additional work from the broader field of pretraining and transfer learning due to its relevance PDE problems.
2.1 PDE Transfer Learning
Many past works consider transferring knowledge between PDE parameters and domains as a form of pre- training. These works often design specific architectures that are tailored for transferring weights or layers between tasks. For example, Goswami et al. (2022) design task-specific layers of DeepONet to be used
2

---

Published in Transactions on Machine Learning Research (09/2024)
Category
PDEs
Characteristic
Reference
Darey, Elasticity
Fine-tuning task layers trausfer between domains
Goswami et al 2022
Poisson, INS
Direct transfer across PDE domains
Chakraborty
aL 2022
Transfer
Poisson, INS, Wave,
Desiging transferrable model by modifying neurons
Zhang ot Al
Heat, Adv, Nag, Burg, NS, AC
Combining operator modules with gating
mipurrs Chakraborty
2023
NS, Elasticity
Pos. cucoding and masked pretraining across PDEs
Tahman ot m. 2024
Poisson, Helm
Scaling model and datasets to characterize transfer Embedding PDEs common space,
Subramanian et ml
2023
CNS. SWE, DiffReact
Axial VIT
el
INS, CNS, SWE, DiffReact
Denoising & Fourier attention with large models/dan
Hao ot al. 2024
Large Modela
Adv, Burg, Diff-Sorp, SWE, NS
Aligning LLM guidance across diverse PDEs Conditional transformer
Shen et al 602
INS, CNS, SWE, DiffReact
large PDE datasets
Haug
2021
Poisso, Helm, NS, Wave, AC
Scaling operator transformers to large datasets
Herde cf m.
2024
Kdv, Burg, KS, INS Heat, Advection, Burg Burg, Adv-Diff, NS
Lie Symmetries for contrastive learuing Physics-informer distance contrastive framework Physical invariances to contrastively learn an encodor
Mialon et al.
2023
Contrastive
Lorsing
Paritiani
2024
Zhamng ot al.
20233
HGO. Elasticity, Tissue
Madel-agnostic meta- learning loss Novel loss term to maximise learming hetween PDDs
tasks
Zhang al.
2023a
Meta-Tcarning IN, GS, NS
Yin nl. 2021
IV, GS, GO, NS
Hyper-network to adapt operators for specific tasks
Kirchmeyet
2022
In-Context
Poisson, Helm, DiffReact, NS
Masked protraining and in-rontext learning for
Chen al.
2024
Poisson, DiffReact
In-context learning by prompting transformer
Yang al
2023
Table 1: A review of past works on pretraining neural operators for PDEs. We organize works by approximate categories and describe their data and methods.
with different domains of 2D Darcy Flow and Elasticity problems. Another approach proposed by "Tripura & Chakraborty (2023) is to design different operators that learn specific PDE dynamics and combine these in mixture of experts approach, motivated by the observation that PDEs can often be compositions of cach other. To address the issue of transferring between physical domains that can have different numbers of vari-
ables, Rahman et al. 2024 extend positional encodings and self-attention to different codomains/ codomains/channels /channels.
2.2 Large PDE Modeling
An extension of transfer learning is to train large models on diverse physics datasets, with the intention of learning transferable representations through scaling behavior (Wei et al. 2022 Kaplan et al.) 2020 Brown
et al. 2020). 6T initially explores this scaling behavior by training large neural operator models on large PDE datasets to evaluate its ability to adapt to different coefficients. |MCabbe et al. (2023) propose a tailored architecture for solving problems across different physics, and Hao et al.] (2024) expand on this by making architectural advancements and training on more diverse physics. Despite different approaches and datasets, these works generally rely tailored, scalable architectures for large PDE datasets; pretraining is framed physics prodiction across diverse physics and fine-tuning is done on the pretraining distribution or on unseen coefficients/PPe (PDieennttt/.......... PDES.
2.3 PDE Contrastive Learning
Following the success of contrastive learning in the vision domain (Chen al. 2020 Bardes et al.
2022
Zbontar et al. 2021), various methods for PDE contrastive learning have been proposed. Mialon et al. 2023
propose contrastive learning framework in which augmented PDE samples are represented in a similar way in latent space: notably augmentations are done with physics-preserving Lie augmentations (Brandstetter et al.) 2022). Zhang et al. (2023b) follow a similar approach in which physically invariant samples are clustered together in latent space, while Lorsung & Farimani (2024) rely on PDE coefficients to define a contrastive loss. In general, contrastive methods have extensive literature and theory, however they tend to be challenging to pretrain and may have incremental gains in the PDE domain.
2.4 Meta/In-context Learning for PDEs
Additional past work considers adapting meta-lcarning Finn et al.) 2017) paradigms from the broader deep learning community to the PDE domain. Zhang et al. (2023a) consider a direct adaptation of model-agnostic meta-learning to PDE tasks, while Yin et al. (2021 and Kirchmeyer et al.) (202)) apply novel losses and
3

---

Published in Transactions on Machine Learning Research (09/2024)
architectures maximize shared learning across different tasks. Following in-context learning trends of transformer models (Dong et al.] 2023), Chen et al. (2024) and Yang et al. (2023) explore using in-context learning to prompt models with PDE solutions to generalize to unseen PDE coefficients.
2.5 Pretraining and Transfer Learning
While PDE problems contain unique structure and challenges, works from the broader field of pretraining and transfer learning can still be leveraged to inform hypotheses and inspire novel work. In particular, many previous works seek to understand how models can learn representations that transfer across tasks or domains, as well as design architectures and pretraining tasks to optimize this (Jiang et al. 2022 Liu et al. 2023a; [Zhuang et al. p020p. Under this framework, most previous work in the PDE domain is considered supervised pretraining, with the notable exceplion of contrastive methods. Additionally, novel works tend to focus improving PDE-specific architectures, rather than designing specific objectives for task or domain transfer. We hypothesize that this is a reflection of the need for numerical accuracy in physics problems; prior work generally focuses on empirically improving prediction accuracy on different benchmark PDEs rather than leveraging theoretical justifications. This could highlight potential future opportunities to examine more complex or principled pretraining learning methods such as adversarial learning or causal learning.
3 Methods
3.1
Data Augmentations
Following the prevalence of data augmentation in the broader deep learning community (Chen et al. 2020 Perez & Wang 2017), consider the use of data augmentations adapted to the PDE domain.
3.1.1 Lie Point Symmetry Data Augmentations
We consider recent body of work proposing Lie Point Symmotry Data Augmentations (Brandstetter al. 2022 Mialon et al. 2023), a set of PDE-specific data angmentations that preserve the underlying dynamics. Mathematically, given PDE,
derive a set of transformations (g1- 9............. each with a parameter 6n,, that can be randomly sampled to modulate the strength of the transformation. Since some PDEs may exhibit more Lie symmetries than others, we consider only shifting the PDE solution in space (Shift), which is valid for all PDEs considered, to ensure a fair comparison between datasets. For further details on mathematical theory and its implementation in augmenting PDEs, we refer the reader to Mialon et al. 2202)) and Olver (1986).
3.1.2 Physics-Agnostic Data Augmentations
In computer vision literature, many successful data augmentations heavily modify inputs (Chen et al. 2020); in particular, cropping and cutting out portions an image would not respect physics adapted to the PDE domain. Following this, we investigale the effect of data augmentations that are physics-agnostic, that they can be applied to any PDE since the augmentation does not preserve the underlying dynamics. consider adding
Following recent work denoising neural operator architectures (Hao et al.) 2024), Gaussian noise during pretraining (Noise). Furthermore, we consider scaling the PDE solution (Scale), in which the PDE solution values are scaled by a random constant, an approach similar to color distortion, whereby the hue, contrast, saturation, or lightness of an image is scaled. Although these visual characteristics do not have direct counterparts in the PDE domain, their scaling can interpreted as scaling the intensity of PDE solution. For certain simple PDEs, scaling can preserve physics, but this is not generally true due to nonlincaritics in more complex PDEs, such as the Burgers equation or details on hyperparameters and the implementation of data augmentations can be found in Appendix D.4
equations. Additional
3.2 Pretraining Strategies
In this work, we consider using pretraining strategies that are agnostic to the neural operator architecture to ensure compatibility with different applications and future architecture advances, and describe them in
4

---

Published in Transactions on Machine Learning Research (09/2024)
Computer Vision Prstraining
PDE Preductive Pestruining
Sort
Cocff
Shuflle
Model
Mask?
Model
Derieartin d,u, d,t,
Classiff Shuffled? Sorted?
Masked wix.y,)
Figure 1: An illustration of pretraining strategics adapted from computer vision (CV) and predicting PDE characteristics. Left: CV methods can be described by different shuffling mechanisms and losses. Binory pretraining only classifies if sequence shuffled or not, while TimeSort, SpaceSort and Jigsaw sort sequences shuffled in various ways, cither along the spatial, temporal, combined dimensions. Right: PDE data has inherent structure that can be leveraged to predict underlying characteristics. Coefficentss of the PDE can be regressed, well as its spatial and temporal derivatives. Additionally, inputs can be masked to regress the solution field and learn underlying dynamics.
Figure [ This approach is also consistent with the broader computer vision domain, where models are fully shared between pretraining and downstream tasks and can be adapted different architectures (e.g. CNN, ViT) (Chen al. 2020 Xie et al., 2022 He al. 2021 We provide further details on design considerations and the implementation of pretraining strategies in Appendix D.3
3.2.1 Computer Vision Strategies
Inspired by diverse pretraining strategies to learn image representations, we adapt many pretraining strategies from the computer vision (CV) domain to the PDE domain. In general, these strategies aim to train models through predicting visual attributes sorting spatio-temporal sequences to learn visual representations without labels.
Firstly, we consider an early work that pretrains a model to verify if a video is in the correct temporal order (Misra et al. 2016). This problem is formulated as a binary classification task which a shuffled video and the original video are assigned separate labels; within this work, we refer to this as Binary pretraining, Subsoquent work proposed methods that not only verify temporal order, but can also sort temporally shuffled video frames (Lee et al., 2017). This is generally formulated as n-way classification task, where denotes the number of permutations in which sequence of frames be sorted. In the context of physics data, can opt to shuffle the data spatially temporally, such we refer to these two pretraining strategies as TimeSort or SpaceSort. Empirically, SpaceSort does not perform well, so we omit this strategy from our results and analysis. For completeness, we include a preliminary set of results demonstrating this in Appendix
An extension of sorting samples that have been shuffled along single dimension (e.g., time, space) is sort samples shuffled across all dimensions. For images, sorting images shuffled along both the x and y axes is implemented by solving jigsaw puzzles, challenging task that reassembles image from its shuffled patches (Noroozi & Favaro 2016). This work has been extended to the video domain by solving spatio-temporal puzzles (Kim et al.) 2018). The extension to PDE data requires sorting data that have been partitioned into discrete patches and shuffled along the space and time axes; we refer to this strategy as Jigsaw. One issue is that the number of possible classes scales with the factorial of the number patches, and many shuffled sequences are not significantly different from each other. To mitigate this, we sample the top k shuffled permutations that maximize the Hamming distance between the shuffled and the original sequence [Noroozi & Favaro 2016;; this ensures that models see diverse samples during pretraining while limiting the number of classes in the pretraining task.
3.2.2 PDE Predictive Strategies
Within the PDE domain, there are physics-specific characteristics that PDE data exhibit that can be lever- aged for pretraining: this is analogous to predicting motion appearance statistics in vision pretraining
5

---

Published in Transactions on Machine Learning Research (09/2024)
tasks (Wang et al. 2019 Yao al. 2020 One strategy considers the fact that PDE data depends equation variables and coefficients, and predicting these coefficients from the PDE data could be useful. This is implemented as a regression task, where the coefficient values are regressed from a snapshot of PDE data; we refer to this stratogy as Coefficient.
Additionally, PDE data can be described by the derivatives of current physical values. For example, many finite difference schemes rely on spatial and temporal derivatives of the current vector or scalar field to propose pretraining strategy that predicts the spatial
advance the solution in time. Inspired by this, and temporal derivatives of PDE data. For 2D PDEs, this is implemented a regression tasks where the fields (Ma, My. Maz; Myy: U1) are regressed from solution u; we refer to this strategy as Derivative.
Lastly, numerical solutions of PDEs lend to leverage information of local relationships to solve equations. For example, finite difference schemes use information from neighboring nodes to calculate spatial derivatives. Motivated by this, we propose a pretraining strategy that randomly masks data space and time and uses this incomplete information to reconstruct the full solution. This is implemented by patching the solution in space and time, randomly replacing masked patches with learnable mask token, and regressing the true solution; refer to this strategy Masked, We adapt mask tokens to be learnable in physics space, rather than embedding space, in order to be used by any model beyond just transformers.
3.2.3 Contrastive Strategies
A common strategy for pretraining in computer vision domains is to exploit similarities in the data to align samples in latent space. A proposed strategy do this for PDEs is Physics Informed Contrastive Learming (PICT.), which uses Generalized Contrastive Loss (Leyva- Vallina et al. 2023 to cluster PDE data based on their coefficients in latent space (Lorsung & Farimani 2024). Another strategy for self-supervised learning of PDE dynamics is using an encoder to align Lie augmented or physically invariant latent PDE samples (Mialon et al., 2023) Zhang et al.) 2023b). Both works require the use of a specific encoder along with the neural operator backbone, since a separate model is contrastively pretrained to provide conditioning information to a backbone for physics prediction. To adapt these strategies to our experimental setup, we consider directly pretraining the neural operator contrastively with these strategies. However, these methods did not seem to show significant improvements over no pretraining, as such, they are omitted from our main results and analysis. Interested readers are directed to Appendix E for preliminary results regarding these methods.
4 Experiments
To evaluate the effectiveness of the proposed pretraining strategies and data augmentations, consider diverse set of experiments and neural operator architectures to train on. In particular, hope to understand whether different architectures or datasets influence pretraining performance and construct holistic view of protraining for diverse PDE applications. We provide an overview of the sctup and the different experiments possible in Figure] 2
4.1 Data
We consider predicting physics for the 2D Heat, Advection, Burgers, and incompressible Navier-Stokes equations. These equations describe diverse range of fluid phenomena and form tasks of varying difficulties. For our experiments, we consider pretraining on a combined set of 2D Heat, Advection, and Burgers data, which contain 9216 data samples (3072 for each equation), as well as fine-tuning on a smaller set of 1024 unscen samples for each PDE. We only pretrain on the Hcat, Advection, and Burgers equations since the numerical data for these PDEs are easier to generate, and as a result, transferring pretrained knowledge more challenging PDEs can be evaluated as palentially useful method.

---

Published in Transactions on Machine Learning Research (09/2024)
4.1.1 Heat, Advection, and Burgers Equations
The 2D Heat, Advection, and Burgers equations are given by:
deu- - -v- = Heat
(1)
au+ Vu = 0, Advection
(2)
d.u + u(c Vu) - ~veu = 0, Burgers
To ensure a diverse set of physics data, the equation coefficients are randomly sampled according to Zhou & Farimani (024b).. particular, for the Heat equation, we sample v EE [2 x 10-3,2 x 10- for the Advection equation, we sample с = [Cx, cy] E [0.1, and for the Burgers equation, we sample v e [7.5 10-3, 1.5 10, and c = [c., E [0.5, 1.0]3; we refer this dataset as in-distribution (In). Since where the downstream
these equations also comprise the pretraining set, additionally consider dataset comes from separale distribution; in this case, sample E 10 2 10 for the Heat equation, c [cz. Cyl E 3.0) for the Advection equation, and E [5.0 10 3,7, x 10 and [Cx, E [1.0, 1.25] for the Burgers equation. We refer to this dataset as out-of-distribution (Out).
In all cases, periodic boundary conditions are enforced and the solution solved in a domain (æ, y) [-1, 1, 1]2 from 0 to 2. Furthermore, initial conditions are randomly from summation of sine functions: the parameters are uniformly sampled from from Aj [-0.5, 0.5], E -.4,, (1,2,3),44 Lui e {1,2,3}, [0, 2T) while fixing J 5, 2:
x0, = Σ Assin(2nl + 2 + (1))
(4)
For additional information on data splits and numerical methods, we refer readers to Appendix
D.1
4.1.2 Incompressible Navier Stokes Equations
The incompressible Navier Stokes equations considered for fine-tuning pretrained models to predict more challenging physics. To ensure consistency between the pretraining and fine-tuning tasks, we use the vorticity form of the Navier-Stokes equation in order to predict a scalar field following the setup in Li et al. [2021]
Onw + n Ve -w- Www = =wwwww=s(a. u= Vxu= f(x,y) A(sin(27Tr y)) (2
(5)
(6)
We formulate this problem with periodic boundary conditions, variable viscosity v, and variable forcing function amplitude A. Specifically, the viscosity is sampled uniformly from v ((1,2,3, 5, 6,8,8 9} 10 (6,7,8,8)))) } and the amplitude is uniformly sampled from A ((1,2,3, 4, 7,8,9, 10) 10-33. The data is generated in domain (x. 1) and from t= 0 to t 7.75, following the setup from Lorsung et al. h2024b: ; furthermore, the initial conditions wp are generated from Gaussian random field according to Li
et al. 2021b.
4.2
Neural Operators
To compare different pretraining and data augmentation strategies, consider their effects on improving the PDE prediction performance of different neural operators. Specifically, we consider the neural operators: Fourier Neural Operator (FNO) (Li et al. 2021), DeepONet (Lu et al. 2021a) and OFormer (Li et al. 2023a). Additionally, we consider the Unet model; while it is not explicitly a neural operator, is commonly used in
literature and has shown good performance (Ronneberger et al.) 2015 Gupta & Brandstetter 2022) These
neural operators are first trained using a pretraining strategy before being fine-tuned on a PDE prediction fask; this could either be fixed-future prediction to model static solution autoregressive prediction to

---

Published in Transactions on Machine Learning Research (09/2024)
Pretraining
Fine-tuning
Data/Augs Pretrain Data + Noise
Model FNO DeepONel Unet Ete..
Sontepas Trunsler Contrastive Masked Eie..
Model FNO DeepONel Unet Ete..
Data/Dast
Tacks
Pretrain New Coeffis
Fixed-Future
Autoregressive
Etc...
New PDEs
Figure 2: Experimental Setup. During pretraining we consider different data augmentations, model choices, and pretraining lasks and evaluate their downstream performance through fine-tuning on physics prediction tasks. During fino-tuning, we leverage the same pretrained model improve fixed-future or au- loregressive prediction the same pretraining data distribution, unseen coefficients, or new PDEs. Through this setup we can explore a wide variety of pretraining strategies and angmentations and quantify their effects on different models, PDEs, datasets, and tasks.
model a time-dependent solution. In all experiments, prediction tasks are formulated using only solution field values and grid information. Additional details on the model hyperparameters and implementation can be found in Appendix D.2
4.3 Pretraining Strategies
We compare models pretrained with different strategics with a baseline model that has not been pretrained (None) as well as a model trained with the same physics prediction objective on the pretraining dataset, more commonly known as transfer learning (Transfer). Furthermore, we vary the size of the fine-tuning dataset study the effects of pretraining when given scarce downstream data. The fine-tuning dataset is also varied between data samples that are within the pretraining distribution (In), outside the pretraining distribution with respect to the PDE coefficients (Out), or on samples from an unseen PDE (NS). Lastly, we study the effects of adding data augmentations during pretraining and fine-tuning.
4.4 Data Augmentation
Data augmentation is implemented by doubling the pretraining and fine-tuning data, where cach sample has 50% chance of being augmented. Our noise augmentation adds small amount of Gaussian noise to each frame independently, while our shift and scale angmentations are applied uniformly to the entire trajectory.
4.5 Fixed Future and Auto-regressive Prediction
To model physics problems with static solutions, we consider predicting PDE solution field at a fixed timestep after initial snapshot of the PDE data. In particular, given the PDE data from t = 1 to 8, models are trained to predict the PDE solution at 32.
Alternatively, to model physics problems with lime-dependent solutions, we consider auto-regressively pre- dicting PDE solutions directly after a current snapshot of PDE data. This is implemented using PDE data on the interval [t, t + 8) as an input to predict future PDE solutions on the interval [t + 8, + 16). In addi- tion, we use the pushforward trick (Brandstetter al. 2023) to stabilize training. This introduces model noise during training by first predicting a future time window from ground-truth data and then using this noisy prediction model input; importantly, no gradients are propagated through the first forward pass. Additional details on training parameters can be found in D.5
5 Results
We now systematically benchmark our pretraining and data augmentation strategies, as well their combi- nation. Presented below results on our autoregressive task. Fixed-future results given in appendices
8

---

Published in Transactions on Machine Learning Research (09/2024)
Table 2: Effects of Pretraining for Auto-regressive Prediction: We present comparisons of different pretraining strategies after pretraining on the Heat, Advection, and Burgers equations and fino-tuning in 500 unseen samples. The insights are distilled into two tables, for full results see Appendix A
(a) The best pretraining strategy varies with model and dataset choice. We compare the high- performing pretraining strategies on autoregressive prediction; although performance varies widely, transfer learning performs well in many settings.
(b) Different models display different benefits from pretraining. We compare the improvement of the highest performing pretraining strategy to no pre- training. The models show different capacities to be pre- trained.
Best Pretraining Method
Improvement w/ Best Strategy
Model
Heat
Advection
Burgers
NS
Model
Heat
Advection
Burgers
NS
FNO
Derivative
Transfer
PICL
None
FNO
14.43%
7.459%
1.430%
0.000%
Nee
PICL
Transfer
Transfer
Transfer
NeepONet
3.580%
1.852%
15.74%
2.894%
OFormer
Transfer
PICL
Transfer
None
OFormer
38.91%
4.594%
17.12%
0.000%
Unet
Transfer
TimeSort
Transfer
Transfer
Unet
19.16%
1.899%
9.706%
1.862%
A and B and generally show the same trends as our autoregressive results. ct al.
We use Relative L2 error
2021
for both training and evaluation in all of our experiments.
5.1
Comparison of Pretraining Strategies
We benchmark our proposed PDE pretraining strategics on different neural operators and datasets, and show the condensed results for auto-regressive prediction in Table 2 For detailed comparison, we present results of different PDE pretraining strategies for fixed-future and auto-regressive tasks on all datasets in Appendix Additionally, we consider cases where the fine-tuning datasct contains coefficients unseen during pretraining, and present these out-of-distribution results in Appendix A as well.
Through these experiments, we find multiple insights. Firstly, we observe that the pretraining performance varies with the choice of model and dataset. Specifically, different models benefit differently from pretraining, as well as based on the predicted PDE and task (i.e. fxed-future vs. auto-regressive). However, transfer learning generally performs well across different tasks, models, and datasets, suggesting that it is a good choice for a pretraining task. This is also reflected in the literature, where previous work generally focuses on transferring knowledge between datasets (Chen et al.] 2021 Goswami et al.) 2022 Chakraborty et al. 2022 Tripura & Chakraborty/ 2023) or pretrain by predicting physics of large datasets (Hao et al., 2024 McCabe et al.) 2023; Subramanian et al., 2023). We hypothesize that transfer learning is effective since PDE data is inherently unlabeled; physics prediction uses future timesteps as a label, similar to next-token prediction for GPT models, which is cast as self-supervised learning. When the data is sufficient, using surrogate objectives such derivatives sorting sequences may not be effective as the true objective of fixed-future of auto-regressive prediction. Another observation is that pretraining frameworks are generally dependent on specific architectures; for example, many CV pretraining strategies shuffle patches of data, which can introduce arbitrary discontinuities and high-frequency modes in FNO models, yet are not as challenging for convolutional models such as Unet. Furthermore, pretraining strategics are also dependent on the downst.ream task; for example, Derivative pretraining works well for auto-regressive prediction but not fixed future prodiction, as the solution at a distant timestep is very different from the current derivatives, but the solution at the next. timestep is highly dependent on the current derivatives.
Secondly, observe that directly adapting computer vision methods to the physics domain generally results in poor performance. In many experiments, using CV pretraining method would often hurt performance compared to not pretraining. This points to a general difference between CV and physics tasks. In the vision domain many downstream tasks are classification based (i.e. ImageNet, Object Detection, etc.), which results many pretraining tasks modeled around classification, whereas physics prediction is a high- dimensional regression task. Beyond this, physics predictions not only need be visually consistent, but also nunerically accurate, which can be difficult to learn from classification task. In fact, using physics-based
9

---

Published in Transactions on Machine Learning Research (09/2024)
Table 3: Effects of Data Augmentation for Auto-regressive Prediction: We present comparisons of different pretraining strategies combined with data augmentations after pretraining on the Heat, Advection, and Burgers equations and fine-tuning in 500 unseen samples. The data is distilled into two tables; for full rosults see Appendix B
(a) The best pretraining with data augmentation strategy varies with model and dataset. Different models benefit from different augmentations when paired with pretraining strategies. Note: denotes PICL and denoles transfer learning.
(b) Adding data augmentations consistently im- proves performance. When choosing the correct com- bination of pretraining and data augmentation strategies, we find it improves performance during autoregressive prediction compared to baselines.
Best Angmentation
Improvement w/ Best Augmentation
Model
Heat
Advection
Burgers
NS
Model
Heat
Advection
Burgers
NS
FNO
Shift
Scale
None
None
FNO
14.21%
8.287%
1.411%
0.000%
DeepONet
Shift
Shift
Shift!
Shift
DecpONet
8.745%
1.337%
13.21%
3.761%
OFormer
Noise!
None
Noise'
Noise!
OFormer
35.35%
4.426%
16.288%
13.23%
Unet
Shift!
Shift?
Shift
Noise
Unet
30.051%
0.735%
10.322%
3.434%
pretraining methods, such as transferring between prediction tasks, regressing derivatives, or physics- informed contrastive loss, generally results in better performance.
Lastly, we observe that different models have different capacities for pretraining. For example, the OFormer architecture, which is based on transformers, benefits greatly from pretraining in many scenarios; this could be because transformers lack inductive bias and can model arbitrary relationships. Furthermore, Unet ar- chitectures also benefit consistently from pretraining: this reflected common convolutional architectures used for pretraining in the CV domain, such as ResNet (He al. 2015). DeepONet and FNO show smaller improvements with pretraining, suggesting that the architectures are less tailored for pretraining. This is especially true for FNO; we hypothesize that the learned Fourier modes may be very different between tasks, resulting in challenges when transferring weights to new tasks.
5.2 Comparison of Data Augmentations
To study the effects of angmenting data during pretraining and finetuning, we conduct experiments in which data augmentations are added to three pretraining strategies ( (None, Transfer, PICL). These experiments are run to compare data augmentations to a baseline model that is not pretrained, as well as its effects on the most effective pretraining stralegies (i.e. Transfer, PICI... The results are summarized for auto-regressive prediction, and the complete results can be found in Appendix B We find the best augmentation by considering the protraining strategy and augmentation pairing with the lowest error. To calculate its improvement, this error is compared to model that is not pretrained, however, to isolate the effect of pretraining, this can be compared to a corresponding pretrained model without data augmentation in the Appendix.
Table
We find that different models benefit from different augmentations; for example, DeepONet performs well with shifted data, but OFormer performs well with noised data. However, across models, datasets, and downstream tasks, one can generally find data augmentation that improves performance. This suggests that the most effective pretraining frameworks should incorporate data augmentation strategy, and indeed the best- pest-performing models considered in this study often make use of data augmentations, Furthermore, when an augmented model is compared to its bascline, the performance tends to increase even when the baseline is pretrained, albeit. modestly. Transfer learning performs best in nine of our 12 cases, and shift augmentation performs best in eight of our 12 cases, with their combination performing best in six, suggesting that this combination improves performance best across different data sets and models. We believe that data augmentations can help due to the fact that PDE data remains scarce; numerical simulation is needed for high quality data, and result emulating larger dataset with augmentations is beneficial.
10

---

Published in Transactions on Machine Learning Research (09/2024)
Table 4: Effects of Downstream Dataset: We compare the effect pretraining without data aug- mentation when the downstream dataset varied-both with the number of samples or the distribution of samples. We find that pretraining benefits more in data-scarce regimes, as well as when the downstream data is similar to the pretraining data.
(a) Pretraining is more beneficial when down- stream data is scarce. Improvement is measured by comparing the best pretraining method with pretrain- ing. We report the average improvement across the Heal, Adv, and Burgers PDEs for given # samples and model.
(b) Pretraining improves performance for both in and out-of-distribution downstream tasks. For each distribution of downstream data, find the best improvement from pretraining and average across PDEs. Models show varying generalization capacities.
Best Improvement over None
Best Improvement over None
# Samples
FNO DeepONet OFormer
Unet
Distribution
FNO DeepONel
OFormer
Unet
100
-5.533%
6.755%
47.60%
23.05%
6.875%
6.630%
19.57%
13.592%
250
4.111%
7.361%
29.60%
18. 18.18%
Out
3.920%
-3.383%
34.058%
27 27.696%
500
6.875%
6.630%
19.57%
13.59%
NS
-8.658%
2.899%
-1.608%
1.865%
1000
2.026%
6.135%
13.27%
2.995%
5.3
Scaling Behavior
We compare the effect of pretraining for different mumbers of downstream samples in Table 4 this effect by finding the best pretraining method for a given model, PDE, and dataset size, then calculating its improvement over no pretraining: after calculating the improvement, we average this metric across the Heat, Advection, and Burgers PDEs for auto-regressive prediction. In general, we observe a trend in which the improvement of pretrained models diminishes the number of fine-tuning samples increases, which is expected as fine-tuning data approaches the pretraining dataset size. Tt follows that if the downstream data is abundant, directly training on this would be optimal. Additionally, despite these trends, the relative im- provement of different pretraining strategies remains approximately constant between different reanstream dataset sizes. An exception to these trends the FNO model; we hypothesize that learned Fourier modes may be more challenging to fine-tune than other learning mechanisms such as attention matrices or convolutional kernels.
We measure
For detailed comparison of the scaling behavior in individual datasets and models, we refer readers to Appendix
C Empirically, we observe higher variance between random seeds when using a smaller dataset for fine-tuning. Furthermore, the advection equation can generally be learned with fewer samples and the performance is approximately constant with increasing dataset size. Additionally, different models and pre- training strategies display different scaling behaviors, with some models and pretraining strategies displaying greater increases in performance when fine-tuning to scarce data. This further underscores the importance of proper architecture choices that scale well, such using transformer-based neural operators. Lastly, scaling behavior is more pronounced in fixed-futuee experiments; this could be because there is less data in fixed-future experiments due to only predicting single target per data sample as opposed to predicting multiple targets across longer auto-regressive rollout.
5.4 Generalization Behavior
We compare the effect of varying the distribution of the downstream dataset on the performance of pretrained models. In particular, we compare fine-tuning to unseen coefficients of the same equation (Out) as well fine-tuning to an unseen PDE with novel initial conditions and forcing terms (NS); these results are shown in Table A with 500 fine-tuning samples for auto-regressive prediction. In general, we observe reduced performance when finc-tuning to the Navier-Stokes equations, compared to fine-tuning to samples within the pretraining distribution (Jn). For certain models, this also holds when fine-tuning to a dataset with unseen coefficients (Out). These generalization behaviors are also approximately consistent between different sample sizes of the fine-tuning datasct. It is important note that certain protraining frameworks generalize better than others: for example, Coefficient pretraining largely hurts performance, since the fine- tne-tuning distribution contains different. coefficients by construction.
11

---

Published in Transactions on Machine Learning Research (09/2024)
We note that the OFormer and Unet architectures show better performance when fine-tuning to out-of- distribution samples; we hypothesize, that this is due to shifts in coefficients causing easier phenomena to model. For example, increasing the diffusivity in the heat equation causes transient effects to be concentrated in a few initial timesteps and sparse behavior for the majority of the rollout. Nevertheless, under certain conditions, pretraining shows generalization to unseen coefficients and PDEs, which a promising direction.
6 Discussion
6.1 Theoretical Insights
There are many insights from the broader deep learning community that can help inform our results. In our study, an important factor in pretraining performance is the choice of model architecture, which is refeectd both in both PDE and deep learning literature. Many recent PDE works have chosen to include attention
modules in their architcctures (Hao et al. 2024 Herde et al.) 2024 Li et al. 2023a McCabe et al.) 2023 Hang et al. 2024), which is supported by the observation that translormer archilectures can be expressive by making the least assumptions on the structure of data (Jiang et al.) 2022). We similarly find that OFormer tends to display larger improvements compared to other architecturcs. Furthermore, Kolesnikov et al. (020) find that training on larger datasets is vital for transferability, which also observe by limiting the fine-tuning data.
Additionally, the use of data augmentations can be understood as assembling a more comprehensive training set such that the distance between the training set and validation/tess /test sets is reduced (Shorten & Khoshgof taar 2019). In the context of PDEs, physics data can contain different coefficients, initial conditions, and boundary conditions; augmentations can help to mimic invariances to these conditions (e.g., spatially shifting data to model different initial conditions, scaling data to mimic different coefficients, etc.) such that models can perform well in spite of these challenges. The utility of data augmentation is further exacerbated by the difficulty in generating high-quality PDE data from numerical simulation, which could motivate future work in exploring more complex data augmentation approaches such as generating synthetic samples from GANs Sandfort et al., 2019) or neural style transfer (Gatys et al. 2016).
Lastly, we consider the connection of different PDE protraining stratogics to broader deep learning pretraining insights. In general, it is understood that unsupervised learning is beneficial due to the abundance of unlabeled data and the human labor required to label samples (Bengio et al., 2021). However, in the context of PDE data, the goal is usually to predict future frame of a current solution or to upsample low-resolution physics, both of which do not require human intervention to label data. In addition, numerical simulation is required to generate PDE data, which is the limiting factor in assembling large datasets. As a result, we hypothesize that this limitation the main factor as to why unsupervised PDE methods relying on surrogate objectives do not perform as well as supervised pretraining, also known as transfer learning. Currently, unsupervised methods rely on the same data as supervised methods, both because of the computational cost of numerical simulation and because of the lack of human labor needed to label supervised samples. Addressing this limitation could be a promising direction for future work unsupervised PDE learning.
6.2 Best Practices
Our experiments have demonstrated that transfer learning combined with the shift augmentation performs best in the majority of our experiments. For pretraining strategies, we believe that transfer learning performs best because additional pretraining strategies are not able to extract additional, task-specific information better than transfer learning. Having identical training tasks is simply more effective in this case. For data augmentation, the shift augmentation is the only augmentation that preserves the underlying dynamics of our data since all of our systems are spatially invariant. We believe that this improves performance over the other augmentation strategies, which do not preserve the underlying dynamics. For further applications, transfer learning + shift augmentation is a good initial pretraining strategy, but practitioners will likely need to tune and adapt additional strategics to achieve maximal performance. Furthermore, previous work [Brandstetter et al.) 2022) has demonstrated improved performance by combining multiple augmentations, which may help across many systems. However, due to computational cost, we leave this to future work for
12

---

Published in Transactions on Machine Learning Research (09/2024)
each practitioner's specific application, as best practices are likely model and dataset dependent. Lastly, to maximize the benefit of pretraining, recommend collecting as much pretraining data as possible.
7
Conclusion
In this work, we compare pretraining strategies for PDEs by examining pretraining frameworks that can be used across different models and datasets. In particular, we consider adapting CV pretraining to the PDE domain through sorting spalio-temporal data to learn underlying dynamics without labels. Furthermore, we derive several PDE characteristics that can be predicted, such as its coefficients, derivatives, or reconstructed input. Lastly, we implement existing contrastive as well as transfer learning strategies to construct diverse set of pretraining strategies. Notably, these strategies can be applied any model and PDE problem and are flexible to future advances in architectures datasets.
Through pretraining with different frameworks and data augmentations, we compare their effects on dif- ferent PDEs, models, downstream datasets, and fine-tuning tasks. We find that pretraining can be highly dependent on model and dataset choices, but in general transfer learning or physics-based strategies do well. Furthermore, we find that directly adapting pretraining strategies from other domains often fails, motivat- ing the need to design PDE-specific pretraining frameworks. Lastly, we observe that different models have different capacities for pretraining, with transformer and CNN based architectures benefiting the most from pretraining and highlighting the need for architectures that have high capacity and transferability.
To further understand PDE pretinining, we investigale the effect of adding data augmentations and varying the fine-tuning dataset. We find that data augmentations consistently benefit performance, with the shift augmentation showing best performance most often. Combining transfer learning with shift augmentation shows the best performance in the majority of lest cases. Additionally, pretraining performance is accentu- ated when the fine-tuning dataset is scarce or similar the pretraining distribution. Through establishing deeper understanding of pretraining for PDEs, we hope that future work can leverage these insights to propose new pretraining strategies and expand current architectures.
References
M. Baer. findiff software package, 2018. URL htttge//gghhab.ccc/a/arbb///iiff bitta://gitthn com/maroba/finddff
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg:
regularization for self-supervised learning. In International Conference on Learning Representations, 2022. URL https:
Yoshua Bengio, Yann Lecun, and Geoffrey Hinton. Deep learning for ai. Commun. ACM, 64(7):58 65, jun 2021. ISSN 0001-0782. doi: 10.1145/3448250. URL https: /t/p://////oo.r//11... 1145/3448250
Johannes Brandstetter, Max Welling, and Daniel E. Worrall. Lie Point Symmetry Data Augmentation for Neural PDE Solvers, May 2022. URL http:://aaww..og//abb/////////// org/abs////2 22622026222200233 arXiv:2202.076433 [es].
Johannes Brandstetter, Daniel Worrall, and Max Welling. Message passing neural pde solvers, 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Choss, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Zhonglin Cao, Rishikesh Magar, Yuyang Wang, and Amir Barati Farimani. Moformer: Self-supervised transformer model for metal-organic framework property prediction. Journal of the American Chemical Society, 145(5): 2967, 2023. doi: 10.1021/jacs.21111 2c14420 URL https:///oi. org/ei..rgo..... 1021j/ja............ PMTD: 36706365.
2c1 1420
13

---

Published in Transactions on Machine Learning Research (09/2024)
Ayan Chakraborty, Cosmin Anitescu, Xiaoying Zhnang, and Timon Rabezuk. Domain adaptation based transfer learning approach for solving PDEs on complex geometries. Engineering with Computers, 38(5): 4569 4588, October 2022. ISSN 1435-5663. doi: 10.0007/:00336-02--016111 URL https://doi.orr: 10.
Fei-Long Chen, Du-Zhen Zhang, Ming-Lun Han, Xiu-Yi Chen, Jing Shi, Shuang Xu, and Bo Xu. Vlp: A survey on vision-language pre-training. Machine Intelligence Research, 20(1):38 56, Feb 2023. ISSN 2731-5398. doi: (00000//1633-02-111300 URL https://dol.org//u 1tt../000///11100022--1111
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Gcoffrey E. Hinton. A simple framework for con- trastive learning of visual representations. CoRR, abs/ /000/200005709 2020. URL https://arxiv.orr/aan
2002.05709
Wuyang Chen, Jialin Song, Pu Ren, Shashank Subramanian, Dmitriy Morozov, and Michael W. Mahoney. Data-efficient operator learning via unsupervised pretraining and in-context learning, 2024.
Xinhai Chen, Chunye Gong, Qian Wan, Liang Deng, Yunbo Wan, Yang Liu, Bo Chen, and Jie Liu. Transfer learning for deep neural nelwork-based partial differential equations solving. Advances in Acrodynamics, 36)):36, December 2021. ISSN 2524-6992. doi: 10. 10.116//=22774001--000-1 URL https://doi. org/10. 1186/s42774-021-000997
Ze Cheng, Zhongkai Hao, Xiaoqiang Wang, Jianing Huang, Youjia Wu, Xudan Liu, Yiru Zhao, Songming Liu, and Hang Su. Reference neural operators: Learning the smooth dependence of solutions of pdes on geometric deformations, 2024.
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zhong, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. A survey on in-context learning, 2023.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. agnostot meta-learning for fast adaptation of deep networks, 2017.
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2414-2423. 2016. doi: 2.1109/CVPR.2016
Somdatta Goswami, Katiana Kontolati, Michael D. Shields, and George Em Karniadakis. Deep transfer operator learning for partial differential equations under conditional shift. Nature Machine Intelligence, 4 (12):1155 1164, December 2022. ISSN 2522-5839. nature. com
doi: 10.1038/ (412256-022-0066--- URL https://www. Number: 12 Publisher: Nature Publishing Group.
Jayesh K. Gupta and Johannes Brandstetter. Towards Mutts-ppattotemmprallseeee Generalized PDE Model-
ing, November 2022. URL httt:://axxlv.org/bbb..
15616
arXiv:2209.15011 [cs].
Zhou Hang, Yuezhou Ma, Haixu Wu, Haowen Wang, and Mingsheng Long. transformers are universal pdc solvers, 2024.
Unisolver: Pde-conditional
Zhongkai Hao, Chang Su, Songming Liu, Julius Berner, Chengyang Ying, Hang Su, Anima Anandkumar, Jian Song, and Jun Zhu. Dpot: Auto-regressive denoising operator transformer for large-scale pde pre- training, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners, 2021.
14

---

Published in Transactions on Machine Learning Research (09/2024)
AmirPouya Hemmasian and Amir Barati Farimani. Reduced-order modeling of fluid flows with transformers. Physics of Fluids, 35(5):057120 05 2023. ISSN 1070-6631. doi: 10.1063/5.0151515 URL https://doi. htg/10.1063/55
Maximilian Herde, Bogdan Raonić, Tobias Rohner, Roger Käppeli, Roberto Molinaro, Emmanuel de Bézenac, and Siddhartha Mishra. Poscidon: Efficient foundation models for pdes, 2024.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovee. Strategies for pre-training graph neural networks, 2020.
Nail H. Tbragimov. CRC Handbook of Lie Group Analysis of Differential Equations: Symmetries, Eract Solutions, and Conservation Laws. Number v. 1. Taylor & Francis, 1993. ISBN 9780849344886, URL https://bookss. .aaooks.goooe google......oooooo.. mongle.com.a him.ooooooddddmmyyyggggog
Junguang Jiang, Yang Shu, Jianmin Wang, and Mingsheng Long. Transfcrability in Deep Learning: A
Survey, January 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural langnage models, 2020.
Dahun Kim, Donghyeon Cho, and In So Kweon. Self-supervised video representation learning with space-time cubic puzzles. 11 2018. URL
Matthieu Kirchmeyer, Yuan Yin, Jérémie Donå, Nicolas Baskiolis, Alain Rakotomamoniy, and Patrick Gallinari. Generalizing to new physical systems via context-informed dynamics model, 2022.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning, 2020. URL /ttt:ttt/////xxi..orrr
1912. 11370
Hsin-Ying Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Unsupervised representation learning by sorting sequences. 8 2017. URL http://///aaivvorrr/////0//
María Leyva- Vallin,, Nicola Strisciuglio, and Nicolai Petkov. Data-efficient large scale place recognition with graded similarity supervision. CVPR, 2023.
Zijie Li, Kazem Mcidani, and Amir Barati Farimani. Transformer for partial differential equations' operator learning, 2023a.
Zijie Li, Dule Shu, and Amir Barati Farimani. Scalable transformer for pde surrogate modeling, 2023b.
Zijie Li, Authony Zhou, Saurabh Patil, and Amir Barati Farimani. Cafa: Global weather forecasting with factorized attention on sphere, 2024.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations, 2021.
Zongyi Li, Nikola Borislavov Kovachki, Chris Choy, Boyi Li, Jean Kossaifi, Shourya Prakash Otta, Moham- mad Amin Nabian, Maximilian Stadler, Christian Hundt, Kamyar Axizaadenshheli, and Anima Anandku- mar. Geometry-informed neural operator for large-scale 3d pdes, 2023c.
Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards Out-Or- Distribution Generalization: A Survey, July 2023a.
Yuxuan Liu, Zecheng Zhang, and Hayden Schaeffer. Prose: Predicting operators and symbolic expressions using multimodal transformers. arXiv preprint arXiv:2309.16814 2023b.
Cooper Lorsung and Amir Barati Farimani. PICL: Physics Informed Contrastive Learning for Partial Differ- wrg//as/////...aaa..og
ential Equations, January 2024. URL http:///arr...orrgaa////....... 16327 arXiv:2401.16327 [physics].
15

---

Published in Transactions on Machine Learning Research (09/2024)
Cooper Lorsung, Zijie Li, and Amir Barati Farimani. Physics informed token transformer for solving partial differential equations. Machine Learning: Science and Technology. 5(1 )001032, feb 2024. doi: 10. 10.1088/ 2632-21122 211222133dddde URL
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via. deeponet based the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218 229, March 2021a. ISSN 2522-5839. doi: 10.1088/s22256021-0000222 URL http:
7/dx.doi... org/10. 1038/s422561 0006//2226-0011-0002---
Lu Lu, Xuhui Meng, Zhiping Mao, and George Fm Karniadakis. DeepXDE: deep learning library for solving differential equations. SIAM Review, 63(1)2208 228, 2021b. doi: 10.1137/19M1274067.
Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engincering, 393:114778, April 2022. ISSN 0045-7825. doi: (..1016/j.cma.2022. 114778. URL http://dxx.... doi. owg//o.g//11/....ooo 1016/ cma. 2022.
http://dxx.do..org110..000
114778
Michael McCabe, Bruno Régaldo-Saint Blancard, Liam Holden Parker, Ruben Ohana, Miles Cranmer, Al- berto Bietti, Michael Eickenberg, Siavash Golkar, Geraud Krawezik, Francois Lanusse, Mariel Pettee, Tiberiu Tesileanu, Kyunghyun Cho, and Shirley Ho. Multiple Physics Pretraining for Physical Surrogate Models, October 2023. URL http://arxiv. org/ Nwrg/abs/23100 02999 arXiv:2310.09994 [cs, stat].
Kazem Meidani, Parshin Shojace, Chandan K. Reddy, and Amir Barati Farimani. Snip: Bridging mathe- matical symbolic and numeric realins with unified pre-training, 2024.
Grégoire Mialon, Quentin Garrido, Hannah Lawrence, Danyal Rehman, Yann LeCun, and Bobak T. Kiani. Sell-Supervised Learning with Lie Symmetries for Partial Differential Equations, July 2023. URL http:
//arxiv. wwwpiii.org//baa. httgg//bb/2007// 05432 wrXiv:2307.05432 [cs, math].
Ishan Misra, C. Lawrence Zitnick, and Martial Hebert. Shuffle and learn: Unsupervised learning using temporal order verification. 3 2016. URL jattp://arxiv. orggabs///// wrgg/////603....////// 1ttp//663.00000 08561
Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K. Gupta, and Aditya Grover. Climax: A foundation model for weather and climate, 2023.
Mchdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. 3 2016. URL http:://arxiw.org/bbbb/ 09246
Peter Olver. Applications of Lie Groups to Differential Equations. Springer New York, NY, 1986.
Luis Perez and Jason Wang. The effectiveness of data augmentation in image classification using deep learning, 2017.
Md Ashiqur Rahman, Robert Joseph George, Mogab Elleithy, Daniel Leibovici, Zongyi Li, Boris Bonev, Colin White, Julius Berner, Raymond A. Yeh, Jean Kossaifi, Kamyar Azizzadenesheli, and Anima Anandkumar. Pretraining codomain attention ncural operators for solving multiphysics pdes, 2024.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation, 2015.
Veit Sandfort, Ke Yan, Perry Pickhardt, and Ronald M. Summers. Data augmentation using generative adversarial worksrks (cyclegan) to improve generalizability in et segmentation tasks. Scienlific Reports, 911):16884, Nov 2019. ISSN 2045-2322. doi: 10.1038 141033/4415980111 019- URL https://6oi.orr//00 1098/841198-019-527777
Madeline C. Schiappa, Yogesh S. Rawat, and Mubarak Shah. Self-supervised learning for vidcos: A survey. 6 2022. doi: 10.1145/3577925. 3577925
URL http://wwwaw...oraaas/////00110. doi, org/10.11455 1145/
16

---

Published in Transactions on Machine Learning Research (09/2024)
Junhong Shen, Tanya Marwah, and Ameet Talwalkar. Ups: Efficiently building foundation models for pde solving via cross-modal adaptation, 2024.
Connor Shorten and Taghi M. Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6(1):60, Jul 2019. ISSN 2196-1115. doi: 10.1186/4400337-11--019700 URL https: 7/doi. org.0/..g11..o
Shashank Subramanian, Peter Harrington, Kurt Keutzer, Wahid Bhimji, Dmitriy Morozov, Michael Ma- honey, and Amir Gholami. Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior, May 2023. URL [cs, math].
00258, arXiv:2306.00258
Makoto Takamoto, Francesco Alesiani, and Mathias Niepert. CAPE: Chameelatttentiinn--aaedd PDE param- eter embeddings for sciML, 2023. URL
Tapas Tripura and Souvik Chakraborty. A foundational neural operator that continuously learns without forgetting, October 2023. URL
2211////www..rr/////211...11110. 18885 arXiv:2310.18885 [cs].
Jiangliu Wang, Jianbo Jiao, Linchao Bao, Shengfeng Hc, Yunhui Liu, and Wei Liu. Self-supervised spatio- temporal representation learning for videos by predicting motion and appearance statistics. 2019. URL
http://arxiv. org/ aba 1904. 03597
Yuyang Wang, Rishikesh Magar, Chen Liang, and Amir Barati Farimani. Improving molecular con- trastive learning via faulty negative mitigation and decomposed fragment contrast. Journal of Chem- ical Information and Modeling, 62(11))2713 2711-22273 2022a. doi: 10.1021/acs sac.jjeim.2c004999 URL https:
7/doi.org 10. 1021/acc....... jeim. 2c00495
PMTD: 35638560.
Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 4(3):279-287, Mar 2022b. ISSN 2522-5839. doi:
URL https://doi: org/ 00000///225000020044777777
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modcling, 2022.
Liu Yang, Siting Liu, Tingwei Meng, and Stanley J Osher. In-context operator learning with data prompts for differential equation problems. Proceedings of the National Acudemy of Sciences, (3333112221111112 2023.
Yuan Yao, Chang Liu, Dezhao Luo, Yu Zhou, and Qixiang Ye. Video playback rate perception for self- superrisedpaatiotenppraa representation learning. 6 2020. URL http://arxiv..rrg wwtp:iiiv.rg/a///////// 0aabs/2006.1.
Yuan Yin, Ibrahim Ayed, Emmanuel de Bézenac, Nicolas Baskiotis, and Patrick Gallinari. Leads: Learning dynamical systems that generalize across environments. 2021. URL http::/arxiv.ort wttp:/ii..rg//b///1100 04546
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction, 2021.
Lu Zhang, Huaiqian You, Tian Gao, Mo Yu, Chung-Hao Lee, and Yue Yu. Metano: How to transfer your knowledge on learning hidden physics, 2023a.
Rui Zhang, Qi Meng, and Zhi-Ming Ma. Deciphering and integrating invariants for neural operator learning with various physical mechanisms. National Science Review, pp. nwad336, December 2023b. ISSN 2095-5138, 2053-714X. doi: 10.1093/nst /nw/////////36.... URL http://arrt..rrg//bb//21.1. org/abs/2311..1... 14361 1rX1v:2311.14361 [physics].
17

---

Published in Transactions on Machine Learning Research (09/2024)
Zezhong Zhang, Feng Bao, Lili Ju, and Guannan Zhang. Transnet: Transferable neural networks for partial differential equations, 2023c.
Anthony Zhou and Amir Barati Farimani. Faultformer: Pretraining transformers for adaptable bearing fault classification. IEEE Access, 12:70719-70722 2024a. doi: 10.1
Anthony Zhou and Amir Barati Farimani. Masked autoencoders are pde learners, 2024b.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A Comprehensive Survey on Transfer Learning, June 2020.
18

---

Published in Transactions on Machine Learning Research (09/2024)
A Comparison of Pretraining Strategies
A.1 Fixed Future Experiments
Table 5: Models are pretrained on 9216 combined 2D Heat, Advection, and Burgers samples and finetuned 500 samples for each PDE. Normalized L2 errors (x10-l) are calculated on 256 validation samples and averaged over five sceds. The lowest errors are given in dark grey and second lowest errors are given light grey.
(a) Fixed Future Pretraining Results.
PDE
Model
None
Transfer
Binary
TimeSort
Jigsaw
CofffcdentDeriiatiieeaaaaeed
PICL
FNO
0.240
0.467
0.813
0.771
0.838
0.387
2.742
0.557
0.182
DeepONet
0.669
0.246
0.755
0.435
0.598
0.467
0.602
0.483
0.675
Heat
OForee
0.762
0.275
7.520
8.822
3.450
1.898
0.413
0.531
0.630
Unet
0.150
0.061
0.378
0.339
0.483
0.234
0.131
0.118
0.145
FNO
3.533
1.517
7.741
5.427
6.138
6.442
6.205
4.522
3.555
DeepONet
9.907
9.587
10.006
9.814
9.978
9.875
9.926
9.840
9.952
Adv
OFormer
9.645
5.334
10.006
10.022
10.006
10.010
9.878
9.795
9.206
Unet
3.962
1.488
5.747
5.568
6.509
9.286
4.909
4.058
3.802
FNO
0.704
0.675
1.238
1.120
1.226
1.139
4.461
0.896
0.694
DeepONet
4.096
3.37
4.758
3.638
3.869
3.776
3.987
3.674
4.195
Burgers
OFormer
1.92
1.517
9.318
9.792
5.024
3.610
2.112
1.997
1.994
Unet
1.027
0.771
1.382
1.174
1.450
1.168
0.918
0.822
0.989
FNO
2.112
2.147
3.500
6.285
3.530
3.874
6.200
2.386
2.232
DeepONet
5.560
5.226
7.650
5.907
15.208
5.712
5.990
5.610
5.514
NS
OFormer
3.744
3.801
6.056
6.099
6.099
5.445
4.631
4.670
4.056
Unet
2.279
1.403
3.261
2.847
3.488
2.493
2.341
2.332
2.262
(b)
Duitof-Distribution Fixed Future Pretraining Results.
PDE
Model
None
Transfer
Binary
TimeSort Jigsaw
CoefffcienDDerivatiieeeeeee Masked
PICL
FNO
7.507
1.619
8.842
8.371
8.957
8.966
13.610
7.219
8.282
DeepONet
1.008
3.187
1.507
2.570
13.584
2.845
1.846
1.517
2.089
Heat
OFormer
11.142
6.87
12.291
11.981
12.106
11.568
11.862
11.408
11.317
Unet
5.510
1.034
7.738
6.147
12.880
6.240
4.944
3.968
5.373
FNO
1.456
1.766
2,406
2.221
2.048
2.013
2.298
2.477
1.407
DeepONet
9.600
9.558
9.955
9.581
9.654
9.616
9.558
9.555
9.563
Adv
OFormer
8.749
8.23
9.923
9.974
9.750
10.013
7.558
9.187
8.775
Unet
1.952
2.563
2.746
2.486
2.614
3.162
2.291
1.84
1.802
FNO 0.195
0.454
0.685
0.688
0.742
0.307
1.734
0.627
0.19
DeepONet
0.189
0.122
0.307
0.150
0.240
0.150
0.154
0.144
0.164
Burgers
OFormer
0.528
0.163
6.134
6.928
2.400
1.558
0.387
0.448
0.585
Unet
0.342
0.054
0.333
0.218
0.333
0.147
0.202
0.240
0.340
19

---

Published in Transactions on Machine Learning Research (09/2024)
A.2 Auto-reete rugo-regressive Experiments
Table 6: Models are pretrained on 9216 combined 2D Heat, Advection, and Burgers samples and finetuned 500 samples for each PDE. Normalized 1.2 errors (x10-1) calculated 256 validation samples and averaged over five seeds. The lowest errors are given in dark grey and second lowest errors are given in light grey.
(a)
Autoregressive Pretraining Results.
PDE
Model
None
Transfer
Binary
TimeSort
Jigsaw
CoollicienDDerivatiiveaaaaa
PICL
FNO
2.730
5.507
3.888
4.704
3.878
4.838
2.336
3.984
2.584
DeepONet
2.374
2.429
2.618
2.592
3.046
2.589
2.352
2.32
2.289
Heat
OFormer
4.410
2.694
24.214
11.398
5.498
6.982
3.277
3.274
4.418
Unet
3.357
2.378
3.286
2.768
2.586
2.406
2.464
2.39
3.019
FNO
30.890
28.586
30.669
29.888
31.571
29.571
29.875
29.584
30.676
DeepONet
27.971
27.453
28.058
28.778
28.637
28.307
28.861
28.387
28.016
Adv
OFormer
30.102
30.784
29.677
29.674
29.293
29.299
30.467
30.774
28.71 719
Unet
30.640
30.832
30.17
30.058
30.992
31.027
30.579
30.310
30.355
FNO
5.104
5.696
6.362
6.640
5.373
6.310
5.168
5.466
5.031
DeepONet
5.101
4.298
5.706
5.341
5.638
5.702
5.024
5.190
5.167
Burgers
OFormer
7.734
6.41
25.731
18.102
10.157
10.026
7.059
7.101
8.293
Unet
5.440
4.912
6.339
5.763
5.277
5.312
5,280
5.197
5.656
FNO
5.884
6.708
8.626
11.211
7.293
7.276
9.245
6.393
6.086
DeepONet
6.461
6.274
8.954
6.607
7.118
6.659
6.526
6.587
6.427
NS
OFormer
10.300
10.466
18.433
16.358
13.065
12.592
10.996
11.750
12.380
Unet
5.854
5.745
6.461
6.110
6.233
6.011
5.902
5.813
6.285
(b) Out-of titto--Dsstrbuttion Autoregressive Pretraining Results.
PDE
Model
None
Transfer
Binary
TimeSort
Jigsaw
DooffidentDDeriiattiveeeasss
PICL
FNO
25.418
22.835
26.810
41.450
26.346
26.400
25.968
23.83
25.623
DeepONet
3.062
3.914
3.981
4.208
15.344
5.101
4.291
29.002
3.166
Heat
OFormer
35.888
17.203
33.862
34.864
35.389
33.472
32.803
34.528
32.857
Unet
16.998
3.965
20.390
17.475
21.936
20.397
13.866
9.126
12.961
FNO
24.733
23.405
24. 24.166
24.970
24.579
24.426
24.922
24.710
24.730
DeepONet
26,480
26.179
27.219
26.755
26.832
26.864
26.214
26.288
26.406
Adv
OFormer
25.210
25.344
29.968
28.877
25.325
25.658
25.174
24.269
25.014
Unet
24.627
24.976
25.053
25.062
24.733
24 14.688
24.630
24.749
24.558
FNO
1.443
3.830
2.688
4.630
2.778
2.646
1.498
3.376
1.424
DeepONet
1.357
1.408
1.677
1.552
2.150
1.642
1.133
1.629
1.247
Burgers
OFormer
3.181
1.706
21.411
8.346
3.942
5.360
2.141
2.198
3.032
Unet
1.594
1.491
1.930
1.706
1.661
1.517
1.514
1.498
1.804
20

---

Published in Transactions on Machine Learning Research (09/2024)
B Comparison of Data Augmentations
B.1 Fixed Future Experiments
Table 7: Models are pretrained 9216 combined 2D Heat, Advection, and Burgers samples and finetuned 500 samples for each PDE. Each baseline (e.g, None, PICL, Transfer) is followed by its variants with different augmentations (e.g, Noise, P-Shift, T-Scale). Normalized L.2 errors (x10-1) are calculated on 256 validation samples and averaged over five seeds. The lowest errors are given in dark grey and second lowest errors are given in light grey.
(a) Fixed Future Pretraining Results.
PDE
Model
None
Noise
Shift
Scale
PICL
P-Noise
P-Shift
P-Scale
Transfer
T-Noise
T-Shift
T-Scale
FNO
0.246 0.183
0.182
0.179
0.182
0.389
0.332
0.169
0.418
0.407
0.445
0.449
DeepONet
0.670
0.638
0.637
0.661
0.493
0.455
0.457
0.562
0.449
0.407
0.401
0.434
Heat
OFormer
0.763
0.482
0.535
0.650
0.901
0.425
0.467
0.918
0.510
0.340
0.338
0.497
Unet
0.147
0.163
0.162
0.092
0.145
0.176
0.099
0.163
0.130
0.137
0.133
0.081
FNO
3.539
3.509
3.549
3.339
3.631
3.722
3.704
3.464
2.017
1.703
1.679
1.695
DeepONet
9.906
9.807
9.792
9.794
9.713
9.594
9.599
9.737
9.606
9.565
9.557
9.560
Adv
OFormer
9.643
9.508
9.566
8.661
9.690
9.689
9.591
9.559
9.296
8.818
8.716
6.948
Unet
3.979
3.893
3.906
3.457
3.802
3.320
3.142
3,473
2.401
2.211
2.192
1.930
FNO
0.698
0,583
0.584
0.721
0,640
0.643
0.627
0.761
0,645
0.643
0.636
0.831
DeepONet
4.092
3.840
3.841
3.791
4.101
3.929
3.937
4.018
3.956
3.774
3.770
3.854
Burgers
OFormer
2.408
2.158
1.752
1.556
Unet
0.950
0.844
0.989
0.727
FNO
2.112
2.122
2.124
2.301
2.232
2.574
2.534
2.335
2. 2.428
2.581
2.602
2.747
DeepONet
5.560
5.543
5.544
5.552
5.514
5.316
5.320
5.517
5.492
5.313
5.313
5.333
NS
OFormer
3.744
3.415
3.399
3.569
4.056
3.696
3.694
3.922
3.962
3.629
3.609
3.605
Unet
2.279
2.247
2.238
2.309
2.262
2.290
2.131
2.318
2.678
2.603
2.573
2.520
(b)
Out-of-Distribution Fixed Future Pretraining Results.
PDE
Model
None
Noise
Shift
Scale
PICL
P-Noise
P-Shift
P-Scale
Transfer
T-Noise
T-Shift
T-Scale
FNO
7.721
7.818
7.805
6.409
8.282
8.670
8.807
7.328
2.431
2.842
2.726
2,806
DecpONet
1.019
1.056
1.073
1.027
2,089
2.023
2.091
1.228
1.150
1.368
1.384
1.250
Heat
OForee
11.14
11.58
11.61
11.78
11.32
11.34
11.42
11.51
11.46
11.00
11.18
9.956
Unet
5.504
5.128
5.203
3.299
5.373
5.070
3.894
3.261
2.491
2.139
2.040
1.783
PNO
1.457
1.374
1.372
1.407
1.317
1.583
1.601
1.604
1.537
DeepONet
9.599
9.581
9.585
9.584
9.563
9.513
9.511
9.497
9.523
9.511
9.493
9.492
Adv
OFormer
8.750
8.214
8.251
7.179
8.775
8.833
8.682
8.472
8.997
9.082
8.968
7.807
Unet
1.955
1.967
2.014
1.623
1.802
1.597
1.582
1.281
2.117
2.137
2.165
1.757
FNO
0.214
0.173
0.170
0.148
0.190
0.261
0.265
0.115
0.400
0.425
0.455
0.592
DeepONet
0.188
0.145
0.146
0.182
0.164
0.126
0.131
0.192
0.126
0.122
0.122
0.122
Burgers
OFormer
0.526
0.288
0.326
0.536
0.585
0.289
0.343
0.780
0.362
0.201
0.201
0.368
Unet
0.341
0.322
0.323
0.313
0.340
0.281
0.216
0.303
0.284
0.229
0.223
0.252
21

---

Published in Transactions on Machine Learning Research (09/2024)
B.2 Auto-regen sugo-segressive Results
Table 8: Models are pretrained on 9216 combined 2D Heat, Advection, and Burgers samples and finetuned
500 samples for each PDE. Each baseline (e.g, None, PICI, Transfer) is followed by its variants with different augmentations (e.g, Noise, P-Shift, T-Scale). Normalized L2 errors (x10-l) are calculated on 256 validation samples and averaged over five seeds. The lowest errors are given in dark grey and second lowest
errors are given in light grey.
(a)
Autoregressive Pretraining Results.
PDE
Model
None
Noise
Shift
Scale
PICT
P-Noise
P-Shift. P-Scale
Transfer
T-Noise
T-Shift
T-Scale
FNO
2.731
2.345
2.343
2.405
2.584
2.908
2.830
2.428
4.338
5.398
5.174
5.215
DeepONet
2.184
2.410
1.993
2.158
2.289
2.120
2.166
2.371
2.347
2.297
2.118
2.378
Heat
OFormer
4.540
3.778
3.854
4.074
4.418
3.658
3.433
4.674
3.190
2.935
3.010
3.218
Unet
3.301
2.695
2.776
2.387
3.019
2.339
2.349
2.494
2.301
2.340
2.276
2.494
FNO
30.89
30.88
31.02
30.67
30.68
29.94
30.22
30.11
28.59
28.33
28.65
28.33
DeepONet
27.98
28.66
28.04
28.01
28.02
28.13
27.840
28.20
28.14
28.04
27.55
27 81.8
Adv
OFormer
30.05
31.04
30.29
30.31
28.72
28.89
28.84
29.83
30.18
30.23
30.61
30.11
Unet
29.94
30.56
30.12
30.55
30.36
30.37
29.72
30.64
30.45
30.17
30.64
30.75
PNO
5.103
5.127
5.076
5.302
5.031
5.098
5.142
5.311
5.874
6,428
5.988
DeepONet
4.884
5.080
4.740
4.577
5.167
4.675
4.666
4.671
4.757
4.484
4.239
4.653
Burgers
OFormer
7.754
7.157
7.058
7.519
8,293
7.267
7.044
8.183
6.867
6,491
6.538
Unet
5.503
5.334
5.335
5.202
5.656
5.206
5.229
4.980
4.957
4.976
4.935
PNO
5.884
6.129
6.092
6.032
6.086
6.068
6.078
5.973
6.724
6.874
6.931
7.553
DeepONet
6.461
6.397
6.404
6.400
6.427
6.393
6.376
6.434
6.310
6,241
6.218
6.276
NS
OFormer
10.30
8.96
9.011
9.012
12.38
10.60
10.25
11.16
10.78
8,937
9.037
9.183
Unet
5.854
5.653
5.799
5.851
6.285
5,880
5.883
5.676
5.756
5.686
5.705
5.697
(b)
Out-of-Distribatio Antoregressive Pretraining Results.
PDE
Model
None
Noise
Shift
Scale
PICL
P-Noise
P-Shift
P-Scale
Transfer
T-Noise
T-Shift
T-Sccale
FNO
25.42
25.48
25.50
23.92
25.62
26.01
25.84
23.69
23.89
25.775
25.82
25.73
DeepONet
3.267
3.182
3.9
3.197
3.166
3.400
3.232
3.119
3.704
4.027
3.922
3.445
Heat
OFormer
35.67
36.71
36.93
35.54
32.86
32.47
32.53
35,80
25.34
26.61
25.97
23.23
Unet
16.95
15.37
15.66
11.66
12.96
9.51
10.43
8.279
4.649
4.563
4.687
4.172
FNO
24.73
25.06
25.05
24.92
24.73
25.22
25.16
24.96
23.69
23.39
23. 15
23.316
DeepONet
26.48
26.03
25.81
25.64
26.41
25.96
25.61
25.64
26.23
25.74
25.98
25.37
Adv
OPormer
25.24
25.27
25.36
25.08
25.01
24.60
24.69
25.35
25.26
25.36
25.23
24.99
Unet
24.48
24.96
24.70
24.90
24.56
24.93
24.58
24.81
25.14
24.46
24.88
25.09
FNO
1.442
1.414
1.443
1.469
1.424
2.121
1.787
1.597
4.185
4.580
4.505
5.062
DeepONet
1.212
1.335
1,167
1.229
1.247
1.261
1.291
1.411
1.278
1.467
1.460
Burgers
OFormer
3.135
2.688
2.920
3.032
2.437
2.216
3.404
2.044
1.941
1.908
Unet
1.632
1.463
1.377
22

---

Published in Transactions on Machine Learning Research (09/2024)
C
Comparison of Downstream Dataset Size
FNO errors across Heat, Advection, and Burgers datasels.
(b) DeepONet errors across Heat, Advection, and Burgers datasets.
OFormer across Heat, Advection, and Burgers datasets. Ius
(d) Unet errors across Heat, Advection, and Burgers datasets.
Figure Fixed Future Scaling Behavior: For each model, a specific PDE/distribution displayed. Within each graph, the error of various pretraining strategies at different sample sizes displayed. Validation errors are averaged over 5 seeds, and error bars denote I std-dev. Derivative errors are omitted outliers.
23

---

Published in Transactions on Machine Learning Research (09/2024)
FNO errors across Heat, Advection, and Burgers datasets.
(b) DeepONet errors across Heat, Advection, and Burgers datasets. DCCrkeedcoionet
(c) OFormer errors across Heat, Advection, and Burgers datasets.
(d) Unet errors across Heat, Advection, and Burgers datasets.
Figure 4: Auto-regressive Scaling Behavior: For each model, specific PDE/distributiin is displayed. Within each graph, the performance of various pretraining strategies at different sample sizes is displayed. Validation errors are averaged over 5 sceds, and error bars denote 1 std-dev.
24

---

Published in Transactions on Machine Learning Research (09/2024)
D
Implementation Details
D.1 Dataset Details
We generate data according to the equations outlined in 4.1 We provide additional details here:
Pretraining During pretraining, 9216 total samples are generated, with 3072 samples of the 2D Heat, Advection, and Burgers equations respectively. The samples are generated with a resolution of (nz, R2, myy) = (32, 64, 64) or (ne, Rz,, Ny) = (32,32, (2,322 32) on the domain (z. [-1, 1]2 from 0 1 = 2; the discretization depends the downstream resolution of the data. We sample equation coefficients from defined pretraining distribution. Heat, Advection, and Burgers equation samples are generated with a finiteddifeereness scheme; a first-order central difference is used to discretize the diffusive term, a frrst-order upwinding scheme is used to discretize the nonlinear convection term, and time is discrctized with a forward Euler scheme. In addition, the advection equation is solved with its analytical solution.
Training/ Training/Finetunng FFnneuunin During
we generate equations using a procedure similar to pre- training and sample coefficients either in the pretraining distribution or from a disjoint distribution to test generalization to unseen coefficients. For fine-tuning on the Navier-Stokes equations, we higher resolution of (mt, Rez, nyy) = (32, 64, 64), otherwise experiments are with resolu- tion of (R,, ny)) (32,32, 32). We generate 1024 samples for the Heat, Advection, Burgers, and Navier-Stokes equations to Irain with. An additional 1024 ant--of-distrbbution samples for the Heat, Advection, and Burgers equations is also generated. Additionally, the Burgers equation, initial con- ditions are unchanged to evaluate fine-tuning to a reference problem undergoing different dynamics, such as in design optimization problems (Cheng et al. 2024).
Validation Validation samples are generated similarly to fine-tuning samples, also with equation coefficients sampled from either the pretraining or disjoint distribution. We generate 256 samples for the Heat, Advection, Burgers, and Davier-Stokes equations.
D.2 Model Details
Table 9: Hyperparameters for architectures used.
(a) FNO
(b) DeepONet
(c) OFormer
(d) Unet
Parameter
Value
Paramcter
Value
Parameter
Value
Parameter
Value
Modes
4
Branch Size
256
Hidden dim
32
Hidden channels
16
Width
48
Trunk Size
256
Heads
# Blocks
8
# Layers
4
Branch Layers
Encoder depth
Dim Scaling
(1,2,4)
Params
300k
Trunk Layers
3
Docoder depth
# Params
1M
Activation
SiLU
Latent channels
32
Params
250k
Params
70k
We implement modern FNO and Unet architectures according to Gupta & Brandstetter (022 Furthermore, we implement. DeepONet architectures according to DeepXDE (La al.] (2021b), and use the original implementation for OFormer (Li et al. 2023a). The hyperparameters used for the models are described in Table 9 For FNO, we use the standard spectral convolution and residual layers, well as additional input channels for the y, and time coordinates and multiple timesteps. The Unet implementation uses multiple ConvNext blocks followed by an pysample/dowssapp) block at each layer. Additionally, an attention block is used al the middle layer between upsampling and downsampling. When implementing OFormer, an attention module is used to aggregate information from the inputs as well as output query coordinates; this is aggregated into latent encoding which is propagated with an MIP and decoded at future timesteps. Lastly, DeepONet uses an implementation that bundles multiple timestops into both the branch and trunk net MLPs, these predictions combined to ouput multiple future timesteps.
25

---

Published in Transactions on Machine Learning Research (09/2022)
20
10
-0
-20
-20
-00
20
42
(a) Embeddings after Binory pre- training. Labels are defined as 0 for shuffled or 1 for original samples.
(b) Emnbeddings after TimeSort (c) Embeddings after Jigsaw pre- pretraining. Labels are scaled be- training. Labels scaled between tween 0 and 1 for 24 classes.
0 and 1 for 1000 classes.
Figure 5: t-SNE Embeddings of CV Pretrained Models: We display latent embeddings after pretrain- ing models on Binary, TimeSort. or Jigsaw objectives. We see that models learn to sort patl/clssify shuffled samples well and can visualize the relative difficulties of the proposed protraining strategies.
D.3 Pretraining Details
During pretraining, different strategies require different implementations and hyperparameters. A considera- tion is that many models need linear head during pretraining to project model outputs to the classification or regression dimension. Since models will be used for physics prediction, their outputs will be in the shape of the solution field, rather than cross entropy probabilities or regressed values. We use wiigt CNN projector to downsample and flatten model outputs to the desired dimension. Models are generally trained for 200 epochs with a batch size of 32 using Adam with a learning rate of 1e-3, weight decay of 1e-6, and OneCycle scheduler for five seeds. A note is that for binary pretraining, the task is much easier and therefore the pretraining can be done with 100 epochs.
Binary:
Binary pretraining implemented by shuffling a sample time and randomly choosing shuffled or original input with corresponding labels of or 1 to be used for classification. We use CNN head to project model outputs to single logit for binary cross-entropy loss. Within this framework there are few design decisions. The difficulty of the task can be modulated by the Hamming distance between the shuffled sample and the original sample. For example, if the shuffled sample is not changed much (e.g. only two frames are swapped), the difference between a sorted and shuflled sample is small and thus more challenging to distinguish. We can leverage this to gradually decrease the Hamming distance of shuffled samples to incrementally increase the difficulty of the task over pretraining. Empirically, this does not make large difference during training so we choose to omit this curriculum learning for simplicity.
An additional consideration is the probability of sampling shuffled or sorted sample. In theory, there many more shuffled samples than sorted samples (i.c. more labels with 0 vs. 1); therefore, it may be beneficial to sample more shuffled samples and use a weighted binary cross-entropy loss. In practice this does not significantly affect training, so we uniformly sample sorted or shuffled samples. A final consideration is that PDE solutions generally do not exhibit large changes in time, therefore, we patchily the time dimension when shuffling to create larger changes in shuffled patches. In general, models are able to learn distinguish between shuffled and original inputs very well, and we display t-SNE embeddings of pretrained FNO model on validation set of shuffled and unshuffled samples in Figure 5a
TiineSor/SSaaceSe Sorting along a single dimension is implemented by patchifying the solution field along the desired dimension and shuffling these patches. This is done to create more distinct dif- ferences in the shuffled solution, with the patch size controlling the number of permutations of the
26

---

Published in Transactions on Machine Learning Research (09/2024)
shuffled sequence. The permutation number affects the difficulty of the sorting task, with large permutation numbers being more difficult since each permutation represents a different class. To mitigate this, we set the patch size to ensure a sequence length of 4 when shufiling, resulting in 4! = 24 classes or permutations of the solution field. The CNN projection head is modified accord- ingly to output 24 logits for cross-entropy loss. In general, spatial sorting does not work well nor does training converge, so we omit this from the results; aliasing effects or periodic boundary conditions can make some spatially shufiled samples extremely similar or identical sorted samples. However, temporal sorting tends to work well, and we display t-SNE embeddings of pretrained FNO model on validation set of temporally shuffled samples in Figure 5b
Jigsaw:
Jigsaw is implemented similarly to other sorting frameworks, however due to sorting along multiple axes the number of possible shuffled sequences quickly increases. We mitigate this by using spatial and temporal patches to ensure a sequence length of 8 when shuffled, resulting in 8! = 40320 possible permutations. This is still a large number of classes for a lask, therefore we deterministically choose 1000 samples with the largest Hamming distance between the shuffled sequence and original sequence. Contrary to the binary case, shuffled samples with larger Hamming distances are more challenging due to needing to sort more patches. The CNN projection head is modified accordingly to output 1000 logits for cross-entropy loss. In general, jigsaw sorting tends to be more challenging, however, models can still display reasonable performance; display t-SNE embeddings of pretrained FNO model on a validation set of jigsaw shuffled samples in Figure 5c
Coefficient: Coefficient regression is implemented by extracting coefficient values from the PDE metadata. The CNN projection head is then modified to output the corresponding number of logits for an MSE loss.
Derivative: We generate labels for derivative regression through taking spatial and time derivatives (w,, Uz My, Maa, } of the PDE solution field using FinDiff (Baer 2018). This introduces an ad- ditional design consideration as the label has more values than the input. We modify the CNN projection head to upsample model outputs after convolution to the desired dimension and apply an MSE loss.
Masked:
Masked inputs are generated by splitting inputs into spatial and temporal patches, and sclecting random subset of these to be masked. In our experiments, we choose to mask 75% of patches. Masked patches are replaced with learnable mask token, and the full input is passed to the model to reconstruct the original solution field. Since the output shape is the same the downstream target, projection head is not strictly needed, but we still include a CNN projection head and apply an MSF loss. This follows previous work; models learn transferable latent features by abstracting reconstruction-speeiit behavior to a decoder (Chen et al. 2020 He et al. 2021
PICL: PICT. uses the Generalized Contrastive Loss function Leyva- Vallina et al.
2023 given in equation
Pir 2
Vaes max(T dphysice (u,, uu,,), 0y)
LOOL uz) = dehyysios (u,, u,)² (u,,
2
When working with multiple data sets simultaneously, a vector of operator coefficients con- structed as O. The similarity between systems is given by magnitude-awarr cosine similarity:
/ 1,.-9,1
th.s (0,, 02) = max (10 II] The distance between samples is calculated in two parts for a given time t: daystem (u4, u.) = a - u/, and dupiite = (())( - (u,)), where Ge is our parameterized model, and F(.) our numerical update. dupdate is anchored to daystem to account for mode collapse, giving us the loss function:
dphysics (u,, u2) = ||daystem (u4, M2) dupdate (u;, uz))l² T is a hyperpa- rameter that defines a margin, above which samples are considered to be from different classes. For pretraining, we construct the operator coefficient vector as 0 = [|rgrege M, |/ II]
D.4 Data Augmentation Details
We implement three data augmentations evaluate their effects on model performance: noise, shift, and scale.
27

---

Published in Transactions on Machine Learning Research (09/2024)
Noise Gaussian noise is added to data samples and targets through sampling Gaussian at zero mean and prescribed variance: Moise = X NO, 7). Empirically, we set the variance to 10-1; when noise levels are too high, model performance can significantly deteriorate.
Shift Using the Fourier shift theorem, samples can be shifted in space and resampled in the spectral domain [Brandstetter et all 2022). Shifting PDE solutions in space preserves physics, since the PDEs considered in this work are invariant across space. Mathemalicale or looking up the Lie groups for the 2D Advection, Heat, and Burgers equations, for which there are many, and noting that the solutions can be shifted along the X or Y axes (Ibragimov 1993). We uniformly sample the magnitude of the shift between [-0.5, 0.5).
this can be verified by deriving
Scale Scaling PDE solutions respects physics for the Heat and Advection equations, but not the Burgers equation. However, we still choose to include this augmentation to evaluate the effect of physically inconsistent augmentations; practice, scaling PDE solutions still improves model performance. The implementation done by multiplying PDE solutions by a constant, which we uniformly sample between 0.5..
D.5 Fine-tuning Details
During fine-tuning, models trained until convergence for fixed-future auto-regressive prediction and re- peated for five seeds. In fixed-future prediction, models are given the solution field at t [0, 8) and the larget is at = 32. For auto-vegressive prediction, models are given the solution field at = [0, 8) and the target is at t = [8, 16). After this prediction, the models use their own output to predict the next step = [16, 24) until the time horizon of 32. To stabilize anto-regrossive rollout, we implement temporal bundling and the pushforward trick (Brandstetter et al. 2023). Losses are calculated using relative L2 norm (Li et al. 2021]; validation losses averaged across batch size and accumtlated over timesteps or, in the case of fixed-future prediction, at only one timestep. For experiments with different fine-tuning sample sizes, samples are randomly chosen from 1024 possible samples to reach the desired number of samples for each seed. These fine-tuning datasets are independently resampled from the same pool for each seed, and larger datasets are not required to be strict supersets of the smaller datasets, which could increase the vari- ance of results. We use an Adamn optimizer with a learning rate of le-3, with weight decay of 1e-6, and a CosineAnnealing scheduler. All experiments are run on NVIDIA GeForce RTX 2080Ti GPU.
28

---

Published in Transactions on Machine Learning Research (09/2024)
E
Auxiliary Results
During our experimentation, there were different pretraining strategies that did not improve model perfor- mance over a baseline of no pretraining. These are omitted from consideration in the results and analysis, however, for completeness, preliminary results that informed this decision are reported here.
E.1 Sorting Spatially Shuffled Sequences
In this section, we compare SpaceSort with baseline without pretraining. Both models are fine-tuned on 500 samples that are within the pretraining distribution from the respective PDEs on a fixed-future prediction objective. We hypothesize that spatially shuffing PDEs may confuse models during pretraining since it destroys important information, as physical systems need to be continuous space.
Table 10: Models are pretrained on 9216 combined 2D Hoat, Advection, and Burgers samples and finctuned on 500 samples for each PDE. L2 errors (x10-3) are calculated on 256 validation samples and averaged over five seeds.
Table 11: Fixed-Future Pretraining Results with SpaceSort
PDE
Model
None
SpaceSort
FNO
0.75
2.61
DeepONet
2.13
2.39
Heat
OFormer
2.38
10.78
Unet
0.46
1.51
FNO
11.04
19.17
DeepONet
30.95
31.17
Adv
OFormer
30.13
31.26
Unet
12.37
20.34
FNO
2.20
3.83
DeepONet
14.04
14.63
Burgers
OFormer
5.99
15.70
Unet
3.21
4.53
E.2 Lie Contrastive Self-Supe Learning
In this section, compare VICRegBardes et al. 2022 against bascline FNO training for fixed-future prediction task. We use values of y 1, = 25, X 25, matching values from the original paper. In this case, we use the first 8 frames to predict the system state at frame 32.
Table 12: Models are pretrained on 9216 combined 2D Heat, Advection, and Burgers samples and finetuned on 100 samples from the Heat equation. L. errores (x10*) are calculated on 256 validation samples and averaged over five seeds.
Table 13: Fixod-Future Pretraining Results with VICReg
Dataset
FNO
FNO VICReg
Heat
1.45
8.03
29


Reviews:
[{"review_id": "eeVlFwIXHg", "reviewer": "Reviewer_PjtC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "However, there are some essentially weak points. The content is too distracting so that the person who read it is hard to capture the key points. Although the authors do so many experiments, I still cannot get the point. If the author can re-written the content and show the experiment clearly, i.e. where is strong by adding pretraining.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is a well-written experimental paper. Although there are many highlights such as comparing various pretraining methods without optimizing architecture choices to characterize pretraining dynamics on different models and datasets, pretraining performance can be further improved by using data augmentations, and pretraining is additionally beneficial when fine-tuning in scarce data regimes or when generalizing to downstream data similar to the pretraining distribution.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "The authors should rewrite the manuscript to make the key point clearer.", "broader_impact_concerns": "I do not think that this work has important influences.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JbFjzZ5Wcs", "reviewer": "Reviewer_qE5d", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nWell-explained methods, including comparison to generic image regression tasks, good illustrative diagrams.\n\n* Useful literature review, including many references in this domain of which I at least was not aware\n* interesting auxiliary question about the interactions of data augmentation and transfer learningn\n\n## Weaknesses\n\n* The empirical results are not supported by much theory\n* This paper is at the intersection of two fields, transfer learning and neural operator learning, and it faces a common challenge for such multi-field papers: how to satisfy readers from both fields. I am sympathetic to the fact that this is difficult to do, but I nonetheless think the authors need to do make some changes to be effective here.\n\n* Unclear reproducibility and poor positioning in relation to existing literature on benchmarking PDE tasks; \n  The paper does not, it seems, use directly, or extend the existing PDE benchmark datasets \n\n  * [pdearena](https://github.com/pdearena/pdearena)\n  * [PDEBench](https://github.com/pdebench/PDEBench/)\n\n  Maybe these methods are not useful in their case, but the \"shape\" of the data used in the pre-training, covariate-shift and fine tuning does look a lot like the stuff in the standard data sets, which leads to the question of why not use the exist methods rather than introduce new data sets\n\n* The paper does not put its results in the context of pretraining, domain adaptation, transfer-learning  fine-tuning generally, treating the case of neural operator learning as essentially an isolated field. It might be true that fine-tuning and transfer learning have little to add to the special case of neural operators, but this looks like a non-trivial assertion. The main section addressing this relationship seems to be sections 2.1 and 2.4:\n\n  >Many past works consider transferring knowledge between PDE parameters and domains as a form of pre-\n  >training. These works often design specific architectures that are tailored for transferring weights or layers\n  >between tasks. For example, Goswami et al. (2022) design task-specific layers of a DeepONet to be used\n  >with different domains of 2D Darcy Flow and Elasticity problems. Another approach proposed by Tripura &\n  >Chakraborty (2023) is to design different operators that learn specific PDE dynamics and combine these in\n  >a mixture of experts approach, motivated by the observation that PDEs can often be compositions of each\n  >other. To address the issue of transferring between physical domains that can have different numbers of vari-\n  >ables, Rahman et al. (2024) extend positional encodings and self-attention to different codomains/channels\n\n  >Additional past work considers adapting meta-learning (Finn et al., 2017) paradigms from the broader deep\n  >learning community to the PDE domain. Zhang et al. (2023a) consider a direct adaptation of model-agnostic\n  >meta-learning to PDE tasks, while Yin et al. (2021) and Kirchmeyer et al. (2022) apply novel losses and\n  >architectures to maximize shared learning across different tasks. Following in-context learning trends of\n  >transformer models (Dong et al., 2023), Chen et al. (2024) and Yang et al. (2023) explore using in-context\n  >learning to prompt models with PDE solutions to generalize to unseen PDE coefficients\n\n  We've already admitted that the wider field of transfer learning might have something to say about operator learning in particular, so now we must wonder why their results are not used to inform the interesting questions of the paper with regards to data augmention.\n\n  The taxonomy in Table 1 suggests that the community of practitioners who look at both transfer learning and PDE operator learning might be somewhat separate, so this separation probably reflects the conventions of the field.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper functions both as a survey of existing methods of transfer learning for neural PDE autoregressive operator (i.e. forward propagator) learning, and as novel research comparing SOTA methods under an interesting treatment condition.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Critical changes\n\n1. At least mention the wider field of transfer learning and provide a rationale as to why generic transfer learning and data augmentation theory results are not helpful to this particular sub-field.\n2. Do something to address the lack of reproducibility standardisation in the data set. In order of preference I would suggest\n\n   1. Use the existing standard pde datasets ( [pdearena](https://github.com/pdearena/pdearena)/ [PDEBench](https://github.com/pdebench/PDEBench/)) or if they are not suitable, \n   2. submit pull requests to the maintainers so that they are suitable, or if this is not productive,\n   3. publish the code or datasets used in the results so that future works are comparable with this one\n\n## Recommended changes\n\n1. if you could apply any of the existing transfer learning theory to explain the results here that would be valuable.\n1. Use some of the theoretical results from the wider field, about optimal transfer methods etc (adversarial learning! invariant learning)\n\nThere are several taxonomies and communities of transfer learning who might be useful for this: \n\n  * [thuml/A-Roadmap-for-Transfer-Learning](https://github.com/thuml/A-Roadmap-for-Transfer-Learning#domain-adaptation)\n  * [zhaoxin94/awesome-domain-adaptation](https://github.com/zhaoxin94/awesome-domain-adaptation) \n  * [http://transferlearning.xyz](http://transferlearning.xyz/)\n\nA partial listing of recent and highly-cited works in transfer learning is\n\n```\n@article{ImanReview2023,\n  title = {A {{Review}} of {{Deep Transfer Learning}} and {{Recent Advancements}}},\n  author = {Iman, Mohammadreza and Rasheed, Khaled and Arabnia, Hamid R.},\n  year = {2023},\n  month = mar,\n  journal = {Technologies},\n  volume = {11},\n  number = {2},\n  eprint = {2201.09679},\n  pages = {40},\n  doi = {10.3390/technologies11020040},\n  archiveprefix = {arXiv}\n}\n\n@misc{JiangTransferability2022,\n  title = {Transferability in {{Deep Learning}}: {{A Survey}}},\n  shorttitle = {Transferability in {{Deep Learning}}},\n  author = {Jiang, Junguang and Shu, Yang and Wang, Jianmin and Long, Mingsheng},\n  year = {2022},\n  month = jan,\n  number = {arXiv:2201.05867},\n  eprint = {2201.05867},\n  publisher = {arXiv},\n  archiveprefix = {arXiv}\n}\n\n@misc{KaddourCausal2022,\n  title = {Causal {{Machine Learning}}: {{A Survey}} and {{Open Problems}}},\n  shorttitle = {Causal {{Machine Learning}}},\n  author = {Kaddour, Jean and Lynch, Aengus and Liu, Qi and Kusner, Matt J. and Silva, Ricardo},\n  year = {2022},\n  month = jul,\n  number = {arXiv:2206.15475},\n  eprint = {2206.15475},\n  publisher = {arXiv},\n  doi = {10.48550/arXiv.2206.15475},\n  archiveprefix = {arXiv}\n}\n\n@article{LiuDeep2022,\n  title = {Deep {{Unsupervised Domain Adaptation}}: {{A Review}} of {{Recent Advances}} and {{Perspectives}}},\n  shorttitle = {Deep {{Unsupervised Domain Adaptation}}},\n  author = {Liu, Xiaofeng and Yoo, Chaehwa and Xing, Fangxu and Oh, Hyejin and Fakhri, Georges El and Kang, Je-Won and Woo, Jonghye},\n  year = {2022},\n  month = aug,\n  journal = {APSIPA Transactions on Signal and Information Processing},\n  volume = {11},\n  number = {1},\n  publisher = {Now Publishers, Inc.},\n  doi = {10.1561/116.00000192}\n}\n\n@misc{LiuOutOfDistribution2023,\n  title = {Towards {{Out-Of-Distribution Generalization}}: {{A Survey}}},\n  shorttitle = {Towards {{Out-Of-Distribution Generalization}}},\n  author = {Liu, Jiashuo and Shen, Zheyan and He, Yue and Zhang, Xingxuan and Xu, Renzhe and Yu, Han and Cui, Peng},\n  year = {2023},\n  month = jul,\n  number = {arXiv:2108.13624},\n  eprint = {2108.13624},\n  publisher = {arXiv},\n  doi = {10.48550/arXiv.2108.13624},\n  archiveprefix = {arXiv}\n}\n\n@inproceedings{MansourDomain2009,\n  title = {Domain {{Adaptation}}: {{Learning Bounds}} and {{Algorithms}}},\n  shorttitle = {Domain {{Adaptation}}},\n  booktitle = {{{COLT}}},\n  author = {Mansour, Yishay and Mohri, Mehryar and Rostamizadeh, Afshin},\n  year = {2009},\n  month = feb,\n  eprint = {0902.3430},\n  publisher = {arXiv},\n  doi = {10.48550/arXiv.0902.3430},\n  archiveprefix = {arXiv}\n}\n\n@article{NiuDecade2020,\n  title = {A {{Decade Survey}} of {{Transfer Learning}} (2010--2020)},\n  author = {Niu, Shuteng and Liu, Yongxin and Wang, Jian and Song, Houbing},\n  year = {2020},\n  month = oct,\n  journal = {IEEE Transactions on Artificial Intelligence},\n  volume = {1},\n  number = {2},\n  pages = {151--166},\n  doi = {10.1109/TAI.2021.3054609}\n}\n\n@misc{TanSurvey2018,\n  title = {A {{Survey}} on {{Deep Transfer Learning}}},\n  author = {Tan, Chuanqi and Sun, Fuchun and Kong, Tao and Zhang, Wenchang and Yang, Chao and Liu, Chunfang},\n  year = {2018},\n  month = aug,\n  number = {arXiv:1808.01974},\n  eprint = {1808.01974},\n  publisher = {arXiv},\n  doi = {10.48550/arXiv.1808.01974},\n  archiveprefix = {arXiv}\n}\n\n@article{WangGeneralizing2023,\n  title = {Generalizing to {{Unseen Domains}}: {{A Survey}} on {{Domain Generalization}}},\n  shorttitle = {Generalizing to {{Unseen Domains}}},\n  author = {Wang, Jindong and Lan, Cuiling and Liu, Chang and Ouyang, Yidong and Qin, Tao and Lu, Wang and Chen, Yiqiang and Zeng, Wenjun and Yu, Philip S.},\n  year = {2023},\n  month = aug,\n  journal = {IEEE Transactions on Knowledge and Data Engineering},\n  volume = {35},\n  number = {8},\n  pages = {8052--8072},\n  doi = {10.1109/TKDE.2022.3178128}\n}\n\n@book{WangIntroduction2023,\n  title = {Introduction to {{Transfer Learning}}: {{Algorithms}} and {{Practice}}},\n  shorttitle = {Introduction to {{Transfer Learning}}},\n  author = {Wang, Jindong and Chen, Yiqiang},\n  year = {2023},\n  series = {Machine {{Learning}}: {{Foundations}}, {{Methodologies}}, and {{Applications}}},\n  publisher = {Springer Nature},\n  address = {Singapore},\n  doi = {10.1007/978-981-19-7584-4},\n  isbn = {978-981-19758-3-7 978-981-19758-4-4}\n}\n\n@misc{WilsonSurvey2020,\n  title = {A {{Survey}} of {{Unsupervised Deep Domain Adaptation}}},\n  author = {Wilson, Garrett and Cook, Diane J.},\n  year = {2020},\n  month = feb,\n  number = {arXiv:1812.02849},\n  eprint = {1812.02849},\n  publisher = {arXiv},\n  doi = {10.48550/arXiv.1812.02849},\n  archiveprefix = {arXiv}\n}\n\n@book{YangTransfer2020,\n  title = {Transfer {{Learning}}},\n  author = {Yang, Qiang and Zhang, Yu and Dai, Wenyuan and Pan, Sinno Jialin},\n  year = {2020},\n  edition = {1},\n  publisher = {Cambridge University Press},\n  address = {Cambridge},\n  doi = {10.1017/9781139061773},\n  isbn = {978-1-107-01690-3}\n}\n\n@inproceedings{ZhangBridging2019,\n  title = {Bridging {{Theory}} and {{Algorithm}} for {{Domain Adaptation}}},\n  booktitle = {Proceedings of the 36th {{International Conference}} on {{Machine Learning}}},\n  author = {Zhang, Yuchen and Liu, Tianle and Long, Mingsheng and Jordan, Michael},\n  year = {2019},\n  month = may,\n  pages = {7404--7413},\n  publisher = {PMLR}\n}\n\n@article{ZhangTransfer2024,\n  title = {Transfer {{Adaptation Learning}}: {{A Decade Survey}}},\n  shorttitle = {Transfer {{Adaptation Learning}}},\n  author = {Zhang, Lei and Gao, Xinbo},\n  year = {2024},\n  month = jan,\n  journal = {IEEE Transactions on Neural Networks and Learning Systems},\n  volume = {35},\n  number = {1},\n  pages = {23--44},\n  doi = {10.1109/TNNLS.2022.3183326}\n}\n\n@misc{ZhuangComprehensive2020,\n  title = {A {{Comprehensive Survey}} on {{Transfer Learning}}},\n  author = {Zhuang, Fuzhen and Qi, Zhiyuan and Duan, Keyu and Xi, Dongbo and Zhu, Yongchun and Zhu, Hengshu and Xiong, Hui and He, Qing},\n  year = {2020},\n  month = jun,\n  number = {arXiv:1911.02685},\n  eprint = {1911.02685},\n  publisher = {arXiv},\n  doi = {10.48550/arXiv.1911.02685},\n  archiveprefix = {arXiv}\n}\n```", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eU2JJspfQD", "reviewer": "Reviewer_rDWh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper is generally clearly written and includes a good introduction to the problem and discussion of related works\n- Good variety of experimental configurations (target physics, augmentations, pretraining strategies, etc.)\n- The results are well categorized to help the reader understand them despite the varied experimental configuration\n\n**Weaknesses**\n- The paper might benefit from additional detail on model architectures\n- Some details of the experiments could also be more explicitly described\n- A few tests might have benefited from more independently seeded\n  networks in cases with high variance (but understandably there are limits to what is feasible)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper evaluates a range of pretraining and data augmentation\nmethods for neural operators, testing a variety of network\narchitectures against a range of target physics. The results may\nprovide some insight into which pretraining approaches work well for\nPDE tasks, which could help guide future work. The authors include\nsome experiments testing the impact of larger vs smaller fine-tuning\ndatasets and in- vs. out-of-distribution samples in the fine-tuning\nphase vs. pretraining.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "No particularly critical changes in my opinion, mostly some additions\nI believe would add clarity.\n\n1. It would be helpful to have more detail on network architectures in\n   appendix D.2. As a particular example, for the U-net there are a\n   variety of possible structures, and a more complete description\n   would be useful.\n2. Similarly I believe it would also be helpful to provide some\n   details of the coefficient distributions used to generate the\n   datasets in appendix D.1.\n3. A minor question: in appendix D.3 you note that networks are\n   \"generally trained for 200 epochs...\" were there exceptions? If\n   not, it would help to make this a bit more definitive.\n4. Will the code used for the experiments be available? I don't\n   believe it is critical for the review, but would be helpful for\n   readers who want full details of the experiments beyond what can be\n   communicated in the paper.\n5. For the augmentation experiments in Table 3 and appendix B.2 you\n   mention the results are compared against a model without\n   pretraining? In that case, does it reflect the effect of the data\n   augmentation independent of the pretraining step? For example, for\n   Burgers-Unet Table 3b lists a ~10% improvement but based on the\n   table in the appendix, the improvement over plain Transfer appears\n   to be more modest. It does appear that the augmentation provides\n   some further benefit, but this could use some additional\n   discussion.\n6. Another very minor note: in appendix D.4 you note that the scale\n   augmentation does not respect physics for Burgers. You mention this\n   as a possible issue in section 3.1.2. It may be good to briefly\n   note that this does in fact affect one of the systems you consider.\n7. Another question on the construction of the fine-tuning datasets:\n   Looking at appendix D.5 it is a bit clear whether you resampled the\n   smaller fine-tuning sets independently for each independent network\n   training run, or whether the sets were fixed for each size? Were\n   the larger sets chosen as strict supersets of smaller data sets? I\n   think this could do with a bit more clarity since depending on the\n   structure it could add more variance to some of the results.", "broader_impact_concerns": "No concerns that I can identify", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hQVyexeQKE", "reviewer": "Reviewer_yf78", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe paper performed extensive experiments, covering a wide range of models, PDE problems, pretraining and augmentation methods, and downstream problems.\n\n**Weaknesses**\n\nIn my humble opinion, many conclusions are primarily empirical and could benefit from more analysis. Some conclusions might be anticipated and could benefit from better justification of interest. Some claims are not fully supported. Please see examples below.\n\n1. The main conclusion that \"directly adapting computer vision methods to the physics domain generally results in poor performance\" might be anticipated given the different stuctures of the problems. It is unclear whether a more thoughtful adaptation considering the structure of PDE problems would work better. Although negative results are important, the reviewer is not sure whether it is already known to the field that CV methods would not work well in PDE. Also, the motivation for adapting CV methods could be better justified. Why focus on CV, rather than methods in other ares, e.g. natural language processing?\n\n2. The paper could benefit from more insight and theoretical explanations into why some methods work better than others. E.g., why does shift augmentation outperform others?\n\n3. The authors found that transfer learning + shift augmentation performs best in general. However, it is unclear how combination of multiple augmentations will perform, which is most common in machine learning practice.\n\n4. Table 4: The conclusion that \"Pretraining is more beneficial when downstream data is similar to pretraining data.\" is not fully supported. Two out of four methods (OFormer and Unet) showed contraditory results.\n\n5. The conclusions that \"Pretraining is more beneficial when downstream data is scarce\" and \"Pretraining is more beneficial when downstream data is similar to pretraining data\" in Section 5.4 seem to be well-known in machine learning.\n\n6. Some claims are not supported by data. E.g.,\n  \n   \"Empirically, SpaceSort does not perform well, so we omit this strategy from our results.\"\n\n   \"However, these methods did not seem to show significant improvements over no pretraining, as such, the results are omitted from the paper.\"\n\n   Could the authors provide more evidence for these claims, either by referring to previous work or by providing their own results?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors compared pretraining and data augmentation methods for PDE models and attempted to adapt pretraining strategies in computer vision to PDE. From experimental results, the authors draw the following conclusions. First, the best-performing pretraining or augmentation method varies for different network architectures and problems. Second, transfer learning and shift augmentation generally perform best. Third, directly adapting CV methods do not work well. Fourth, pretraining is more effective when finetuning is performed on smaller datasets and similar data distributions.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would suggest the authors consider the following changes to improve paper.\n\n- Elaborate on the motivation for adapting CV pretraining methods to PDE.\n- Explain more clearly why certain results are novel or unexpected, and thus interesting to other researchers. Summarize a few \"best practices\" for researchers in the field looking to pretrain their model.\n- Include more insights and explanations for why some methods perform better than others.\n\n\nPlease see additional questions below.\n\n- \"Furthermore, we consider scaling the PDE solution (Scale), an approach similar to a color distortion, in which the PDE solution values are multiplied by a random constant.\"\n\n  What does \"similar to a color distortion\" mean?\n\n- It is unclear which parts of the Methods are novel and which parts are not. E.g., which augmentations in Section 3.1 have been used for PDE before? Which pretraining strategies in Section 3.2 have been used for PDE before?\n\n- \"This is implemented by patching the solution in space and time, randomly replacing masked patches with a learnable mask token, and regressing the true solution; we refer to this strategy as Masked.\"\n\n  It appears this pretraining strategy only applies to transformer models because of the use of mask tokens. Could the authors explain how a \"mask token\" could be used in a CNN? Also, is this proposed by Zhou & Farimani (2024b)?\n\n- \"Both works require the use of a specific encoder along with the neural operator backbone; to adapt these strategies to our experimental setup we consider directly pretraining the neural operator contrastively with these strategies.\"\n\n  Could the authors explain in more detail what the \"specific encoder\" is? Why is it used in previous works? And how did the authors mitigate the need for this encoder?\n\n- Tables 2 and 3: \nThe authors only showed what the best-performing method is and its performance for each case. It would be more informative to show the performance of all methods, so that the gap between methods is clear.\nAlso, is there any pattern in when each method performs best? Is there any rule of thumbs extracted from these results that can inform model training?", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciated the effort put in this empirical investigation and praised it as it can be very interesting for practitioners looking fo squeezing extra performance out of neural operators. At the same time, they have raised a number of concerns, the major ones being the lack of theory, unclear reproducibility, and missing comparisons to existing literature on benchmarking neural PDE predictors. 

During the rebuttal, the authors addressed well the last two points and managed to flip some of the reviewers, now providing these recommendations: accept, leaning accept and leaning reject. The criticism still remaining is that this work is essentially empirical and has no supporting theory. I agree this is the case, but this cannot be a reason to reject the paper, as its claims and contributions are supported by enough (empirical!) evidence. 

The manuscript could be accepted as is, but I require the authors to make another pass spotting typos and fixing the width of Table 1.
Decision: Accept (poster)