Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning
Authors: Wuyang Chen
Jialin Song
Pu Ren
Shashank Subramanian
Dmitriy Morozov
Michael W. Mahoney
Venue: NeurIPS 2024 poster (2024)
Keywords: scientific machine learning
unsupervised pretraining
neural operators
foundation models
Primary area: machine_learning_for_physical_sciences
TL;DR: We introduce unsupervised pretraining and in-context learning for Scientific Machine Learning, significantly enhancing data efficiency and generalizability of neural operators.

Abstract: Recent years have witnessed the promise of coupling machine learning methods and physical domain-specific insights for solving scientific problems based on partial differential equations (PDEs).  However, being data-intensive, these methods still require a large amount of PDE data. This reintroduces the need for expensive numerical PDE solutions, partially undermining the original goal of avoiding these expensive simulations. In this work, seeking data efficiency, we design unsupervised pretraining for PDE operator learning. To reduce the need for training data with heavy simulation costs, we mine unlabeled PDE data without simulated solutions,
and we pretrain neural operators with physics-inspired reconstruction-based proxy tasks. To improve out-of-distribution performance, we further assist neural operators in flexibly leveraging a similarity-based method that learns in-context examples, without incurring extra training costs or designs. Extensive empirical evaluations on a diverse set of PDEs demonstrate that our method is highly data-efficient, more generalizable, and even outperforms conventional vision-pretrained models. We provide our code at https://github.com/delta-lab-ai/data_efficient_nopt.

Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning
Wuyang Chen' Simon Fraser University
Jialin Song"
Pu Ren
Simon Fraser University
Lawrence Berkeley National Laboratory
Shashank Subramanian Lawrence Berkeley National Laboratory
Dmitriy Morozov
Lawrence Berkeley National Laboratory
Michael W. Mahoney
International Computer Science Institute
Lawrence Berkeley National Laboratory University of California, Berkeley
Abstract
Recent years have witnessed the promise of coupling machine learning methods and physical domain -ppeii- insights for solving scientific problems based on partial differential equations (PDEs), However, being data-intensive, these methods still require a large amount of PDE data. This reintroduces the need for expensive numer- ical PDE solutions, partially undermining the original goal of avoiding these expen- sive simulations. this work, seeking data efficiency, we design unsupervised pre- training for PDE operator learning. To reduce the need for training data with heavy simulation costs, we mine unlabeled PDE data without simulated solutions, and we pretrain neural operators with physics-inspired reconstruction- -hased proxy tasks. To improve diitr--diitiibttioon performance, we further assist neural operators flexibly leveraging similarity- -based method that learns in-context examples, with- out incurring extra training costs or designs. Extensive empirical evaluations on di- verse set of PDEs demonstrate that our method is highly data-efficient, more gener- alizable, and even outperforms conventional vision -rretonneee models. We provide our code at htttt::://// thub. cote ta- lab- ai/data, efficient_mme officieen_nopt.
Introduction
Recent advancements in machine learning methodology have shown promise in solving partial differential equations (PDEs) [24] 67 38
39 255. A significant development in this area the concept of operator learning for PDEs. This approach differs from traditional neural network methods, which are restricted to fixed-dimension input and output, since neural operators focus on learning mappings between function spaces 38
45
44
550
45 44). Like other neural network methods, neural operators are recognized to be universal approximators for any continuous operator [46] (38), enabling them to approximate any physical operator, including solution operators for various parametric PDE families. A solution operator is defined as function that maps physical inputs to output solutions. Previous work has shown that simple settings, neural operators can effectively capture complex, multi-scale dynamic processes 45
46 74 73
However, neural operators tend suffer from problem common other deep networks, namely the need for enormous quantities of data. Limited availability of data is common in science and
38th Conference on Neural Information Processing Systems (NeurIPS 2024).
"Equal contribution.

---

engineering. High-fidelity numerical simulations are computationally costly or even infeasible for many applications [7T] For example, an extreme-scall simulation of magnitude 7.0 earthquake at frequencies up to 10 Hz in San Francisco requires 3600 Summit GPU nodes and 42.7 hour [53
Motivated by this data-efficiency challenge, recent works, particularly in natural language processing (NLP) and computer vision (CV), have focused on unsupervised (or self-supervised) pretraining to reduce the cost of collecting generating labeled data. Such pretrained models have been shown be highly data-efficient in downstream fine-tuning 7
22 g and they can even become few-shot learners
without any downstream data [3. In CV, researchers collect large amounts of natural images without any manual labels, and then pretrain visual encoders with proxy tasks, such as Soise-Contrastive Estimation (NCE) [59], masked reconstruction [28]. rotation and jigsaw prediction [58] 20). In NLP,
people typically pretrain models via next-word prediction or masked tokens B 12
However, unsupervised pretraining is still largely underexplored in Scientific Machine Learning (SciML). Therefore, our core question is: How can we design unsupervised pretraining for operator learning reduce the data simulation costs?
In this work, we resort to unsupervised pretraining for neural operator learn- ing to achieve data efficiency in SciML. We overview our framework in Fig- ure [ First, we define unlabeled data for PDES, which avoids heavy compu- tation costs for simulating PDE solu- tions. We propose two physics-inspired reconstruction-basse
Stage
Unsupervised Pretralning no simulated solution
Pretrain with Physics- Inspired Proxy Tasks
10
ippts (Unlabeled POE Datal
Predicted
Fine-tuning reduced simulation costs
Train with
Stago
Fewer Somples
proxy tasks, and we pretrain neural operators on unla- beled PDE data. We demonstrate that with unsupervised pretraining, our neu- ral operators not only improve counter- parts trained with more simulated data, but also they outperform off-the-shhe the-shelf pretrained checkpoints from other popu- lar domains (such as CV) that are ready for fine-tuning. Then, to further im- prove the data efficiency during out-of- distribution (OOD) inference, we design similarity-base -bsed method that learns in-context examples 3 49 80 881 [47].
Inputs
Predicted Solutions
In-Context Learning
Out-of-Diee Disriiiiibttion Inference no training cost
Stage
Query Inputs
Predicted Solution
Figure 1: Overview of our framework for data-efficies neu- ral operator learning (with our contributions highlighted in red). Stage 1: Unsupervised pretraining only unlabeled PDE data. Stage 2: Fine-tuning with reduced simulation costs of PDE data. Stage 3: Test-time in-context examples can improve the neural operator's 's -itt---diittiuution per- formance, without additional training costs.
This approach introduces zero overhead during training: one just maintains the standard training pipeline, and it can be seamlessly plugged in for OOD infer- ence, without further fine-tuning. In more detail, we summarize our main contributions:
1. We introduce unlabeled PDE data and unsupervised pretraining for data-efficient neural operator learning. We show that method can achieve better performance than models trained with more simulated PDE solutions, or fine-tuned from public checkpoints pretrained on other benchmarks, demonstrating the importance of unsupervised pretraining on comain-speciff PDE data.
2. We propose similarity -ased method to improve the OOD generalization of neural operators, which is flexible and can scale up to large number of unseen in-context examples ("demos").
We provide detailed empirical evaluations on both diverse PDE benchmarks and also several real- world scenarios, demonstrating that we can achieve both strong forward modeling performance and significant savings in PDE simulations.
II this paper, we use the terms "unsupervised pretraining" and "self-supervised pretraining" interchangeably.

---

2 Related Works
2.1 Machine Learning for Scientific Modeling
There has been a long history of using learning-basse based methods to model physical scientific phenom- ena
A representative line of work is so-called physies-ifformed neural networks 69] which try to incorporate physics in neural networks by including the differential form of the PDE as an additional physics loss regularization term. However, this paradigm is confined to specific PDE scenarios (e.g.. fixed PDE coefficients), instead of being more physics-agnostic. Moreover, recent work has highlighted several fundamental "issues" with PINN- based methods [40] 15]
4
42
6
[]
(PINNS)
67
87
19
18
On the other hand, operator learning methods, including Fourier Neural 38 and Deep Operator Network (50, have achieved progress approximating the solution operators of PDEs. Although these data-driven approaches show promise in learning PDE so- lutions, they (like many neural network-baat data. For operator learning, such data usually computationally expensive to simulate [67 2 84). More recently, people have tried to generate synthetic PDE solutions to train SciML models [27 In contrast, our method works unlabeled PDE data. This approach is distinct from the generation of synthetic PDE data, and it could be further combined as a semi coni-supervised learning strategy.
Operators 45
methods) rely on vast quantities of high-fidelity labeled
2.2 Unsupervised Pretraining and Foundation Models
Unsupervised (or self-supervised) pretraining is key method in CV and NLP to achieve meaningful
representations 77]. data-efficient fine-tuning [31], and foundation models [. In CV, contrastive
learning learns meaningful features by distinguishing between similar (positive) and different (nega- tive) samples [59] 76 7 56 29]. Masked Autoencoder (MAE) [28] uses a reconstructive approach where parts of the input are masked and the model learns predict masked parts. In NLP, among the most prominent works are large language models (LLMs) such as GPT G 64 65] and BERT [2 66). which leverage token predictions for pretraining. Similar directions also show progress in SciML. For example, [2] and [54] propose to create augmented views in the solution space via Lie Symmetries; 73 study the scaling behavior of supervised pretraining and OOD generalization, charting directions for foundational models for SciML; (43) target learning astronomical foundation models with cross- modal contrastive learning: and (52) build large task-agnostic models with a broad understanding of common physical behavior to serve as foundation models for SciML.
2.3 In-Context Learning (ICL)
In-context learning (ICI.) is promising paradigm that helps deep networks generalize unseen domains with a few in-context examples. Early works in CV seek to learn feature-level correspon- dence between the target and few "shots," such that models can generalize to open-set unseen objects [1 833. In NLP, people find LLMs are naturally few-shot learners [3], and thus tuning or optimizing prompts becomes extremely important to improve the in-context learning performance of LLMS
85 722 More recently, within SciML, a different operator learning strategy, termed
"in-context operator learning." has been proposed 80 477. During both training and inference, the neural operator is asked to make predictions by explicitly leveraging predefined number of so- so-called "demo" examples (pairs of physical parameters and simulated solutions). This approach provides balance between model generalization and addressing data scarcity in the scenario of OOD testing.
3 Methods
In this section, we introduce our framework (outlined in Figure We propose first to pretrain the model with unsupervised pretraining (Sec. (Sec.B.1). which will contribute to the data efficiency and reduced PDE simulation costs during standard training of neural operators. When move to OOD scenarios during inference, we test our models with in-context examples (Sec. 3.2 to avoid further fine-tuning costs.
3.1 Unsupervised Pretraining
The core idea in unsupervised (or self-supervised) pretraining is to train neural network with properly designed proxy tasks. These proxy tasks do not require labeled data, but they are designed
3

---

to be highly related to the supervised learning objectives of interest. While popular in CV and NLP, unsupervised pretraining on unlabeled data has never been explored in PDE operator learning in SciMI. This is due to two unresolved questions: 1) What kinds of unlabeled data can we use to train neural operators? 2) How to design proxy tasks for PDE data? We address these questions below.
3.1.1 Unlabeled PDE Data
How Define Unlabeled PDE Data? In general, when a neural operator is trained on PDE datasets [45] (73, it learns to map inputs (physical parameters, coordinates, forcing functions, initial conditions, etc.) to PDE solutions. Therefore, given set of PDE data (collected via simulations or observations), its unlabeled version is defined as the one without PDE solutions. Our unlabeled PDE data is broader concept of related inputs modeling PDE systems. Let us consider the second-order linear differential equation as general example. It is formulated as
Bis (
Σ bi(x)uz, + +dxxu= f(x),
sJ=1
where x E R represents physical space that varies different systems (e-g. n for 2D time- dependent PDEs); the coefficients (or physical parameters) 0ij; b; are known from the physical process; u is the target solution; and f denotes an external forcing function [57] We can consider two situations where solutions are unavailable:
Time-independent equations: following 45 73 our unlabeled PDE data include physical parame- ters (aıj. b;, c), forcing functions (f), and coordinates (grids of the discrete physical space).
Time-dependent equations (c.g., forecasting problems 77, 52 ): without simulating the temporal dynamics, our unlabeled PDE data only include initial snapshot uo(r) that defines PDE systems. Note that collecting snapshots with temporal dynamics in large-scale scenes is more complex than capturing individual snapshots. For example, weather forecasting (30 and smoke dispersion [4] require continuous monitoring and multiple sensors, whereas single measurements are simpler and less resource-intensive Long-term data collection often involves extensive networks and processing, unlike one-time measurements.
For concrete examples of different PDEs and unlabeled data that we will study, see Appendix A There are two main reasons for pretraining only on unlabeled PDE data, as discussed below.
Cheap Generation of Unlabeled PDE Data. One critical reason that leads expensive computa- tional costs when collecting PDE data is the time marching scheme [23 22 in numerical simulation. However, only generating unlabeled PDE data and snapshots without temporal dynamics will be much cheaper than simulating solutions (Table D. making our unsupervised pretraining highly feasible in practice. Our pretraining strategy will be very data-efficient, and can avoid the heavy computational cost of simulating complex time-dependent equations for massive high-fidelity labeled solutions.
Benefits of Pretraining on Unlabeled PDE Data. Beyond the cheap generation, pretraining on unlabeled PDE data has the following benefits. First, regularization against overfitting. Unsuper- vised pretraining can strongly regularize the model towards better generalization. Second, faster convergence. Pretraining on unlabeled PDE data will provide neural operators with domain-adapted initializations and can accelerate training speed. Third, meaningful representations. Pretraining on unlabeled PDE data can help models extract useful representations for subsequent operator learning. We defer our results and experimental details in Figure in Sec. 4.1
3.1.2 Proxy Tasks
To illustrate our general approach of constructing proxy tasks, we choose two variants of reconstruc- tion as our core proxy tasks. In particular, we will input unlabeled PDE data to our neural operators, and after a decoder network, we will force the output to be close to the input. We consider two perturbation variants (or augmented views). They are inspired by real-world settings when people collect scientific data, and they are important invariances we need to introduce to SciML models.
Masked Autoencoder. Masked autoencoders (MAEs) have been shown to be scalable self- supervised learners 288. The method is conceptually simple: remove a portion of the input data, and

---

learn to predict the removed content. These methods enable training in NLP and CV of generalizable models containing over one hundred billion parameters T2 MAE for scientific modeling.
3 Here, we investigate the potential of
Unsupervised Pretraining
Motivation:
PDE dynamics are invariant sparse sensing of the full field scientific data. It is very common scientific data need to be collected from sparse sensors, and that people need to reconstruct generate data for domains without sensors. We enforce our model learn sensor invariance via random mask- ing, and we extract the invariant features over distorted views of the same unlabeled PDE data. The invariance to sparse sensing of scientific data will facilitate the robust- ness of the representations from MAE.
[6 32 that
Most
Reconstruction
Super-resalitio
Inputs (Unlabaled PDE
Augmentted
Predicted Inputs
Inputs
Brur
Figure 2: Overview: unsupervised pretraining via MAE and super-resolution. During pre-training, in the input unlabeled PDE data, random subset (e-g., 70%) of spatial locations are masked, followed by a Gaussian blur. After the encoder and decoder, the full set of imput is required to be reconstructed.
Therefore, we consider MAE as proxy task. Specifically, our MAE is straightforward autoencoding approach that reconstructs the original signal, given its partial observation. Like all autoencoders, our approach has an encoder that maps the observed signal to a latent representation, and decoder that reconstructs the original signal from the latent representation. We randomly sample masks according to a certain masking ratio (i.e., the ratio of removed areas), and the values of masked areas of unlabeled PDE data are set to zero. Our loss function computes the mean squared error (MSE) between the reconstructed and original input. We compute the loss only on masked areas; if no mask applied (i.e., a vanilla autoencoder), the MSE loss will be applied to all spatial areas.
Super-resolution (SR) techniques have emerged as powerful tools for enhancing data resolution, improving the overall quality and fidelity of data representation, and retrieving fine-scale structures. SR a task that involves recovering fine-scale data from corresponding coarse- grained data. It is also popular task PDE learning 9 68). which is train SciML models to preserve the inherent physical properties of scientific data.
Super-resolution.
Numerical solutions of PDEs are expected to exhibit invariance to filtering blur or different resolutions of inputs (37 79]. For instance, in turbulence simulations, the traditional numerical methods always fail to model the expected physical phenomenon with low-resolution meshes due to substantial numerical errors. SR has emerged as a powerful tool for subgrid modeling of PDE dynamics, especially helping capture the critical patterns of turbulence [5] 37]. Given specific input distribution, after fitting the SR objective, neural operators are expected to preserve the inherent physical properties and exhibit invariance filtering blur.
Motivation:
Therefore, we introduce SR as another proxy task. Our objective shares the same motivation as recent SciML works for SR [68]. Specifically, we enforce the model to learn invariant features of unlabeled PDE data that are immune to resolution and blur. Blurry snapshots often occur when the resolution too low to accurately represent the details in the original content. To do so, we apply Gaussian filter to blur the unlabeled PDE data, and the autocncoder will reconstruct the high resolutsontion input with fine- aineggriindd details. Instead of applying fixed blurring, randomly sample the variance of the Gaussian filter from a certain range as augmentations.
3.1.3 PDEs
After pretraining on unlabeled PDE data, we fine-tune neural operators on simulated solutions of PDEs. We study two time-independent PDEs (Poisson, Helmholtz) and two time-dependent PDEs IreactionDiiffssoon Navier-Stokes) We include details of these PDEs in Appendix
3.1.4 Model Architectures
We consider two popular architectures for fair comparisons with previous works. These are encoder- decoder architectures designed to reconstruct the original input given partial observations, where the encoder maps observed unlabeled PDE data to a latent space, and the decoder reconstructs the original conditions. We include visualizations of these architectures in Appendix D
5

---

Fourier Neural Operator. Fourier Neural Operator (FNO) targets learning PDE data in the Fourier space. The original model backbone (encoder) employs Fourier transform and learns lower Fourier modes with linear transforms. The FNO backbone outputs features back to the spatial domain (i.e., the embeddings are on the pixel level). We refer readers the original paper for details [45 46).
Pretraining: We build the decoder to be identical to the encoder (except for the input/output dimension). Unlabeled PDE data randomly masked at the pixel level.
Fine-tuning: After the pretraining, we discard the decoder, and we follow the original design to append two fully-connected layers (with ReLU activations) to predict final spatial-wise solutions.
Transformer. Transformers, which mainly employ self-attention and linear transform blocks, have shown promise in both NLP and CV 778 13
Different from FNO, which directly operates on grids, transformers tokenize and group grids into patches, i.e., each tokenized patch embeds local neighborhood of subgrids. We follow the 3D transformer architecture of Video-MAE 77 52 Our encoder embeds patches by a linear projection with added positional embeddings (just as a standard ViT), and it then processes the resulting set via series of Transformer blocks. For the transformer, the unlabeled PDE data are randomly masked at the patch level.
Pretraining: For the transformer encoder, we only apply it on the subset of tokens that are visible (i.e., unmasked patches), and masked patches are removed. This allows us to train encoders efficiently. The input to the MAF decoder is the full set of tokens consisting of (i) encoded visible patches, and (ii) the mask token. The mask token is a shared and learned vector that indicates the presence of a missing patch to be predicted, We add positional embeddings to all tokens in this full set. Without this, mask tokens would have no information about their location in the input. Following 228 77]. we adopt an asymmetric design where the decoder is more lightweight (shallower and narrower) than the encoder.
Fine-tuning: After the pretraining, the decoder is preserved during fine-tuning, since we need to reconstruct the tokenized patches back to the input.
3.2 Similarity-based Mining of In-Context Examples
Dut-of-distribution (OOD) generalization is a critical technical challenge, not only in SciML but also across multiple domains in Al for science (84). To improve the OOD generalizability of neural operators and to reduce the extra effort of downstream fine-tuning, the following inference paradigm has been proposed: given a query input, the model is also provided with few supporting examples (dubbed "demos"), together with their ground -ruth solutions, to make the final prediction. This approach enables the "open-set" generalization of the model make predictions on unseen samples.
48 61 people developed delicate architectures to find correspondence between the input and the supporting examples. The purpose of this extra architecture/training design is twofold: first, find similarities between the target input and supporting examples; and second, aggregate labels of supporting examples for the final predictions. Recent ICL works [80 81 47 on learning PDE data also adopt this strategy, with transformers and cross-attention layers.
Originally, in the literature on few-shot learning 82 83 55 70
35
find
However, the ICL (in-context learning) of LLMs (large language models) enables a different strategy. The pretraining still standard and simple (next/masked token prediction), without additional training costs. During inference, LLMs can ruto-regressively take any number of few-shot examples, finding similarities between tokens in few-shot examples and those in the target query (via self-attention), and then generate responses by aggregating embeddings of tokens in few-shot examples. This ICL strategy used in LLM is highly scalable and saining-efficiees
Motivated by this, we propose to leverage in-context examples via two steps (Algorithm D.
Similarity by Prediction. We find spatial-wise and temporal-wist similar demos by calculating their distance in the output space. That means, for two input locations over the spatial and temporal domains, if we find their outputs of the trained neural operator similar, then we treat them as similar samples. Following [0 81], we assume demos share the same distribution of physical parameters with the query.
Aggregation. For each spatial-temporal location of the query, after finding its similar samples in demos, we aggregate and average their solutions as the prediction.

---

Algorithm 1: Pscudocode of Similarity-based Mining of In-Context Examples.
Data resolution: x-axis (W), y-axis (H), output temporal steps (T), output channel dimensions for the solution (Couc).
Input: Query input (x)). Paired unlabeled PDE data (X) and solutions (Y E XXXHXXXXXXxoo as J
demos. Trained Neural Operator Model M.. TopK (k) demo solutions to aggregate. Shape: H X W x T X Cout
ÿ = =M(()
Y =M(X)
Shape: J x H W T X Cout > Query-Demm Demo Distance. Shape:
8 j.reshape(-1; Caue) y reshapee1, -1, Cout) W T (J H W. T) x Cout
8 absolute sum -1)
Spatial-wise and temporal wise selection of demos similar to query. index argsor( i,, k]
i,;
Siet = take along along_dim(( reshape -1, Cout), ), index) > Spatial-wise -iiee and temporal -iie aggregation of
solutions from similar demos. Shape: H x W x T x Cout x k.
Return: Diet meen((1)
4 Empirical Results
To illustrate the benefits of our approach, we perform empirical evaluations both PDE benchmarks and real-world observations. Most importantly, our unsupervised pretraining (on unlabeled PDE data, followed by fine-tuning) outperforms neural operators trained from scratch, while requiring fewer PDE simulations (Sec. 4.1 and Sec. 4.2) Moreover, our in-context examples can help the model generalize better to OOD cases (Sec. 4.3 In our experiments, trained models three times with different random seeds for statistical significance. For more experimental details, see Appendix B We also include ablation studies about pretraining hyperparameters in Appendix For visualizations of our pretraining, see Appendix K
(a)
Poisson (FNO)
(b)
Helmholtz (FNO)
(c)
Reaction-Diffie (FNO. PDE-Bench)
unsupervised
1:0
Relative f2 Error
random lntt
unsuperv
random intt
22
-2x10'3
azxing
-2x10'0
0.0
432
unsupervised
1-6x1915
+- random Inft
035
T0"
30
10"
101
167
10*
Number of Training Samples
Number of Training Samples
Number Training Samples
2D Incompressible Navier-Stokes
2D Incompressible Navier-Stokes
2D incompressible Navier-ttokes
(d)
(FNO-3D, PINO Dataset)
(o)
(Video-MAE, PDE-Bench)
(f)
(Video-MAE, PDE-Bench)
unsupervised
aropeeieed
0.07
Relative Ea Errar
random Hit..
rendonink.
vist precal (50
0.06
0.2
0.05
unsupervised
random
-0CTO saviige
vision pretralned (SSv2)
-2010"2
3.00
0.03
099
8.62
0.00
005
10
of Training Samples
Number Training Samples
Rollout
Figure 3: Pretraining neural operators on unlabeled PDE data improves its performance and data efficiency on Poisson (a), Helmholtz (b). Reaction-Diffusion (c), and Navier-Stokes (d and with relative errors at different unrolled steps shown on f). "random init.": models are trained from scratch with random initialization. "vision pretrained (SSv2)": fine-tuning from the publicly available checkpoint for Video-MAE (pretrained on computer vision dataset SSV2 [21] for video understanding). Savings of the number of simulated PDE data (when "random init." achieves the best test error) are shown in red.
7

---

4.1 Unsupervised Pretraining Enables Data-Efficient Operator Learning
Data Efficiency. We first demonstrate that, by leveraging unsupervised pretraining, neural operators can achieve improved errors with less simulated data. In Figure [3] on each PDE, compared with directly training from scratch ('random init."), pretraining on unlabeled PDE data can help neural operators achieve better performance, which can further help reduce the amount of simulated data. Specifically, in our experiments, when we target achieving the best test error of the baseline ("random init."), our method can save 5 x 102 ~ 8 x 10 simulated solutions across diverse PDE systems.
Among all PDEs, we find that Helmholtz (Figure (b)) the most challenging. Both two curves failed to improve the error until we increased the number of simulated data points to over 1024. Meanwhile, the generalization gaps remain high (Figure 12 (b)), indicating low training errors. We suspect that learning on the Helmholtz equation may be exhibiting the grokking issue 63 [6), where the network quickly memorizes the training data, but the improvement in generalizabiltty is delayed.
Unsupervised Pretraining Outperforms Off-the-Shelf Pretrained Checkpoints. Pretraining on unlabeled PDE data is not the only way to save simulation costs. As pretraining is widely adopted in CV. pretrained checkpoints on vision data become publicly available and are ready for fine-tuning, We choose to compare with Video-MAE (77 for state-of-the-an video understanding pretrained on SSV2 121]. As shown in (e), rision-pretrained Video-MAE can only outperform the random initialization with high volumes of simulated data, while its performance suffers when fine-tuned with limited simulations. In contrast, our unsupervised pretraining on unlabeled PDE data can save a significant amount of simulated data.
As errors during testing may quickly accumulate with further timestamps, we also report results with more unrolled steps. We use checkpoints trained with the largest amount of simulated data from above for this study. As shown in Figure (f). at each rollout step. our unsupervised pretraining achieves much better performance. Similarly, vision-pretrained Video-MAE is eventually outperformed by the random initialization at the long rollout step.
Benefits of Pretraining Unlabeled PDE Data. Pretraining on unlabeled PDE data is beneficial beyond achieving better performance with reduced simulations. First, when training on extremely low volumes of data, neural operators tend to overfit, resulting in poor generalization. In Figure Heft, pretraining on unlabeled PDE data can reduce the generalization gap (testing error - training error). We further show that better generalization gaps persist across all PDEs studied (see Figure [2). Second, pretraining on unlabeled PDE data can lead to faster convergence during fine-tuning. Unlike standard random initializations from Gaussian distributions, pretraining on unlabeled PDE data will provide neural operators with domain-adapted initializations and facilitate a much faster convergence rate, shown in Figure m Third, unsupervised pretraining can also help models extract useful representations for subsequent operator learning. From Figure rright, we find that even with pre-extracted features (i.c., fixed encoder and only fine-tuned decoder), our neural operators can still outperform the baselines where both the encoder and decoder are updated during fine-tuning.
2D incompressible Navier-Stokes (Video- MAE, PDE-Bench)
2D Incompressible Navier-Stokes
2D Incompressible Savier-Stokes (Video MME, PDE-Bench)
Generalization Cap (Test -rain))
Relative fa Error (Training)
(Video-Mae MAE, PDE-Bench)
unaupersiied
0.20
urspeveal
0.000
pretra ned (5Sv2)
Relative E, EErorr
vidon presaieed 100ve)
Relattve& Error
vici peet (56x2)
unaupendeed tencoder
10-1
0.00-
0.25
0:30
0.33
6000
202
Number of Tra ning Samples
10
30'
100
250
Epoch
302
400
560
307
Number Training Samples
10 30' 31P
Figure 4: Benefits of our unsupervised pretraining. Reduced overfitting (left): our method consis- tently leads to smaller generalization gaps (test eтror training error) across all PDEs we studied (Fig. 12.. Faster convergence (middle): our unsupervised pretraining can accelerate model conver- gence than both random initialization and vision- pretraine checkpoint. Meaningful representations (right): fine-tuning Video-MAE with fixed encoder (pretrained unlabeled PDE data, red line) can extract meaningful features and outperform the baseline and the vision-pretrained model (both encoder and decoder are updated during fine-tuning).
'Training curves when the number of training samples of 14760.
8

---

4.2 More Comprehensive Experiments on Real-World Data
We now move broader range of benchmarks. We will study real-world and noisy data instead of toy datasets, providing even more comprehensive experiments. These benchmarks are widely studied in previous works 660 62 52].
Datasets. We brief the background, with more details in Appendix D and visualizations in Fig. 15
ECMWF Reanalysis v5 (ERA5) 330 is a public extensive dataset, which delivers hourly data on multiple atmospheric variables spanning from 1979 to today. ERA5 represents a type of atmospheric reanalysis dataset [4 integrating observations from variety of measurement sources with numerical models through data assimilation [3 Essentially, it reconstructs the most accurate estimation of the Earth's atmospheric conditions over time. This dataset has been extensively utilized in prior SciML studies [60] 68]. We focus on forecasting the important and challenging temperature atmospheric variable.
ScalarFlow [4] is reconstruction of real-world smoke plumes. It assembles the first large-scale dataset of realistic turbulent flows. The availability of large, volumetric data set opens up wide range of applications, including re-simulations, novel visualization, and metric evaluations. The dataset contains 104 real-world smoke flow density reconstructions. Each reconstruction is captured from five different viewpoints for 150 temporal frames spanning 2.5 seconds.
Airfoil 75 is large-scale dataset that contains the pressure and velocity simulations of the flow around real airfoils. This dataset has 53880 samples for training and 90 samples for testing. Each sample contains 6 channels. The 3 input channels include the binary spatial mask of the airfoil and the initial velocity of the freestream condition in the x and directions. The output channels include the pressure and the and y velocity components from the simulation at the steady state.
Model and Training. For time-dependent FRA5 (Sec. I..]] and ScalarFlow (Sec. datasets, we adopt the same VideoMAF. architecture 777 used in Sec.[4.] We use 15 consecutive temporal snapshots to forecast the next time step. We train VideoMAE with Adam, with other hyperparameters the same as in Table TableB "N.S. (PDEBench)". For the ineeiindependet steady steady-state Airfoil dataset (Sec. L.3) we adopt the 2D-FNO architecture. We train 2D-FNO with Adam, with other hyperparameters the same in Table 3 column "Poisson".
Results. As shown in Figure 5 compared with directly training operators from scratch ("random init."), pretraining on unlabeled data (2D snapshots of ERAS/ScalarFlow without temporal dynamics, or freestream velocities of Airfoil) can help neural operators achieve better performance on both temperature/flow forecasting and predictions of the steady-state pressure and velocity around airfoils.
ERA5 (Video-MAE)
ScalarFlow Video-MAE)
Airfoil (FNO)
unsupervised random Int.
unsupervised - random Init.
unsupervised random Ini:
Relative Ea Error
0.16
012
0.50
0.44
012
085
101
10"
10
105
10F
122
10
107
101
10
Number of Training Samples
Number of Training Samples
Number of Training Samples
Figure 5: For real-world scientific problems, pretraining neural operators on unlabeled PDE data improves its performance and data efficiency. We study VideoMAE [77] pretrained with unlabeled snapshots (no temporal dynamics), and then fine-tune across different numbers of temporal snapshots on ERA5 (left) and ScalarFlow (middle). We also pretrain 2D-FNO 45) on freestream velocities and fine-tune on time-independent steady-state airflow pressure and velocities (right). "random init.": models are trained from scratch with random initialization.
4.3 In- In-Contex Examples Enable Data-Efficient OOD Generalization
We now move to OOD settings, where models will be tested on PDE data simulated with physi- cal parameters unseen during fine-tuning/traiiing. We include how to simulate OOD samples in

---

Appendix B.2
Neural operators suffer from poor OOD generalization [73 80 Traditionally, im- provements heavily depend on further fine-tuning on simulated data, which requires extra simulation and training costs. We study the benefits of leveraging test-time in-context examples. As shown in Figure when we flexibly scale up the number of demos, we can keep improving FNO's OOD generalization on diverse PDEs. We follow [1] that demos are randomly sampled from the same distribution used to generate the OOD test set. When the number of demos is O, we have the baseline in the OOD setting. Notably, we introduce zero training overhead: we keep the standard training pipeline, and our mining of in-context examples can be seamlessly plugged during OOD inference.
We further provide baseline, which uses features extracted by the backbone of the neural operator htigh-dimensions features before the final output layer) to find similar samples. As we can see, this baseline is worse than our method (both performance and confidence), indicating that the final output of the neural operator can more accurately indicate true similar samples.
See Appendix M further discussions about the benefits of leveraging in-context examples, and Appendix N for visualizations.
2D Incompressible Navier-Stokes
Poisson (FNO)
Helmholtz (FNO)
(FNO-3D, PINO Dataset)
Baseline iSimilanity
Baickbone Feature)
Basetine Isim Bix thone Featune)
1.015
Baseline (Smilarity by Bac Teature
Relative fy Error
f
FND Outputi
iSimilarity
1.8
f
Durs (Sirilat
PND Outpul)
1.000
0.9/
1.4
0.500
PNO Output)
Ours (Similarity by
0.475
1.0
0.450
0.425
0
10
15
20
25
10
Number Demos
Number of Demos
Number of Demes
Figure 6: In-context examples for OOD testing. Our method (blue) improves both & errors and confidence as we increase the number of demos. "Ours (Similarity by FNO Output)": we leverage the output (prediction) of neural operators to find similar samples. "Baseline (Similarity by Backbone Feature)": the baseline uses features extracted by the backbone of the neural operator (high-dimensional features before the final output layer) to find similar samples.
5 Conclusion
In this work, we focus on improving the data efficiency of solving partial differential equations (PDEs) using deep learning, with particular emphasis on unsupervised pretraining and in-context learning (ICL) methods. Our key contributions include introducing unsupervised pretraining for operator learning and a flexible ICL approach that enhances out- out-of-distie of-distribuuton (OOD) generalization without increasing training costs. Through extensive evaluations, we demonstrate that our method is not only more data-efficient, but it also achieves greater generalizability compared to existing approaches. By improving the data efficiency of neural operators for solving PDEs, our approach can significantly reduce the computational costs and energy demands of high-fidelity numerical PDE simulations. Additionally, by making advanced PDE solutions more accessible through efficient pretraining, our method has the potential to accelerate scientific and engineering progress across various fields, ultimately benefiting society. We hope our work will inspire the scientific machine learning (SciML) community further address the high simulation costs and limited OOD generalization of neural operators, contributing to advancements that support both scientific innovation and environmental sustainability.
6 Limitations
Current limitations of our work: 1) We could design more physics-inspired proxy tasks and data augmentation methods for scientific data; 2) We could study more PDE systems in our unsupervised pretraining and in-context learning: 3) We could consider more different neural operator architectures. We expect that addressing these limitations will lead to broader impacts in future works.
10

---

References
[1] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint
2021.
[2] Johannes Brandstetter, Max Welling, and Daniel E Worrall. Lie point symmetry data aug- mentation for neural pde solvers. In International Conference on Machine Learning, pages 2241-2256. PMLR, 2022.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901,, 2020.
[4] Steven L Brunton, Bernd R Noack, and Petros Koumoutsakos. Machine learning for fluid mechanics. Annual Review of Fluid Mechanics, 52:477-508, 2020.
[5] Tianping Chen and Hong Chen. Approximation capability to functions of several variables, nonlinear functionals, and operators by radial basis function neural networks. IEEE Transactions on Neural Networks, 6(4):904 910, 1995.
[6] Tianping Chen and Hong Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE transactions on neural networks, 6(4):911-917, 1995.
[7] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[8] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint MXXv:2003.04297 2020.
[9] Wai Tong Chung, Bassem Akoush, Pushan Sharma, Alex Tamkin, Ki Sung Jung, Jacqueline Chen, Jack Guo, Davy Brouzet, Mohsen Talei, Bruno Savard, al. Turbulence in focus: Benchmarking scaling behavior of 3d volumetric super-resolution with blastnet 2.0 data. In seirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[10] Aaron Defazio and Konstantin Mishchenko. Learning-rate--ree learning by d-adaptation. arXiv preprint arXiv:2301.07773 2023.
[11] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 255. Icee, 2009.
[12] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: deep bidirectional transformers for language understanding. arXiv preprint arXiv: arXiv:1810.0m 1810.0488, 2018.
Pre-training of
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint
arXiv:2010.11929 2020.
[14] Marie-Lena Eckert, Kiwon Um, and Nils Thuerey. Scalarflow: large-scale volumetric data set of real-world scalar transport flows for computer animation and machine learning. ACM Transactions on Graphics (TOG), 38(6):1-16, 2019.
[15] C. Edwards. Neural networks learn to speed up simulations. Communications of the ACM, 65(5):27-29) 2022.
[16] N Benjamin Erichson, Lionel Mathelin, Zhewei Yao, Steven I. Brunton, Michael W Mahoney, and Nathan Kutz. Shallow neural networks for fluid flow reconstruction with limited sensors. Proceedings of the Royal Society A, 476622338) 2020.

---

[17] Li Fei-Fei, Robert Fergus, and Pietro Perona. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence, 28(4):594- 611, 2006.
[18] Han Gao, Luning Sun, and Jian-Xun Wang. Phygeonet: Physics-informed geometry- andatiiv convolutional neural networks for solving parameterized steady-state pdes irregular domain. Journal of Computational Physics, 428: 10079, 2021.
[19] Nicholas Geneva and Nicholas Zabaras. Modeling the dynamics of pde systems with physics- constrained deep auto-regressive networks. Journal of Computational Physics, 403: 103:1090 10906, 2020.
[20] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1803.07788 2018.
[21] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag. et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pages 5842- 5850, 2017.
[22] E. Hairer, C. Lubich, and G. Wanner. Geometric Numerical Integration: Algorithms for Orlinury Differential Equations (Springer Series in Computational Mathematics), volume 31. Springer, 2006.
[23] E Hairer, S P Nørsett, and G Wanner. Solving ordinary differential equations I. nonstiff problemss volume 29. Springer, 1987.
[24] Jiequn Han, Aruuf Jentzen, and Weinan E. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, :55(3)):5505- 8510, 2018.
[25] D. Hansen, D. C. Maddix, S. Alizadeh, G. Gupta, and M. W. Mahoney. Learning physical models that can respect conservation laws. Physica D: Nonlinear Phenomena, 437:13 2024.
[26] Zhongkai Hao, Chang Su, Songming Liu, Julius Berner, Chengyang Ying, Hang Su, Anima Anandkumar, Jian Song, and Jun Zhu Dpot: Auto-regressive denoising operator transformer for scale pde pre-training. arXiv preprint arXiv:2403.03542. 2024.
[27] Erisa Hasani and Rachel A Ward. Generating synthetic data for neural operators. arXiv preprint 2rXiv:2401.02398. 2024.
[28] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[29] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[30] Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz- Sabater, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, et al. The cra5 global reanalysis. Quarterly Journal the Royal Meteorological Society, 199933))
2020.
[31] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for nip. In International Conference on Machine Learning, pages 2790-2799. PMLR, 2019.
[32] Balaji Jayaraman and SM Abdullah AI Mamun. On data-driven sparse sensing and linear estimation of fluid flows. Sensors, 20(13):375,, 2020.
[33] Eugenia Kalnay. Atmospheric modeling, data assimilation and predictability. Cambridge university press, 2003.
12

---

[34] Eugenia Kalnay, Masao Kanamitsu, Robert Kistler, William Collins, Dennis Deaven, Lev Gandin, Mark Iredell, Suranjana Saha, Glenn White, John Woollen, et The ncep/ncar 40-year reanalysis project. In Renewable energy, pages Voll 1oll_146-Voll_ 194. Routledge, 2018.
[35] Dahyun Kang and Minsu Cho. Integrative few-shot learning for classification and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[36] Gene A Klaasen and William C Troy, Stationary wave solutions of a system of equations derived from the nitzuo equations. SIAM Journal on Applied Mathematics, 1400):96-110, 1984.
[37] Dmitrii Kochkov, Jamie A Smith, Ayya Alieva, Qing Wang, Michael P Brenner, and Stephan Hoyer. Machine National Academy of Sciences, 1 118021):22101788111 2021.
computational fluid dynamics. Proceedings of the
[38] Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to PDEs. Journal of Machine Learning Research, 24(89):1-97 2023,
[39] A. Krishnapriyan, A. F. Queiruga, N. B. Erichson, and M. W. Mahoney. Learning continuous models for continuous physics. Communicutions Physics, 6:319, 2023.
[40] Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems, 34:26548 265660, 2021.
[41] Isaac E Lagaris, Aristidis Likas, and Dimitrios I Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions neural networks, 95):9877 1000, 1998.
[42] Isaac E Lagaris, Aristidis C Likas, and Dimitris G Papagcorgiou. Neural-network methods for boundary value problems with irregular boundaries. IEEE Transactions on Neural Networks, 11(5):004110449 2000.
[43] Francois Lanusse, Liam Parker, Siavash Golkar, Miles Cranmer, Alberto Bietti, Michael Eick- enberg, Geraud Krawezik, Michael McCabe, Ruben Ohana, Mariel Pettee, Astroclip: Cross-modal pre-training for astronomical foundation models. arXiv preprint arXiv:2310.03024, 2023.
[44] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhattacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. Advances in Neural Information Processing Systems, 33:6755-6766. 2020.
[45] Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021.
[46] Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-informed neural operator for learning partial differential equations. arXiv preprint arXiv:2111.03794 2021.
[47] Jerry Weihong Liu, N Benjamin Erichson, Kush Bhatia, Michael W Mahoney, and Christopher Re. Does in-context operator learning generalize to domain-shifted settings? In The Symbiosis of Deep Learning and Differential Equations III, 2023.
[48] Jie Liu, Yanqi Bao, Guo-Sen Xie, Huan Xiong, Jan-Jakob Sonke, and Efstratios Gavves. Dynamic prototype convolution network for few-shot semantic segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11553-11562,
2022.
13

---

[49] Robert L Logan IV, Ivana Balažević, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. Cutting down on prompts and parameters: Simple few-shot learning with language models. arXiv preprini arXiv:2 106. 13353, 2021.
[50] Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprini arXiv: 1910/9999........ 2019.
[51] Romit Maulik, Omer San, Adil Rasheed, and Prakash Vedula. Subgrid modelling for two- dimensional turbulence using neural networks. Journal of Fluid Mechanics, 858:122-144, 2019.
[52] Michael McCabe, Bruno Régaldo-Saint Blancard, Liam Holden Parker, Ruben Ohana, Miles Cranmer, Alberto Bietti, Michael Eickenberg, Siavash Golkar, Geraud Krawezik, Francois Lanusse, et al. Multiple physics pretraining for physical surrogate models. arXiv preprini 1rX1::2310.02994, 2023.
[53] David McCallen, Anders Petersson, Arthur Rodgers, Arben Pitarka, Mamun Miah, Floriana Petrone, Bjorn Sjogreen, Norman Abrahamson, and Houjun Tang. Eqsim-a multidisciplinary framework for fault-to-struuctree earthquake simulations on exascale computers part i: Computa- tional models and workflow. Earthquake Spectra, 37(2):707-735, 2021.
[54] Grégoire Mialon, Quentin Garrido, Hannah Lawrence, Danyal Rehman, Yann LeCun, and Bobak T Kiani. supe-vupe learning with lie symmetries for partial differential equations. arXiv preprint arXiv:2307.05.4 307.0432, 2023.
[55] Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmen- tation. In Proceedings of the IEEE/CVF international conference computer vision, pages 1941-6952, 2021.
[56] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant rep- resentations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6707-6717, 2020.
[57] AK Nandakumaran and PS Datti. Partial differential equations: classical theory with modern touch. Cambridge University Press, 2020.
[58] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. European conference on computer vision, pages 69-84. Springer, 2016.
[59] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807 0007700378.... 2018
[60] Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay. Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, et al. Fourcastnet: global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv: 1214, 2022.
[61] Bohao Peng, Zhuotao Tian, Xiaoyang Wu, Chengyao Wang, Shu Liu, Jingyong Su, and Jiaya Jia. Hierarchical dense correlation distillation for few-shot segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23641-23651, 2023.
[62] Michael Poli, Stefano Massaroli, Federico Berto, Jinkyoo Park, Tri Dao, Christopher Ré, and Stefano Ermon. Transform once: Efficient operator learning in frequency domain. Advances in Neural Information Processing Systems, 35:7947-7959, 2022.
[63] Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02117 2022.
(64] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training, OpenAl, 2018.
14

---

[65] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, 1lya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
[66] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-00 2020.
[67] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-infeee informed neural networks:
A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
[68] Pu Ren, N Benjamin Erichson, Shashank Subramaniam, Omer San, Zarija Lukic, and Michael W Mahoney. Superbench: A super-resolution benchmark dataset for scientific machine learning. arXiv preprini arXiv:2306. /4070, 2023.
[69] Pu Ren, Chengping Rao, Yang Liu, Jian-Xun Wang, and Hao Sun. Phyernet: Physics-informed convolutional-
network for solving spatiotemporal pdes. Computer Methods in Applied Mechanics and Engineering. 389:1 14399, 2022.
[70] Xinyu Shi, Dong Wei, Yu Zhang, Donghuan Lu, Munan Ning, Jiashun Chen, Kai Ma, and Yefeng Zheng. Dense gross-query-and-suppee -un-osppoortttt attention weighted mask aggregation for few- shot segmentation. In European Conference on Computer Vision, pages 151-168. Springer, 2022.
[71] Justin Sirignano, Jonathan F MacArt, and Jonathan B Freund. Dpm: A deep learning pde augmentation method with application to large-eddy simulation. Journal of Computational Physics, 423:109811, 2020.
[72] Alessandro Sordoni, Xingdi Yuan, Mlre-Alexandr Côté, Matheus Pereira, Adam Trischler, Ziang Xiao, Arian Hosseini, Friederike Niedtner, and Nicolas Le Roux. Deep language networks: Joint prompt training of stacked llms using variational inference. arXiv preprint arXiv:2306. 12509, 2023.
[73] Shashank Subramanian, Peter Harrington, Kurt Keutzer, Wahid Bhimji, Dmitriy Morozov, Michael W Mahoney, and Amir Gholami. Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior. arXiv preprint arXiv:2306.0025: 00258 2023.
[74] Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems,
2022.
[75] Nils Thuerey, Konstantin WeiBenow, Lukas Prantl, and Xiangyu Hu. Deep learning methods for reynolds-averaged navier-stokes simulations of airfoil flows. AIAA Journal, 58(1):25-33) 2020.
[76] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Com- puter Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt XI 16, pages 776-794. Springer, 2020.
[77] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for sulf-supervised video pre-training. Advances in neural information processing systems,
10093, 2022.
[78] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[79] Ricardo Vinuesa and Steven I. Brunton. Enhancing computational fluid dynamics with machine learning. Nature Computational Science, 2(6):358-366 2022.
[80] Liu Yang, Siting Liu, Tingwei Meng, and Stanley J Osher. In-context operator learning with data prompts for differential equation problems. Proceedings of the National Academy of Sciences, 120039): 2023.
15

---

[81] Liu Yang, Tingwei Meng, Siting Liu, and Stanley J Osher. Prompting in-context operator learning with sensor data, equations, and natural language. arXiv preprint arXiv:2308.05061 2023.
[82] Xiaolin Zhang, Yunchao Wei, Guoliang Kang, Yi Yang, and Thomas Huang. Self-produced guidance for weakly-supervised object localization, In Proceedings of the European conference on computer vision (ECCV), pages 597-613, 2018.
[83] Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas S Huang. Sg-one: Similarity guidance network for one-shot semantic segmentation. IEEE transactions on cybernetics, 50(9):3855- 3865, 2020.
[84] Xuan Zhang, Limei Wang, Jacob Helwig, Youzhi Luo, Cong Fu, Yaochen Xie, Meng Liu, Yuchao Lin, Zhao Xu, Keqiang Yan, et al. Artificial intelligence for science in quantum, atomistic, and continuum systems. arXiv preprint arXiv:2307.0844 08423. 2023.
[85] Yongehao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910 2022.
[86] Xuckai Zhu, Yao Fu, Bowen Zhou, and Zhouhan Lin. Critical data size of language models from grokking perspective. arXiv preprint arXiv:2401/04663 2024.
[87] Yinhao Zhu, Nicholas Zabaras, -Stelio Koutsourelakis, and Paris Perdikaris. Physics- constrained deep learning for high-dimensional surrogate modeling and uncertainty quantifica- tion without labeled data. Journal of Computational Physics, 394:56-8 81, 2019.
16

---

A PDEs
We consider two time-independent PDEs (Poisson, Helmholtz) and two time-dependent PDEs (Reaction-Diffusion Navier-Stokes) as described below. We also describe types of unlabeled data for per-PDE examples. For a general definition of unlabeled PDE data, please refer to Section 3.1.1
1. Poisson: We consider two-dimensional (2D) elliptic PDE that arises in various physical situations, with periodic boundary conditions within a spatial domain f = [0,1]² 1]2:
div K Vuu== f
(1)
where u(x) represents the solution, fa) acts as the source (e.g., forcing) function, and x denotes spatial coordinate. The diffusion coefficient tensor, denoted by K, is employed to measure the physical properties of this system.
Unlabeled PDE Data: In [33, inputs to neural operators have four channels: the source function, and three diffusion coefficients (two diagonal elements and one off-diagonal element in the symmetric diffusion coefficient tensor) expanded to the whole spatial domain. We use this input as the unlabeled data pretrain the neural operator.
2. Helmholtz: We consider the 2D inhomogeneous Helmholtz equation, which is time- line-independent form of the wave equation, with periodic boundary conditions and a spatial domain f = [0, 1|2. The governing equation is given by
- + wu ww= f f in S2
(2)
where u(x) denotes the solution, f(x) represents the source function, and w > is the wavenum- ber (that defines the dynamic properties of the Helmholtz equation). This system produces high-frequency large wavenumber oscillatory patterns, which poses challenges in terms of gener- alization.
Unlabeled PDE Data: In [73], inputs to neural operators have two channels: the source function, and the wavenumber w expanded to the whole spatial domain. We use this input as the unlabeled data to pretrain the neural operator.
3. Reaction Reaction-Difftsioon The 2D Reaction-Diffusee (RD) equation involves the interaction between two nonlinear coupled variables, i.e., the activator u(t, x. y) and the inhibitor v(t, x. y). The RD equation is given by
0₂u (2 u + u) + Ra:
0,0 (0, u) + R,,
where {D., D are diffusion coefficients for and v, respectively, and where R, and R2 are reaction functions, which are defined as the Fitzhugh-Nagumo equation [36]:
R.
u k v,
(4)
R2 u U.
The spatial domain considered for the 2D RD equation is Ω = [-1, 1]2, and the time duration is 1 (0,5).
Unlabeled PDE Data: In PDEBench 74 we forecast the activator u and the inhibitor v (i.e. two input channels) with T = 10. Since each individual snapshot can serve as an initial condition for forecasting, during our unsupervised pretraining we discard the temporal dynamics and use randomly shuffled snapshots of u and y with T pretraining the model only has access to single and individual snapshots without any awareness of temporal dynamics.
unlabeled PDE data. That means, during
4. Navier-Stokes Equation: Lastly, we consider the 2D incompressible Navier-Stokes equation in vorticity form on the unit torus, which formulated
V 0
(5)
p p + Vv) - Vp+
Here, the velocity is defined within the time duration J0, 7) and the spatial domain [o. 1]2 (the vorticity can be formulated as w
V v). Moreover, P is the density, and the coefficient
17

---

signifies the viscosity of fluid, and f is the external forcing function. Dirichlet boundary conditions are employed in this system.
Unlabeled PDE Data: We follow the training settings from previous works. 1) For FNO [45], we focus on mapping the vorticity from the initial condition to the full solution (wo w (o. with T 33). The imput includes four channels: the vorticity (which is the initial condition, duplicated for ' times), and three (xy-spatial and temporal) linear mesh position embedding channels. Thus, similar to Sraction-Diffusion above, during pretraining, the model will take individual snapshots of vorticity and position embeddings as unlabeled data. 2) For PDEBench (74), we forecast both xy velocities and pressure (i.e. three input channels), with T 15. The model will take individual snapshots of vorticity and xy velocities as unlabeled data.
We summarize detailed inputs and outputs in Table 1
Table I: Inputs and outputs for learning different PDEs. See Table for resolutions. "NS": Navier-Stokes. "RD":
Difffusinn............
PDE Simulations
Imput
Input Shape
Ouipul
Poissun
Source function (f), diflasion coeflicienis (K)
Cx WCC ()
Potentia ficld
Helmholoz
(f), wavenumber
W
[x]
NS (FNO [
Varticity (w), Spuiioecmpral Cocrlinatess
H X
IT 33)
Vorticity (0.
NS (PDEBench 7
Voloxily (v,, %y). preseur (p)
TXC X ( 15,
Velocity (1., %p pressun (p) at +++
RD (PDHBench
Activator inhibitor
TXCX H
Activator (d), inhibitor at
B Detailed Experiment Settings
B.1 Distributions of Unlabeled PDE Data
In T for the purpose of OOD testing, we summarize the distribution of our unlabeled PDE data during pretraining, fine-tuning, and inference with in-context examples. Ranges of these physical parameters are inspired by [73). During pretraining,
consider wide distribution of unlabeled
PDE data. When training (fine-tuning) our model, we consider in-distribution unlabeled PDE data. Finally, we test similarity based method that learns in-context examples in OOD settings For Helmholtz OOD, we choose narrow range of coefficients ([15, 20)) mainly because its solution is very sensitive the wavenumber, and FNO's performance significantly drops when move to more extreme OOD settings.
Table 2: Ranges of physical parameters (integers) for unsupervised pretraining, training (fine-tuning), and out-of-distribution (OOD) inference.
Poisson (diffusion)
Helmholtz (wave number)
Naiver Stokes (Reynolds number)
Physies Parameters
Unsupervised Pretraining
[1. 20]
[1, 20]
[100, 300, 500. 800, 1000) 300
Training (or Fine-tuning)
[5. 15]
[5. 15]
DiittiDDitiiittinnn Testing
[15, 50]
[15, 20]
10000
B.2 Data Generation
Unlabeled PDE Data. We generate data for Poisson and Helmholtz 773,
Drcaction-Diffssiin on
PDF--Bncc 774 and 2D incompressible Navier-Stokes PINO Dataset 46 following the procedure
mentioned in the paper. For unlabeled data generation, we bypass the computation of solvers, which expedites the generation speed, shown in Table 4
OOD Samples. The OOD data generation procedure is similar to the unlabeled data, except for the changes in the physical parameters coefficients. For Poisson and Helmholtz, we consider changing the range of diffusion eigenvalue and waver number respectively. For Navier-Stokes equation, we change the Reynolds number. We list the coefficients in Table 2
*For Navier Stokes from PDEBench, we use original data. We could not generate our own pretraining/fieetninng data with different ranges of physics parameters. due to a possible mismatch of the provided configuration files and the version of Phiflow used in PDEBench (see GitHub issue at https//iibb..oonneeeeeo/////ssss
18

---

B.3 Training Hyperparameters
We summarize our hyperparameters used during pretraining and fie-tuning/training in Table These hyperparameters strictly follow previous works [45] 73 74 46 52]. We conducted our experiments
on four A100 GPUs, each with 40GB of memory.
Table 3: Hyperparameters for pretraining and training/fne-tuning. "N.S.": : 2D Incompressible Navier-Stokce "DAdapt": adaptive learning rate by D-adaptation TO] "ns": total number of simulated training samples. batch size of (32 us)" is because the total number of training samples might be fewer than 32.
Stage
MOlls
Porson
Helmoott
Reaction Ditrsssonn
N.S. (PINU)
N.S. (PUlatench)
URAS
Scalartiow
Airool
Nunbee of Sumples
46.080
46,000
76, 760
$7,545
718.286
8,760
56, 160
48,104
Leassing Raiz
1 10-3
x 10-3
x 10-3
1 10-1
DAdlgs
I 10-3
1 10-1
1 10-3
Protraiting
Bunch Size
32
32
32
32
32
Rosihtto:: W)
64-64
64x66
128
$12-512
180
180x 120
Fiuchneluzations
500 spechs
eprctt
epratss
100,000
5p0cpochh
500cpocch
500 upodtt
500 opechs
DAdape
x
Thining!
min132,
(1
x
S124512x
Eppcholtttaiioss
500 epechs
500000
500 epoch
500 epochs
500
500 spochs NIA
Rolliate
NIA
NA
91
NDA
C Examples of Simulation Costs
In T we demonstrate the cheap simulation of only unlabeled PDE data, versus simulating both unlabeled PDE data and solutions, on 2D incompressible Navier-Stoke on PINO Dataset [46] and Reaction-Diffusion on PDE-Bench [74] We can see that unlabeled PDE data are extremely cheap to simulate. Therefore, our pretraining method can boost the performance and meanwhile save the heavy cost of data simulations.
Table 4: Simulation time costs on 2D Incompressible Navier- Stokke ("N.S.") on PINO Dataset 26 and Reaction- Diffusion ("R.D.") PDE-Benck (74). "Re": Reynolds number. "D., D.": diffusion coefficients. N: number samples, temporal resolution. for velocities u, v R.D.)
spatial resolution. C: input channels (1 the vorticity N.S.,
Physical
Unlabeled PDE Data
Data+Sokutioss
Data
(scc.
(xoc.]
Size
CPU
GPU
= 100
5499.32
11013.90
N.S.
Re 300
500
3683.02
4059.71
7625.82
8963.39
N X W
AMD EPYC 7963
NVVDDI A (40GB)
800
1
1081 1.11
512
Re 1000
4957.24
10788.69
R.D.
D.
D. 10
10-3
29.65
NxT X 1000
128
AMD BPYC
D Model Architectures
In Figure 7 We show visualizations of architectures described Sec. Sec.3.1.4 3.1.4 Specifically, the FNO
has 67.1M 1M parameters, and the Video-MAE has 23.4M. During pretraining, FNO costs 20 GPU hours, and Video-MAE costs 18 GPU hours. During fine-tuning, FNO costs 4 GPU hours, and Video-MAE costs GPU hours.
E Comparison with Contrastive Learning
Contrastive learning is an important self-supervised pretraining technique studied in computer vision. For fair comparison, we directly compare with MoCo v2 [8) highly-cited self supervisdd learning method also originally implemented in PyTorch, whose core method closely related to SimCLR 7 (originally implemented in TensorFlow).
We compare MoCo v2 with our method on a broader real-world benchmark ERA5 30 Figure g our unsupervised learning method can largely outperform MoCo v2. This extra comprehen-
shown in
19

---

VideoMAE
FNO
ConväD
Input
Posilional Embedding
Linear
Tokens
FFT Linear IFFT
Fourier Layer
Linear
12X Encoder Trensformer Block
(wicin=384, #heads
Linear GeLU Linear
GeLU
Decoder Transformer Block (width= 192, #heads 3)
4X
Fourler Transform
IFFT: inverse Foutler Transform
Output
Output
Figure 7: Visualizations of architectures we studied. Left: FNO 46
Right: VideoMAE 77
sive result demonstrates that method can be widely adopted in real-world problems, outperforming previous unsupervised learning methods.
ERA5 Mideo-MAET
unsupervised
Relative l Error
0.28
0.36
0.22
10'
10'
10'
Number of Training Samples
Figure 8: Comparison between our unsupervised pretraining method versus MoCo v2
g
More Comparisons with Vision Pretrained Models
Beyond Figure Fe for the transformer on time-dependent PDE (Navier-Stokes), we further verify that pretraining unsupervised PDE data makes FNO outperform its off-the-shelf vision-pretrained checkpoints. Specifically, we first pretrain the 2D FNO model on ImageNet [ then fine-tune on the downstream PDE simulation data. As shown in Figure vision-pretrained FNO performs worse during downstream fine-tuning on time-independent PDEs (including both Poisson and Helmholtz equations), confirming that domain-specine pretraining, even unsupervised PDE data, more beneficial than conventional checkpoints pretrained on unrelated domains like computer vision.
G Joint Pretraining Further Improves Performance
We also study if joint pretraining on unlabeled multiple PDE data can bring extra benefits. We combine all 46,080 unlabeled Poisson samples and all 46,080 unlabeled Helmholtz samples (see Table We choose this setting because recent works on SciML foundation models (52 26 also use all samples from each PDE for pretraining. We do zero-paddings for mismatched channels. From Figure 10] we can see that joint pretraining can further improve the performance of fine-tuning on different PDEs.
20

---

Pretrain: ImageNet
Pretrain: ImageNet
Finetune: Poisson (FNO)
Finetune: Helmholtz (FNO)
Relative f2 Error
Relative 12 Error
9.30
unsupervised
unsupervised
0.01
random init.
0.01
random init.
vision pretrained
vision pretrained
Number of Training Samples
Number of Training Samples
Figure 9; Pretraining neural operators unlabeled data improves performance and data efficiency on
Poisson (left), Helmholtz (right). "random init.": models are trained from scratch with random initialization.
"vision pretrained"; fine-tuning from the checkpoint pretrained on computer vision dataset ImageNet [
Pretrain: Poisson + Helmholtz Finetune: Poisson (FNO)
Pretrain: Poisson Helmholtz
Pretrain: NS + RD Finetune: RD (FNO)
Finetune: Helmholtz (FNO)
lalvvaeee t2 Error
Re Relative L Error
unsuperv
6.0
Number Training Samples
Number of Training Samples
Number Training Samples
Figure 10: Joint unsupervised pretraining on multiple PDEs (green solid curve) further improves the data efficiency of neural operators when fine-tuning Poisson (left), Helmholtz (middle), (right). "random init.": models trained from scratch with random initialization. "unsupervised": models pretrained on a single unsupervised PDE data, "unsupervised joint": models are pretrained on a joinr of multiple unsupervised PDE datasets. "NS": Navier Stokes. "RD": Sraction-Diifusiion
H Fine-tuning on Unseen PDEs is Challenging
We also try to fine-tune neural operators on unseen PDEs (i.e. PDEs different from pretraining). Mismatched channels are padded with zeros. We find this will lead worse performance compared with models pretrained on the same PDE, as shown in Figure
Pretrain: Poisson
Finetune: Helmholtz (FNO)
1.00
Relative Fa Error
unsupervised
0.0
unsupervised poisson
10
Number of Training Samples
Figure 11: Fine-tuning FNO (pretrained on Poisson) on unseen samples from Helmholtz.
I Consistently Improved Generalization
Beyond the generation gap we have shown in Figure (left) for the Navier Stokes equation from PDEBench, we further collect generation gaps of our models learned on other PDEs. As shown in Figure I2 on diverse PDE systems, method can contribute to universally reduced overfitting.
21

---

Generalization Gap (Test Train)
(a)
Poisson (FNO)
(b)
Helmholtz (FNO)
unsupervised
120
fardom init
Relative to Error
6.00
0.20
6.1-
unsupervised random init.
0.00
10"
10a
10'
Number of Training Samples
Number of Training Samples
Generalizatioon Gap (Test Train)
(c)
Sraaction-Difusiii (FNO, PDE-Bench)
2D Incompressible Navker-Stokes
(d)
(FNO-3D, PINO Dataset)
8.8
unsupervised
0.00
0.00
unsupervised
Relative iz Error
0.50
0.0
0.40
Number of Training Samples
10
10
Number Training Samples
Figure 12: Universally reduced overfitting (i.c. smaller generalization gaps) on diverse PDEs (a to d).
J More Ablation Studies
J.1 Magnitude of Perturbations during Pretraining.
We study the optimal magnitude of perturbations during pretraining: "Mask Ratio" (1 indicates no visible grids, and 0 means no masks); and "Blur Sigma" for the variance of Gaussian kernel for For example, on 2D) incompressible Navier-Stokes with FNO, shown in Table we find that when training with low volume of data, we should use much stronger perturbations (high masking ratios and strong blur), whereas high volume of data only requires mild perturbations.
blur (larger the more degradation). We show our results in Table 5 6 7
Table Best choice of mask ratio and blur sigma pretraining on Poisson equation.
ASamples
Musk Ratio
Blur Signa
16
0
0
0<1
64
0
0~1
128
0
0-1
256
0
0~1
512
0
0-1
1024
0
0-1
2048
0
0~1
4096
0
0-1
8192
0
0~1
Table 6: Best choice mask ratio and blur sigma for pretraining on Helmholtz equation.
#Samples
Mask
Sigma
16
0.2
Q-1
64
0,2
0-1
128
256
0.2
0-1
512
0.6
0--2
1024
0.6
(1-2
2048
0
0v1
0~1
8192
0
0-005
J.2 Ablation of the Number of Pretraining Samples.
As shown Table 9 the more unlabeled PDE data we use for pretraining, the better quality the pretrained model will be.
22

---

Table 7: Best choice of mask ratio and blur sigma for pretraining on 2D Diffusion Reaction equation.
#Samples
Mask Sigma
404
Q-1
1515
0
0-1
0-2
6060
0.9
0-1
8080
0.3
0-1
12120
0-1
16160
0-1
18180
Table 8: Best choice of mask ratio and blur sigma for pretraining on 2D incompressible Navier-Stokes.
#Samples
Mask Ratio
Blur Sigma
0.7
0-44
16
0.7
O-44
64
0
Q-1
128
256
0.05
512
0.05
1024
0.05
2000
0.05
Table More unlaheled PDE data improve the quality of pretraining. FNO on incompressible Navicr-Stokes, pretrained with mask ratio as 0.7. #Pretraining Samples
2000
57545
Relative fg Error
0.3594
0. 3246
source
mask
prediction
error
0.2
0.2
2.2
0.2
-02
0.8
0.8
1.1
Helm.
-0.0
-0.0
0.4
D.R.
-..0
0.00
0.1
-0.4
-0.1
20.1
20.1
33.4
N.S.
-1177
-11.7
16.7
-43.6
43.6
0.0
Figure 13: Visualization FNO reconstructions of unlaheled PDE data on the Poisson ("Pois.") Helmholtz ("Helm.)). 2D DiffusioonRReaction ("D.R."), and 2D incompressible Navier-Stokes ("N.S.") equations during MAE pretraining. (Mask ratio: 0.1 for Poisson, Helmholtz, and 2D Diffusion-Reaciio incompressible Navier-Stokes.) In masks, only white are visible to the model during pretraining.
equations: 0.7 for
23

---

K Visualization of MAE Pretraining
To demonstrate the efficacy of MAE-based pretraining, show the unlabeled PDE data and its reconstructed version in Figure I3 (MAE pretraining on different PDEs) and Figure [4 (MAE pretraining 2D incompressible Navier-Stokes with varying mask ratios). We that all inputs are accurately reconstructed with low errors and similar patterns.
source
mask
prediction
errer
14.66
14.6
32.5
D.1
16.8
16.5
23.5
02
-3.6
11.9
0.3
-1.8
-18
10.6
0.4
-0.3
-0.3
12.0
0.0
6.5
3.4
20.0
-36.9
-16.9
0.0
+18.3
18.0
06
-11.5
-11.3
0.0
20.1
20.1
33.4
02
43.6
443.6
-.0
51.9
51.9
80.6
0.8
1.6
44.77
19.6
19.6
96.3
0.9
-0.3
28.0
Figure Visualization of FNO reconstructions unlabeled PDE on the 2D incompressible Pavier-Stoker equations during MAE pertaining with mask ratio from 0.1 to 0.9,
L Details and Visualizations of Real-Worle Data
L.1 ECMWF Reanalysis v5 (ERAS)
We utilize data from 2006 to 2015, with the snapshots taken every hours and spatial resolution of 360 x 360. The total number of snapshots is 14600. We apply the mean-standard deviation normalization to the data and downsample the snapshots to spatial resolution of 180 x 180. We split 75% of the data for pretrain and 25% for finetune. For each split, we further separate 80% the data for training, 10% for validation, and 10% for testing.

---

L.2 ScalarFlow
The original spatial resolution is 1062 600. We crop the snapshots 900 600 remove the padding and background. We remove the first 15 timeframes for each simulation to avoid the initial transient phase the beginning of the smoke flow generation. We then downscale the snapshots to 180 x 120 and apply mean-standard deviation normalization to the data. With total of 70200 snapshots, we split 80% of the data for pretrain and 20% for finetune. For each split, we further separate 80% of the data for training, 10% for validation, and 10% for testing.
L.3 Airfoil
We split 80% of the training data for pretrain and 20% for finetune. The original spatial resolution of each sample 128 x 128. We crop the smapshots to 96 x 45 to remove the background. We then apply min-max normalization thannel-wise the sample.
We show visualizations of real-world scientific data in Figure 15
Densty
t=)
[b) 17
(c) 50
(d) 67
30.3
12.1
15.1
(f)
(g)
(h) Pressure
Velocity X
Velocity
Figure 15: We show snapshot examples from ERAS temperature 30] (a. b) and ScalarFlow [14 (c, d) at different temporal steps; and also example of Airfoil mask, velocities, and pressure 75 (e-j).
M Benefits of In-context Examples
We try to understand the benefit of leveraging in-context examples by decomposing the relative MSE error into "scale" and "shape." "Scale" is the slope of linear regression between targets and predictions (closer to the better), indicating the alignment of the range of model outputs with targets. "Shape" is the normalized relative MSE (i.e., model outputs targets normalized by their own largest magnitude before MSE), indicating the alignment of scale-invariant spatial/temporal structures. We show results in Figure 16] We find that the benefit of in-context examples lies in that the scale of the model's output keeps being better calibrated ("scale" being closer to 1) when adding more demos.
numerical simulations or predictions PDEs, there are settings where the scale or magnitude of solutions is more important than the exact shape/pattern:
1. Heat Transfer:
In large-scale systems, the focus might be overall temperature and extreme values. For instance, in evaluating cooling system, the key concern might be the peak temperature rather than the detailed temperature distribution.
2. Fluid Dynamics: For applications like aerodynamics, the overall drag or lift force on an object is often more critical than capturing every detail of the flow pattern, such as in airfoil design.
3. Environmental Modeling: The concentration of pollutants specific locations or total pollutant transport is often more crucial than the exact distribution, such as groundwater flow studies.
25

---

FNO on Poisson
FNO Helmholtz
FNO-3D PINO Dataset incompressible Stvker-Sookss
Scale (Closer the Botter)
8.4
Scalle (Clouer to the Betteri
-1.4
7.28
Scale 6.00
Solettiops Reynulet
Comaller
1.8
Solk (Siepe o Jree: fegsuibr
Shapp Domalleer the
19.2%
Share Jorralow
+0.0
Lta
3.00
126
1E
Figure Benefits in-context examples. analyze the benefit in-context examples for complicated PDE systems, we decompose the relative MSE error into "Scale" and "Shape". "Scale" indicates the alignment of the range of model outputs with targets (closer to the better), via the slope of linear regression. "Shape" indicates the alignment of scale-invariant spatial/temporal structures via normalized relative MSE (i.e. model outputs or targets are normalized by their own largest magnitude before MSE). We find that the benefit in-context examples lies in that the scale of the model's output keeps being calibrated (red line being closer to 1) when adding more demos.
N Visualizations with In-Context Examples
We show visualizations of our similarity-based based mining of in-context examples in Figure 17 In this visualization, we find that the range of numerical solutions (e.g., values colorbars) predicted with in-context examples becomes closer to the target. Meanwhile, based on this visualization, we conclude that OOD generalization challenging for neural operators because of: 1) Significantly different patterns of solutions under different physical parameters; 2) Different value ranges of solutions under different physical parameters.
target
#demo
#demo
#demo
0.1
0.5
0.4
0.2
Pais.
0.0
0.2
0.0
0.1
0.1
0.4
0.1
0.1
Helm.
..0
0.0
0.0
0.0
.02
-0.4
..2
-0.1
70.5
+32.6
50.7
48.9
N.S.
-12266
-75.0
100.3
110.6
Figure 17: Visualizations of mining in co-context examples for FNO in OOD testing. Ranges of solutions predicted with in-context examples (min/max of each snapshot, reflected in colorbars) become closer to the target.
26

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: All claims stated in the abstract and introduction are evaluated in our experi- ments (Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes].
Justification:
We discussed our limitations separate Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
27

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Ye].
Justification: We include instructions for reproducibility in the Appendix, including choices unlabeled PDE data (Appendix AA, data generation and training hyperparameters B and
model structures D We also attached our code in supplement.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
28

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes].
Justification:
We attached our code in the supplement.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
mublic//gudes//CdeSabbiisss ios for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https wtwpol/iigggiiee//eeeeeoe hocpe///ggiddeeeeCouSuuubbibii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes].
Justification:
We include our experimental settings in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes].
Justification:
All our operator pretraining experiments are reproduced for three times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

---

The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative enrrr rates).
show in tables
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes].
Justification:
We discussed the compute resources we used in Appendix B.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httppe///eorrrs..cccpcp/////eeea
Answer: [Yes].
Justification:
Our work does involve any potential harm, societal impact, violation.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
This work is about foundational research data augmentations. It does lead to any societal impact because: 1) our motivations, goals, and results are too distant from social applications
Justification:
make any social impact; 2) mainly public data and benchmarks; and 3) we use public model architectures and follow previous works which have open access.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA].
Justification: The paper poses no such risks because all datasets studied in our work from public benchmarks, and we have clearly cited original papers.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes].
Justification: All datasets studied in our work are from public benchmarks, and we have clearly cited original papers.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
31

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector:
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: The paper does not involve any potential risk regarding research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
32

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "x3d2KhITtw", "reviewer": "Reviewer_oUb1", "summary": "This paper presents an unsupervised pretraining approach for PDE solvers based on Meta Autoencoding and Super Resolution. The authors show that after pretraining, the model can achieve better accuracy than training a solver from scratch. This paper also presents an \"in-context learning\" strategy for inference on out-of-distribution data.", "questions": "- Can you report the model sizes, pretraining cost, and finetuning cost?\n\n- Can the pretrained model be applied to other PDEs which are not seen in pretraining?", "limitations": "The authors discuss some of the limitations of this paper in Sec. 6.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Designing pretraining techniques for SciML foundation model is an important research direction.\n\n- To the best of my knowledge, the proposed pretraining technique is new in the SciML context.", "weaknesses": "- **Some important experimental setups are missing**. In the main body of the paper, it's not even clear what the model size is, what the pretraining datasets are, and whether the model is pretrained on one class of PDEs or multiple classes of PDEs.\n  - A follow-up concern is that according to Table 2 in Appendix B, it seems the authors pretrain different models for different PDEs. This significantly undermines the value of pretraining, because the ultimate goal of pretraining is to obtain a general purpose foundation model which can be easily transferred to different tasks (i.e., different PDEs in this context). If the method requires pretraining for each single class of PDEs, it's not efficient any more, and the comparison in Fig. 3 is not fair because **the proposed approach is still a problem-specific technique** and the pretrained models require additional training.\n\n- **Missing baselines.** For time-independent PDEs, the MAE for image should also be included as a baseline to support the claim of _outperforming conventional vision-pretrained models_.\n\n- I feel that **\"in-context learning\" in Sec. 3.2 is a misnomer**. First, there is no notion of \"context\" in the pretrained model. Second, the presented model is not an auto-regressive model. Third, unlike the self-attentive models, the presented model does not actively or flexibly \"learn\" from the examples; the way to aggregate information is fixed given Algorithm 1. The presented \"ICL\" algorithm deviates too much from the standard practice. I strongly encourage the authors to get rid of  \"ICL\" and refers the proposed approach as a few-shot learning strategy in the paper to avoid misleading the community.\n\n- In Sec. 4.3, the few-shot learning accuracy looks poor, and the standard deviation of the error is not reported.\n\n- Some discussions in the papers are inaccurate or lack justification.\n  - Line 95: BERT does not leverage next token prediction for pretraining.\n  - Line 142: Multiple benefits of pretraining are listed, but no references are provided, and no relevant experiments are presented to justify those arguments (e.g., better generalization, faster training speed, etc).\n  - Line 158: the authors cite the BERT paper and claim that \"_these methods enable training in NLP and CV of generalizable models containing over one hundred billion parameters_\". I don't think any variant of BERT is scaled to one hundred billion parameters.\n\n- Other issues.\n  - Line 26: DNN $\\to$ deep neural network\n  - Line 32: has  $\\to$  have\n  - Line 46: start   $\\to$  start with", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly thank the time and effort of reviewer oUb1 in reviewing our paper!\n\nBefore providing detailed responses, we would like to make a **clarification**:\n\nSince our paper focuses on **unsupervised pretraining** *but not* SciML foundation models (we *never* claimed our pretraining leads to such models), we hope reviewer oUb1 will not assume we are pursuing or comparing with the latest SciML foundation model papers. We believe it is *unfair* to ask us to match their experimental settings.\n\nAs the benefits and strategies of pretraining for SciML are still under-explored, both supervised/unsupervised/per-PDE/joint pretraining are meaningful to this research direction. This is similar to what researchers in the computer vision community explored with single dataset/task pretraining [1, 2, 3] before moving on to foundation models.\n\n---\n---\n\n> **Q1**. Some important experimental setups are missing\n\n1. Model sizes. FNO: 67.1M. Video-MAE: 23.4M.\nIn each experiment, all compared models are of the same architecture and thus fairly compared.\n2. Details about pretraining datasets are clearly documented in Appendix A, which is also clearly referred to in Line 134.\n3. In each experiment, we pretrain our model only on one PDE. As we clarified above, our work does not pursue “Pretraining on multiple classes of PDEs”.\n\n\n> **Q2**. Pretrain different models for different PDEs\n\n1. We provide extra experiments in **Figure 1 in our attached PDF response**. We can see that **joint pretraining can further improve the performance of fine-tuning on different PDEs**. We will include this figure in our camera-ready version.\n2. Although joint pretraining can be further beneficial, experiments of **pretraining on a single PDE are still fundamental and necessary**. These should be studied before moving to joint pretraining and **cannot be ignored or underestimated**. These experiments help analyze the complexity or difficulty of learning each single PDE for neural operators.\n3. **Recent works, like [4], also only adopted per-PDE pretraining, even though their work is claimed to be more related to SciML foundation models than ours**. Previous unsupervised pretraining works in computer vision also focused on a single dataset (like ImageNet).\n\n\n> **Q3**. Missing baselines\n\nOn time-independent PDEs, we mainly focus on the FNO model. There are no publicly available checkpoints of FNO that are pretrained on natural images. Due to limited time during rebuttal, it is also challenging to pretrain an FNO ourselves on large-scale image datasets like ImageNet or JFT-300M. We will try to include this experiment in our camera-ready version.\n\nMeanwhile, on time-dependent PDEs, we studied Video-MAE because we followed previous work [5]. In [5], they also did not consider any image-pretrained MAE models for their claims.\n\n\n> **Q4**. \"in-context learning\" in Sec. 3.2\n\nWe are happy to rename our method.\n\nMeanwhile, we have further corresponding comments:\n1) In recent literature about large language models (LLMs), “context” is mostly referred to in downstream tasks, not in pretrained models, and we did not claim that our context is in our pretrained models;\n2) In-context learning is not directly related to auto-regressive models, see [6];\n3) Models with self-attention are also not learning anything during inference, since all model weights are fixed. Both self-attention and our models are extracting features during inference.\n\n\n> **Q5**. Accuracy and standard deviation of few-shot learning\n\nWe report standard deviations in **Figure 3 in our attached PDF response**.\n\nThe performance of neural operators in out-of-distribution (OOD) settings is known to be poor and challenging. See **Figure B.3 (f) of [4]**: performance on Helmholtz (“SYS-3”) is poor (both their errors and ours are close to 1), even with supervised fine-tuning on a few OOD samples.\n\n\n> **Q6**. Line 95: BERT does not leverage next token prediction for pretraining\n\nThanks! We will update this sentence with a more precise description.\n\n\n> **Q7**. Line 142: Multiple benefits of pretraining are listed, but no references are provided\n\nAs clearly mentioned in Line 148, these experiments about benefits of unsupervised pretraining are in Figure 4 in Sec. 4.1, which is in the main text and should not be ignored.\n\n\n> **Q8**. Line 158: I don't think any variant of BERT is scaled to one hundred billion parameters\n\nThanks! We will cite more papers with models of billion-level parameters that share similar pretraining strategies as BERT.\n\n\n> **Q9**. Other issues.\n\nThanks! We will fix these typos.\n\n\n> **Q10**. Can the pretrained model be applied to other PDEs which are not seen in pretraining?\n\nWe provide experiments for fine-tuning on PDEs unseen during pretraining, shown in **Figure 2 in our attached PDF response**. Fine-tuning on unseen PDEs leads to worse performance, which is expected not only because they have different initial conditions but also due to their mismatched input dimensions. The input dimension of Poisson is four (forcing function and three diffusion coefficients), while the input dimension of Helmholtz is two (source function and the wavenumber). This mismatched input is documented in Section A.\n\n\n[1] “A Simple Framework for Contrastive Learning of Visual Representations” Chen et al. 2020\n\n[2] “Improved Baselines with Momentum Contrastive Learning” Chen et al. 2020\n\n[3] “Masked Autoencoders Are Scalable Vision Learners” He et al. 2021\n\n[4] “Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior“ Subramanian et al. 2023\n\n[5] “Multiple Physics Pretraining for Physical Surrogate Models” McCabe et al. 2023\n\n[6] “In-Context Operator Learning with Data Prompts for Differential Equation Problems” Liu et al. 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MU5IK1EZtX", "reviewer": "Reviewer_oEEz", "summary": "This paper aims at improving the data efficiency of deep learning models for tackling Operator Learning. The paper focuses on two aspects: 1) They pretrain neural operators on data that do not assume labels, i.e. without the target function or the trajectory solution of states. To do so, they rely on a masked auto-encoding task or a super-resolution task, that both can be trained without labels. They show that this pretraining strategy can effectively improve the sample efficiency compared to a random initialization.  2) They propose an algorithm to do in-context learning at test time based on a number of demo examples and show results on out-of-domain PDEs.", "questions": "* What is the best effective strategy for pretaining ? Is it the masking-damasking or super-resolution that yields the best results ? Or do you use both ?\n* Did you observe a difference between the video transformer and FNO for this setup ? Is there one that you would recommend better than the other ?", "limitations": "There is a limitation section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The main strength of the paper is that the pretraining strategy effectively reduces the number of samples to reach a certain target accuracy compared to a random initialization.\n* The method is tested against multiple equations from different levels of difficulty.\n* The comparison with pretrained vision transformers, though a little counterintuitive, is informative on the capacity of transferability of this kind of models to different tasks.\n* The ICL experiments seem to be consistent, the more demos the better the results.", "weaknesses": "* Overall the paper is quite difficult to read, there are many different aspects that are described within the method section and it can be hard to understand in the experiment section which is what.\n* If I am not mistaken there is no consistent comparison between the different tasks and architectures of the method (super-resolution vs masking x FNO vs transformer) for the different datasets. \n* The ICL method should be compared to at least one baseline. I suggest to do a simple kernel method but on the inputs rather than with the predictions of FNO. It can be a kNN or anything, but it would actually give a good understanding of how well the method works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly thank the time and effort of reviewer oEEz in reviewing our paper!\n\n> **Q1**. Overall the paper is quite difficult to read\n\nThanks for this suggestion! We will try to make our teaser figure (Figure 1) clearer and connect it more to the subsections in methods and experiments, so readers can follow easily.\n\n\n> **Q2**. Comparison between the different tasks and architectures of the method\n\n1. To focus on pretraining (model initialization), we keep the network architecture the same in each experiment to ensure that our comparisons in each task are fair.\n2. We study the contribution of masking vs. super-resolution in Appendix G.1. We find that when training with a low volume of data, we should use much stronger perturbations (high masking ratios and strong blur), whereas a high volume of data only requires mild perturbations.\n\n\n> **Q3**. The ICL method should be compared to at least one baseline.\n\nThanks for the suggestion!\n\nWe further provide a baseline for the ICL method in **Figure 3 in our attached PDF response**.\n\nIn our paper, our ICL leverages the output (prediction) of neural operators to find similar samples (lines 251–255). In this new baseline, we try to use features extracted by the backbone of the neural operator (high-dimensional features before the final output layer) to find similar samples. As we can see, in general, this baseline is worse than our original ICL method, indicating that the final output of the neural operator can more accurately indicate true similar samples.\n\n\n> **Q4**. What is the best effective strategy for pretaining? Is it the masking-damasking or super-resolution that yields the best results? Or do you use both?\n\nWe find that combined methods (MAE + superresolution) give the best performance. These results are in Appendix G.1, where we exhaustively studied the choices of hyperparameters for MAE and superresolution.\n\n> **Q5**. Did you observe a difference between the video transformer and FNO for this setup?\n\nWe tend to avoid recommending specific model architectures, as our method is agnostic to the architecture choices. We have proven that our method can help both FNO and transformers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GSLMzjvLLC", "reviewer": "Reviewer_6vSy", "summary": "This paper proposes a pre training strategy for operator learning. They introduce unsupervised training in the case of physical data. 2 architectures are studied: transformers and FNO. Finally, some experiments on different PDEs are conducted to highlight the properties of the model.", "questions": "-\tFor the unlabeled data for time-dependent PDE, how does the PDE ICs are sufficient to identify the PDE? How does the model have access to the PDE parameters? \n-\tIs your model trained on multiple physics? Or pretraining are proceeded on one PDE? \n-\tOn figure3, why are the unsupervised training performing better than training from scratch? How is proceeded this “random init” training? Is it using data with supervised loss? \n-\tWhat is the comparison of other baselines in the OOD setting? \n-\tWhat is the computational cost of training your model compared to other baselines? How many parameters do the baselines contains?", "limitations": "Author have proposed a discussion on the limitations of their paper. However, they did not address societal impact and ecological considerations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and explains step-by-step the algorithms and architecture. \nSeveral experiments highlight the performances of the model on both simulated and real-world datasets, which is a challenging setting. Moreover, statistics are provided for each experiment.", "weaknesses": "As mentioned in the paper, masked auto-encoding has already been proposed in CV applications. The contribution however is unclear for me. ICL and masked-auto-encoding are widely known techniques. Moreover, it is difficult to identify why is the unsupervised pretraining performs best than other baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly thank the time and effort of reviewer 6vSy in reviewing our paper!\n\n> **Q1**. Masked auto-encoding has already been proposed in CV applications. The contribution however is unclear for me.\n\nWe would like to clarify and re-emphasize: The most important contribution of our paper is to define and leverage **unsupervised data** in PDEs and design **unsupervised pretraining** on top of it. This is mentioned in our first contribution bullet (lines 66–69) and also in line 7 of our abstract. There are previous works that try to pretrain neural operators on simulated solutions [1, 2, 3], but no previous works tried to reduce the simulation costs by defining and leveraging unlabeled PDE data. That means, although pretraining is studied in scientific machine learning, **unsupervised pretraining is largely underexplored in the community**. Our work is the first kind, and will inspire the community in this research direction.\n\nWe do not claim that our MAE method is new, and we also provide clear motivation for why we introduce MAE in pretraining neural operators (lines 160–172).\n\n\n> **Q2**. Moreover, it is difficult to identify why is the unsupervised pretraining performs best than other baselines. \n\n> (Similar question) On figure3, why are the unsupervised training performing better than training from scratch? \n\nWe discussed the benefits of our unsupervised pretraining in Figure 4 in Sec. 4.1. There are three reasons:\n\n1. Reduced overfitting, which consistently leads to smaller generalization gaps across all PDEs we studied.\n2. Faster convergence, which can accelerate model convergence more than both random initialization and vision-pretrained checkpoints.\n3. Meaningful representations, which are extracted from pretrained Video-MAE and are helpful during fine-tuning.\n\n\n> **Q3**. For the unlabeled data for time-dependent PDE, how does the PDE ICs are sufficient to identify the PDE? How does the model have access to the PDE parameters?\n\nOur models do not have access to physical parameters. Given the initial condition and boundary condition, the dynamics of a PDE are determined since there is no stochasticity in the PDE.\n\n\n> **Q4**. Is your model trained on multiple physics? Or pretraining are proceeded on one PDE?\n\nOur experiments focus on pretraining on one single PDE.\n\nHowever, we also further studied joint pretraining. We provide extra experiments in **Figure 1 in our attached PDF response**. We can see that joint pretraining can further improve the performance of fine-tuning on different PDEs. We will include this figure in our camera-ready version.\n\n\n> **Q5**. How is proceeded this “random init” training? Is it using data with supervised loss?\n\nIn Figure 3, the “random init” and “unsupervised” models share completely the same fine-tuning settings, and they both use the supervised loss (labeled PDE data) during fine-tuning.\n\nThe only difference is that “random init” uses random initialization of model weights to start the fine-tuning, while “unsupervised” uses pretrained weights (from our unsupervised pretraining) to start the fine-tuning.\n\n\n> **Q6**. What is the comparison of other baselines in the OOD setting?\n\nThanks for the suggestion!\n\nWe further provide a baseline in the OOD setting, attached in **Figure 3 in our attached PDF response**.\n\nIn our paper, our ICL method leverages the output (prediction) of neural operators to find similar samples (lines 251–255). In this new baseline, we try to use features extracted by the backbone of the neural operator (high-dimensional features before the final output layer) to find similar samples. As we can see, in general, this baseline is worse than our original ICL method, indicating that the final output of the neural operator can more accurately indicate true similar samples.\n\n\n> **Q7**. What is the computational cost of training your model compared to other baselines? How many parameters do the baselines contains?\n\n* Pretraining costs. FNO: 20 GPU hours. Video-MAE: 18 GPU hours. (GPU: A100)\n* Model parameters. FNO: 67.1M. Video-MAE: 23.4M.\n\n\n> **Q8**. Societal impact and ecological considerations\n\nOur paper improves the data efficiency of neural operators for solving PDEs, which can significantly reduce computational costs and energy consumption associated with high-fidelity numerical PDE simulations. This improvement can lead to more sustainable scientific research by minimizing the environmental footprint of extensive computational experiments. Moreover, by democratizing access to advanced PDE solutions through more efficient pretraining, our method has the potential to accelerate scientific and engineering advancements in the broader community, benefiting society at large.\n\n[1] “Lie Point Symmetry Data Augmentation for Neural PDE Solvers” Brandstetter et al. 2022\n\n[2] “Self-supervised learning with lie symmetries for partial differential equations“ Mialon et al. 2023\n\n[3] “Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior“ Subramanian et al. 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oScYR4DeCm", "reviewer": "Reviewer_u3i6", "summary": "This paper considers the problem of training neural operators to solve forward partial differential equation (PDE) problems in data-limited settings. The paper motivates this setting with the large data simulation cost incurred by these methods when generating a large training set of PDE solutions. The paper suggests a self-supervised pretraining method derived from self-supervised learning tasks popular in the computer vision community. This pretraining method only requires a dataset of initial conditions, PDE coefficients, and physical parameters; it does not require any PDE solutions. The paper shows the benefit of their method by showing their pretrained models outperform randomly-initialized models or models pretrained on general computer vision tasks; there are many evaluations across a range of PDE problems and real-world experimental data. In addition, the paper considers an \"out-of-distribution\" (OOD) task where PDE coefficient settings are changed at test time and suggests an \"In-Context Learning\" (ICL) strategy, which uses the trained model to adaptively combine examples of ground-truth solutions.", "questions": "General:\n - Line 132 states \"initial snapshot $u_0(x)$ ... is sufficient for defining PDE systems.\" Could the authors explain or qualify this statement? The definition of a PDE system must also include a partial differential equation with parameters, a domain, and possibly boundary conditions like $u_0(x)$.\n - Line 185 states \"Numerical solutions of PDEs are expected to exhibit invariance to filtering blur or different resolutions of inputs.\" Could the authors explain or qualify this statement? There are many examples where qualitative features of PDE solutions depend on the discretization level. One example is the Helmholtz problem considered with high wavenumber and forcing term with high-frequency oscillations. As the resolution of the input forcing term and solution increase, higher-frequency oscillations in the solution can be resolved. For the Poisson equation, one can show analytically that applying a blur to the forcing term will change the solution.\n - The experiment in Appendix E shows encouraging results that the general-purpose self-supervised learning method MoCo v2 does not perform well on the ERA5 task. I believe that if one were to combine Figure 8 and Figure 5a, the result would show that pretraining with MoCo v2 produces almost no improvement over random initialization. This seems like a nice result and wonder if the authors have intuition about why this is the case.\n\nQuestions about PDE experiments:\n - What are the inputs and outputs of the model for each experiment? As I was reading this paper, I was constantly asking this question and spent time jumping around in the appendix looking for details. Would it be possible to collect all of this information in a table for ease of reference?\n - For each PDE experiment, what is the distribution of forcing functions (Poisson, Helmholtz) or initial conditions (Reaction-Diffusion, Navier-Stokes)?\n - How does the \"diffusion\" physics parameter in Table 1 relate to the definition of the Poisson equation in Equation 1? Are elements of K randomly drawn from this range?\n - How does the Reynolds number relate to the definition of the Navier-Stokes equations (Equation 5)?\n - Are there spatial boundary conditions for the Reaction-Diffusion equation?\n - What do the error bars in Figures 3 and 4 represent? Are they showing the min/max of the three independent random seeds?\n\nQuestions about real-world data experiments:\n - The ERA5 and ScalarFlow datasets have temporal observations; the learning task is to predict the state of the system at time $t+16$ given a sequence of state observations at times $[t, t+15]$. What is the unlabeled data used in pretraining for these cases? Is it sequences of state observations at times $[t, t+15]$, for a set of different values $t$?\n - For these experiments, how does using snapshots at $t > 0$ agree with the definition of unlabeled PDE data for time-dependent systems in Section 3.1.1?\n - These experiments use a large number of snapshots to construct the pretraining dataset. In the ERA5 example, this is enough data to build $14,600 \\times \\frac{75}{100} \\times \\frac{1}{16} \\approx 680$ labeled samples. In the ScalarFlow example, this is enough data to build $70,200 \\times \\frac{80}{100} \\times \\frac{1}{16} \\approx 3500$ labeled samples. Could the snapshots used in the pretraining dataset be used in a supervised learning framework? If so, why doesn't the “random init” baseline have access to this data?\n - Are there examples of simulation or experimental settings where it is reasonable to assume we can easily gather snapshots of a system state, but these snapshots could not be used in a supervised neural operator framework? Mentioning such examples in the text would help readers understand the significance of this experiment.\n\nQuestions about the ICL method and OOD experiments:\n - Do the authors have any intuition for why the Poisson and Helmholtz OOD performance is poor, and the Navier-Stokes performance is better? Providing information about the types of PDE problems amenable to in-context learning would be a valuable contribution and would increase the impact of this experiment.\n - Is there a connection between the expected smoothness of a particular PDE’s solution and the performance of the ICL method on that PDE? For example, we know that the solution of the Poisson equation should be relatively smooth, but the proposed ICL method outputs highly non-smooth estimates.\n - I see that ICL with more demos improves the scale of the prediction. Are there problem settings where the scale of the solution is of particular interest? Discussing such settings may better contextualize the results on these challenging OOD tasks.\n\n\nAs they continue their work, I want to ensure the authors are aware of this preprint, which provides a fast method for generating synthetic data without a PDE solver, applicable to certain PDE problems. Fitting the neural operator to synthetic data could be a useful pretraining task if the PDE is known. I am NOT asking the authors to compare their method with this preprint:\n\nE. Hasani and R. A. Ward, “Generating synthetic data for neural operators.” arXiv, Jan. 04, 2024. Available: http://arxiv.org/abs/2401.02398", "limitations": "The authors have addressed some limitations of their work. As mentioned above, I believe the computational expense of the pretraining stage is a limitation of this work which is not discussed in the paper.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This paper addresses important challenges in the field of neural operators and provides well-motivated solutions to these problems. There are a large number of experiments to quantify the effect of these solutions. I will list individual strengths below:\n- The paper considers important problems in the field of neural operators: reducing the dependence on training data generated by PDE solvers, adapting trained models to out-of-distribution problem settings, and quantifying performance on complicated real-world datasets.\n- The experimentation in this paper is very thorough. The paper shows experiments on PDE problems across a broad range of physical parameters as well as real-world datasets ERA5 temperature, ScalarFlow, and Airfoil.\n- The paper's pretraining methods are relatively simple and can be applied to a broad class of problems. It is easy to imagine these methods being used by other members of the community.\n- The paper's proposed ICL method is a creative emulation of attention mechanisms that is applicable to general neural operator methods.", "weaknesses": "I have concerns about the experimental design and worry the experimental results are overstated in the text. The claims about reducing data simulation costs are particularly overstated, and I believe the paper could be improved by contracting its stated goals to focus on real-world problem settings where data is scarce, and simulation methods are unavailable. I will list individual weaknesses below:\n- The paper motivates the limited-data regime due to the computational burden of generating PDE solutions. The stated goal is to \"reduce data simulation costs\". However, the method suggests a pre-training procedure that trains full-sized models on $\\geq 40,000$ samples for $500$ epochs. This also introduces a large computational cost, and this added cost of pre-training is never addressed. Without accounting for this added cost of pre-training, the paper's claims about reducing data simulation costs feel vacuous.\n- The paper fails to contextualize their results on the PDE benchmarks with meaningful baselines. Figures 3 and 5 only consider random initialization of neural network weights and pretrained on a general computer vision task as baselines. This ignores the simple baseline of generating extra samples with the same computational cost as pretraining.\n- Similarly, the paper fails to contextualize the results of the OOD experiment with meaningful baselines. The description of the OOD experiment indicates that the methods in citations [76, 77] are designed to solve the same problem. When reading the text, it seems like these methods could be used as baselines for comparison in Figure 6. If this is not the case, because [76, 77] are solving a weaker version of the problem or require significantly more data or compute resources, this is an advantage of the proposed method and should be highlighted in the paper.\n- The amount and type of unlabeled pretraining data in the ERA5 and ScalarFlow experiments are unusual for a self-supervised learning setting, which raises questions about the significance of this experiment. I have written specific questions about this below.\n- The results of the OOD experiment indicate the OOD task is still a difficult problem for neural operators, but the text does not discuss this difficulty. Without such a discussion, the significance of this experiment and the ICL method is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly thank the time and effort of reviewer u3i6 in reviewing our paper!\n\n> **Q1**. Without accounting for this added cost of pre-training, the paper's claims about reducing data simulation costs feel vacuous.\n\nThanks for this great question!\n1. We further provide extra experiments on joint unsupervised pretraining on multiple PDEs in **Figure 1 of our attached PDF response**. We can see that joint pretraining can further improve the performance of fine-tuning on different PDEs. This means fine-tuning on different PDEs can share the same pretrained checkpoint, which reduces the total computational costs. We will include this figure in our camera-ready version.\n2. Recent works, like [1], also only considered per-PDE pretraining, even though their work is claimed to target SciML foundation models. Moreover, previous unsupervised pretraining works in computer vision also focused on a single dataset (like ImageNet), and they did not compare the costs of pretraining versus collecting more downstream samples.\n3. The most important contribution of our paper is to define and leverage **unsupervised data** in PDEs and design **unsupervised pretraining** on top of it. Although pretraining is studied in scientific machine learning, **unsupervised pretraining is largely underexplored in the community**. Our work is the first kind, and will inspire the community in this research direction.\nIn sum, we totally agree that reducing the total computation costs (training + sample collection) is important, and our promising joint pretraining results (**Figure 1 in our attached PDF response**) encourage us to keep working on this direction, but this should not weaken the core contribution and target in this paper.\n\n\n> **Q2**. Baseline of generating extra samples with the same computational cost as pretraining\n\nActually, since we studied different numbers of samples in Figure 3, we can compare our method with baselines trained with extra generated samples (as indicated by the red arrows showing the savings in the number of samples).\n\n\n> **Q3**. Contextualize the results of the OOD experiment with meaningful baselines, such as [76, 77]\n\n[76, 77] by Liu et al. are related works, but their works cannot be directly compared with ours because: 1) they did not study 2D time-dependent PDEs; 2) they only scaled up to 5 demos, which is fewer than ours. We will highlight this more in our camera-ready version.\n\nMeanwhile, we further provide another baseline in the OOD in **Figure 3 in our attached PDF response**. In our paper, our ICL leverages the output (prediction) of neural operators to find similar samples (lines 251–255). In this new baseline, we try to use features extracted by the backbone of the neural operator (high-dimensional features before the final output layer) to find similar samples. As we can see, in general, this baseline is worse than our original ICL method, indicating that the final output of the neural operator can more accurately indicate true similar samples.\n\n\n> **Q4**. The results of the OOD experiment indicate the OOD task is still a difficult problem for neural operators, but the text does not discuss this difficulty.\n\nThanks for the suggestion!\n\nBased on our visualization in Figure 14 in Appendix K, we believe OOD is challenging for neural operators because of:\n1) Significantly different patterns of solutions under different physical parameters.\n2) Different value ranges of solutions under different physical parameters.\nWe will include this discussion in our camera ready.\n\n\n> **Q5**. Line 132 states \"initial snapshot u0(x) ... is sufficient for defining PDE systems.\" Could the authors explain or qualify this statement?\n\nFor a specific PDE with fixed physical parameters and a spatial-temporal domain, if we are given the initial condition and boundary condition, the dynamics of the PDE will be determined since there is no stochasticity in the PDE.\n\n\n> **Q6**. Line 185 states \"Numerical solutions of PDEs are expected to exhibit invariance to filtering blur or different resolutions of inputs.\" Could the authors explain or qualify this statement?\n\nOur superresolution (SR) objective shares the same motivation as recent works on enhancing scientific data resolutions [2], which is to train SciML models to preserve the inherent physical properties of scientific data while learning the SR process. Given a ***specific*** input distribution, after memorizing the input data and fitting the SR objective, solutions of neural operators are expected to preserve the inherent physical properties and exhibit invariance to filtering blur. Moreover, in the context of pretraining, both objectives – MAE and SR – aim to further extract meaningful representations of input samples from current data distributions and provide a better-adapted initialization of network weights.\n\n\n> **Q7**. The experiment in Appendix E shows encouraging results that the general-purpose self-supervised learning method MoCo v2 does not perform well on the ERA5 task. ... This seems like a nice result and wonder if the authors have intuition about why this is the case.\n\nThe main reason MoCo v2 does not perform well is that it was originally designed for object-centric images, and the loss is applied at the instance level (whole image) rather than at the pixel level (per spatial location), making it suboptimal for solving PDEs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We deeply appreciate the feedback and suggestions from all four reviewers. We are pleased that **all four reviewers recognized** that our paper targets an **interesting and well-known challenge** in scientific machine learning (SciML).

We thank **all four reviewers** for acknowledging our main contribution, which is to **advance the pretraining method for SciML** and **reduce the number of labeled training samples**. This is an important research direction. We also thank reviewers oEEz, 6vSy, and u3i6 for confirming our comprehensive experiments on diverse PDEs, broad ranges of physical parameters, and real-world problems.

We address all questions and concerns in individual responses. Following the NeurIPS guidelines, we also attach a one-page PDF with figures of new experiments.

Meta-review: The authors consider the problem of training neural operators to solve forward partial differential equation (PDE) problems in data-limited settings, motivated by the large data simulation cost incurred by these methods when generating a large training set of PDE solutions. They propose a self-supervised pretraining method derived from tasks popular in the computer vision community, and demonstrate the benefit of their method on a variety of PDE problems and real-world experimental data, compared to other self-supervised baselines. Since the majority of operator learning approaches require “supervised” data obtained by solving the forward PDE, most baselines they compare to are to not necessarily state-of-the-art (only SotA for the unsupervised setting, which has been less studied). The authors also show experiments on an "out-of-distribution" (OOD) task where PDE coefficient settings are changed at test time and an in-context learning strategy is used together with their pretrained models to improve performance relative to baselines, as in in-context operator learning.

After the rebuttal and discussion, reviewers’ scores for the paper increased, converging towards accept. The authors provided numerous clarifications on unclear details, especially methodological and experimental, in the paper during the rebuttal, as well as new experiments which convinced reviewers of the solidity of the paper’s methodology. Reviewers expressed some concerns about the competitiveness of the authors’ experimental baselines, but acknowledge that they are reasonable in the setting of self-supervised pretraining for operator learning that the authors consider, which has been relatively less explored than the supervised case. The AC concurs with the reviewers judgment on these points, seeing the paper as providing a nontrivial contribution to a potentially important research direction (especially in light of the clarifications of computational savings given in the response to u3i6’s review), and therefore recommends acceptance. The authors should not fail to incorporate all improvements in presentation and new experimental results recommended by the reviewers and presented during the rebuttal into the revision in a prominent way (i.e., not just in the appendices), especially regarding the proposed ICL method (per comments of oEEz and oUb1, the clear dependence on the in-context operator learning approach should be emphasized in Section 3.2), results on computational savings given to u3i6, and all clarifications of protocol made in the discussion, especially with u3i6.
Decision: Accept (poster)